
implementation("ai.koog:prompt-tokenizer:0.6.0-rc1")Module for tokenizing text and counting tokens, essential for managing token limits in Large Language Models. Offers various implementations, including regex-based and byte pair encoding methods, to estimate token counts accurately.