
implementation("ai.koog:prompt-tokenizer:0.7.0")Facilitates text tokenization and token counting, essential for managing token limits in large language models. Offers various implementations, including regex-based and byte pair encoding methods, to balance accuracy and performance.