
implementation("ai.koog:prompt-tokenizer:0.5.3")Facilitates text tokenization and token counting, essential for managing token limits in large language models. Offers various implementations, including regex-based and byte pair encoding methods, to balance accuracy and performance.