
implementation("ai.koog:prompt-tokenizer:0.8.0-rc-1")Facilitates tokenization and token counting in text, aiding in managing token limits and optimizing prompt design. Supports various tokenization strategies, including caching and on-demand processing.