
implementation("ai.koog:prompt-tokenizer:0.4.0")Offers interfaces and implementations for tokenizing text and counting tokens, essential for managing token limits in large language models. Includes various tokenizers like `NoTokenizer`, `RegexBasedTokenizer`, and `TiktokenEncoder` for different accuracy and performance needs.