
implementation("ai.koog:prompt-tokenizer:0.2.1")Offers interfaces and implementations for tokenizing text and counting tokens, essential for managing token limits in large language models. Includes various tokenizers like `NoTokenizer`, `RegexBasedTokenizer`, and `TiktokenEncoder` for different accuracy and performance needs.