prompt-tokenizer

Facilitates token counting in text strings, essential for managing token limits in large language models. Offers multiple implementations, including a simple regex-based tokenizer and an advanced byte pair encoding tokenizer. Supports custom tokenizer integration for tailored tokenization strategies.