prompt-tokenizer

Facilitates text tokenization and token counting, essential for managing token limits in large language models. Offers multiple implementations, including a simple regex-based tokenizer and an advanced byte pair encoding tokenizer.