prompt-tokenizer

Facilitates text tokenization and token counting, essential for managing token limits in large language models. Offers various implementations, including regex-based and byte pair encoding methods, to balance accuracy and performance.