prompt-tokenizer

Module for tokenizing text and estimating token counts, essential for managing token limits in large language models. Offers various implementations, including a simple regex-based tokenizer and an advanced byte pair encoding tokenizer.