ktoken

BPE tokenizer designed for seamless integration with OpenAI models, supporting local and remote modes for encoding management. Offers multiplatform compatibility and flexible setup options.

• Readme
• Packages

Package	Supported platforms & targets
com.aallam.ktoken:ktoken:0.4.0	Common, JVM, Native, JS / (15 targets)	Details

Supported targets

com.aallam.ktoken:ktoken:0.4.0

JVM	11
Linux	linux_x64
Windows	mingw_x64
iOS	ios_arm64 ios_simulator_arm64 ios_x64
macOS	macos_arm64 macos_x64
tvOS	tvos_arm64 tvos_simulator_arm64 tvos_x64
watchOS	watchos_arm32 watchos_arm64 watchos_simulator_arm64 watchos_x64

Close