Enables fast and efficient tokenization for natural language processing tasks, utilizing pre-trained models and customizable pipelines for handling various languages and text processing needs.
Kotlin Multiplatform binding of 🤗 Tokenizers.
[!IMPORTANT] This library is currently under development. Partiality of the official API is prioritized for personal reasons. Contribution is welcome if you feel like something important is missing.
Whenever maven central is available, following Gradle configuration can be utilized for importing this library.
build.gradle.kts
kotlin {
sourceSets {
commonMain.dependencies {
implementation(libs.hgtk.core)
}
}
}libs.versions.toml
[versions]
hgtk = "0.0.8"
[libraries]
hgtk-core = { module = "com.zhufucdev.hgtk:core", version.ref = "hgtk" }Kotlin Multiplatform binding of 🤗 Tokenizers.
[!IMPORTANT] This library is currently under development. Partiality of the official API is prioritized for personal reasons. Contribution is welcome if you feel like something important is missing.
Whenever maven central is available, following Gradle configuration can be utilized for importing this library.
build.gradle.kts
kotlin {
sourceSets {
commonMain.dependencies {
implementation(libs.hgtk.core)
}
}
}libs.versions.toml
[versions]
hgtk = "0.0.8"
[libraries]
hgtk-core = { module = "com.zhufucdev.hgtk:core", version.ref = "hgtk" }