
Evaluation harness for on-device large language models: declarative eval DSL, graders and backends, runs with quota-aware retries, records rich JSON results, and generates HTML/Markdown comparison reports.
Evaluation harness for the language models built into Android and iOS. Write an eval suite once in Kotlin Multiplatform, run it on Gemini Nano and Apple Foundation Models, grade the outputs, and compare platforms, model versions and prompt versions in one report.
Docs in apps/docs · Maven Central · Changelog · Sample report
Status: 1.1.0 is on Maven Central. The iOS backend has run on an iOS 27.0 simulator. The Android backend has run on a Pixel 11 Pro; see the table below.
The sample app (`samples/iosApp`) on an iOS 27.0 simulator, using the Mac host's Foundation Models.
The sample app (`samples/androidApp`) on a Pixel 11 Pro, Android 17, using Gemini Nano (nano-v4-full).
The OS updates the system model, not your app. A prompt that worked can answer differently after a system update, and different phones run different model versions. Google's ML Kit documentation recommends running an evaluation dataset for each Gemini Nano version, but gives no on-device tool for it. Apple's Evaluations framework covers Apple platforms only.
commonMain.dependencies { implementation("io.github.jantursky:proctor:1.1.0") }
androidMain.dependencies { implementation("io.github.jantursky:proctor-mlkit:1.1.0") }
iosMain.dependencies { implementation("io.github.jantursky:proctor-ios:1.1.0") }plugins { id("io.github.jantursky.proctor") version "1.1.0" }val suite = evalSuite("ticket-classifier") {
prompt(version = "v1") { input -> "Classify this support ticket as Billing, Bug, Account, Feature or Other: $input" }
case("My card was charged twice", id = "double-charge") { expect("Billing") }
case("The app crashes when I open the camera", id = "crash") { expect("Bug") }
graders { exactMatch(ignoreCase = true) }
repeat(3)
passWhen { score >= 0.8 }
}
val backend = ForegroundBackend(MlKitBackend(MlKitConfig.PreviewFast), ProctorHostActivity::isForeground)
val results = EvalRunner(backend).runAll(suite)
results.forEach { ResultStore.forApp(context).write(it) }Pull the *.proctor.json files from the device (scripts/pull-results.sh) and run ./gradlew proctorReport.
proctor/ holds the suite DSL, graders, the runner and the result format. It depends on no model SDK.proctor-mlkit/ calls the ML Kit GenAI Prompt API directly, proctor-ios/ calls Foundation Models through a Swift bridge.proctor-report/ and the Gradle plugin in proctor-gradle-plugin/ turn a folder of those files into an HTML and a Markdown report, with diffs between runs and a CI pass condition."Checked on" lists only runs that happened. Everything else is unverified.
| Platform | Backend | Checked on | Status |
|---|---|---|---|
| iOS simulator, iOS 27.0 | Foundation Models | Mac host, macOS 27.0.1 | Run. The simulator uses the host's model. |
| iPhone | Foundation Models | Unverified | |
| Android, stable model (full and fast) | ML Kit Prompt API | Pixel 11 Pro, Android 17, Gemini Nano 4 | Run. Classifier 12/12 on both, extraction 3/4 on both. |
| Android, Gemini Nano 4 preview (full and fast) | ML Kit Prompt API | Pixel 11 Pro, Android 17, AICore Developer Preview | Run. Fast: classifier 11/12 and 12/12, extraction 3/4. Full: classifier 12/12, extraction 3/4. |
| JVM (report, Gradle plugin) | CI | Run |
Android needs a physical device with a locked bootloader that supports the Prompt API. Emulators have no AICore. AICore serves only the foreground app and enforces a per-app quota. The Galaxy S25 supports only the feature-specific GenAI APIs, not the Prompt API.
On iOS, the Foundation Models framework has no model identifier. Proctor records an OS-derived version and labels it that way.
Proctor sends nothing anywhere: no network code, no telemetry. Prompts, outputs and result files stay on the device until you pull them. An LLM judge or an embedding provider that you pass in may call a cloud service; that is your choice.
ondevice-ai and ondevice-llm: KMP wrappers for calling the system models. They do not evaluate.I read the READMEs of Dokimos and promptfoo, not their source.
Evaluation harness for the language models built into Android and iOS. Write an eval suite once in Kotlin Multiplatform, run it on Gemini Nano and Apple Foundation Models, grade the outputs, and compare platforms, model versions and prompt versions in one report.
Docs in apps/docs · Maven Central · Changelog · Sample report
Status: 1.1.0 is on Maven Central. The iOS backend has run on an iOS 27.0 simulator. The Android backend has run on a Pixel 11 Pro; see the table below.
The sample app (`samples/iosApp`) on an iOS 27.0 simulator, using the Mac host's Foundation Models.
The sample app (`samples/androidApp`) on a Pixel 11 Pro, Android 17, using Gemini Nano (nano-v4-full).
The OS updates the system model, not your app. A prompt that worked can answer differently after a system update, and different phones run different model versions. Google's ML Kit documentation recommends running an evaluation dataset for each Gemini Nano version, but gives no on-device tool for it. Apple's Evaluations framework covers Apple platforms only.
commonMain.dependencies { implementation("io.github.jantursky:proctor:1.1.0") }
androidMain.dependencies { implementation("io.github.jantursky:proctor-mlkit:1.1.0") }
iosMain.dependencies { implementation("io.github.jantursky:proctor-ios:1.1.0") }plugins { id("io.github.jantursky.proctor") version "1.1.0" }val suite = evalSuite("ticket-classifier") {
prompt(version = "v1") { input -> "Classify this support ticket as Billing, Bug, Account, Feature or Other: $input" }
case("My card was charged twice", id = "double-charge") { expect("Billing") }
case("The app crashes when I open the camera", id = "crash") { expect("Bug") }
graders { exactMatch(ignoreCase = true) }
repeat(3)
passWhen { score >= 0.8 }
}
val backend = ForegroundBackend(MlKitBackend(MlKitConfig.PreviewFast), ProctorHostActivity::isForeground)
val results = EvalRunner(backend).runAll(suite)
results.forEach { ResultStore.forApp(context).write(it) }Pull the *.proctor.json files from the device (scripts/pull-results.sh) and run ./gradlew proctorReport.
proctor/ holds the suite DSL, graders, the runner and the result format. It depends on no model SDK.proctor-mlkit/ calls the ML Kit GenAI Prompt API directly, proctor-ios/ calls Foundation Models through a Swift bridge.proctor-report/ and the Gradle plugin in proctor-gradle-plugin/ turn a folder of those files into an HTML and a Markdown report, with diffs between runs and a CI pass condition."Checked on" lists only runs that happened. Everything else is unverified.
| Platform | Backend | Checked on | Status |
|---|---|---|---|
| iOS simulator, iOS 27.0 | Foundation Models | Mac host, macOS 27.0.1 | Run. The simulator uses the host's model. |
| iPhone | Foundation Models | Unverified | |
| Android, stable model (full and fast) | ML Kit Prompt API | Pixel 11 Pro, Android 17, Gemini Nano 4 | Run. Classifier 12/12 on both, extraction 3/4 on both. |
| Android, Gemini Nano 4 preview (full and fast) | ML Kit Prompt API | Pixel 11 Pro, Android 17, AICore Developer Preview | Run. Fast: classifier 11/12 and 12/12, extraction 3/4. Full: classifier 12/12, extraction 3/4. |
| JVM (report, Gradle plugin) | CI | Run |
Android needs a physical device with a locked bootloader that supports the Prompt API. Emulators have no AICore. AICore serves only the foreground app and enforces a per-app quota. The Galaxy S25 supports only the feature-specific GenAI APIs, not the Prompt API.
On iOS, the Foundation Models framework has no model identifier. Proctor records an OS-derived version and labels it that way.
Proctor sends nothing anywhere: no network code, no telemetry. Prompts, outputs and result files stay on the device until you pull them. An LLM judge or an embedding provider that you pass in may call a cloud service; that is your choice.
ondevice-ai and ondevice-llm: KMP wrappers for calling the system models. They do not evaluate.I read the READMEs of Dokimos and promptfoo, not their source.