Kotlin Multiplatform normalization suite — deterministic, byte-stable canonical forms for hashing and matching identifiers and text.

The point is one property: the same input produces the same canonical bytes, on any platform, at any time. If you hash a value into a token and throw the original away — blind tokenization, contact discovery, breach-safe matching — a one-byte difference between two platforms is not a degraded match, it is an undetectable miss. This library exists to make that impossible.

It works just as well for ordinary normalization (search keys, dedupe, display) through the same interface.

[!WARNING] This suite is alpha (0.0.x), and its shape is still being worked out. Policy ids, constant names and module boundaries may change between releases, and some changes will be breaking. Each one is listed in the changelog. Once a release is published, a policy's canonical bytes never change in place. What may change during alpha is which policies exist and what they are called. Before you store a token derived under a policy, check the changelog for the release you depend on, and do not treat an id from an unreleased design as final.

Every normalizer in the roster is built: email, credit-card/PAN, IBAN, IPv4, IPv6 and networks, MAC addresses, UUIDs and usernames in :quodlibet; configurable text normalization in :unicode; hostnames, domains and URLs in :ubilibet; UTS-39 skeletons in :confusables; and phone numbers in :phone.

Each module is its own coordinate: depend on the normalizers you use and you carry nothing else. Every module exposes :common transitively, and :ubilibet and :confusables bring :unicode with them, so you never name those yourself.

The example below installs :quodlibet, which is the table-free bundle — email, PAN, IBAN, IP addresses and networks, MAC addresses, UUIDs and usernames. Swap or add coordinates from the table above for the rest.

Moving from 0.0.1? The email normalizer was published as io.github.aughtone.normalize:email:0.0.1 and now lives in :quodlibet. Change the coordinate; nothing else moves. The package, every type name, the canonical output and the policy versions are unchanged, so no stored value is affected. The relaxed email policy has since been renamed twice: EmailPolicy.Lenient (email.lenient) became ByteStableV1Lenient (email.byte-stable+lenient) in 0.0.2, and in 0.0.3 it is ByteStableV1Subaddressed (email.byte-stable+subaddressed), because it keeps the +-subaddress rather than relaxing a rule. Its bytes never changed; only the name and id did. email:0.0.1 stays on Maven Central.

Moving from 0.0.2? Three changes are breaking, and none changes a stored byte. The Unicode normalization forms moved to text policy ids: nfc.u17 is now text.u17+nfc, and likewise for nfd, nfkc and nfkd, with TextPolicy.NfcU17 and its siblings kept as presets. The subaddress-keeping email policy is ByteStableV1Subaddressed (email.byte-stable+subaddressed), as above. A module that rebuilds policies from ids overrides PublishedPolicies.resolveBase rather than resolve, and NormalizationStep.link became links. The changelog lists each one.

// build.gradle.kts
kotlin {
    sourceSets {
        commonMain.dependencies {
            implementation("io.github.aughtone.normalize:quodlibet:0.0.3")
        }
    }
}

Or with a version catalog:

# gradle/libs.versions.toml
[versions]
aughtone-normalize = "0.0.3"

[libraries]
aughtone-normalize-quodlibet = { module = "io.github.aughtone.normalize:quodlibet", version.ref = "aughtone-normalize" }

Targets: JVM, Android, iOS (arm64, x64, simulatorArm64), JS (browser), wasmJs (browser), Linux x64.

These are deliberately different promises, and the distinction matters more here than in most libraries:

• A published policy's output is frozen forever. EmailPolicy.ByteStableV1 (id email.byte-stable, version 1) will produce the same bytes for the same input in every future release. A rules change mints a new policy version; it is never an in-place improvement, because that would orphan every token already derived under the old one. Store policyId and policyVersion beside anything you derive.
• The Kotlin API is not stable yet. At 0.0.x names and signatures may still move. Pin an exact version.

Start at docs/README.md, which explains how the documentation is organized. The knowledge base itself lives in docs/knowledge/:

• 📐 Specifications: how the suite is built and the standards it is held to.
• 🧭 Architecture Decision Records: hard-to-reverse choices and why the alternatives lost.
• 🔬 Research: investigations and designs still being worked out.
• 📖 Developer Guides: building, extending and releasing this project.
• 📌 Reference: settled facts kept close, including what this suite has decided not to build.
• 📜 Changelog: history of changes and release notes.

Work is tracked as issues rather than documents — see WORKFLOW.md.

import io.github.aughtone.normalize.email.normalizeEmail
import io.github.aughtone.normalize.email.EmailPolicy
normalizeEmail(value, EmailPolicy.ByteStableV1)
    .onSuccess { normalized ->
        // the hash is stable across platforms and builds; keep the policy identity beside it
        store(hash(normalized.canonical), normalized.policyId, normalized.policyVersion)
    }
    .onFailure { failure -> log(failure.exception) }   // a typed, value-free EmailNormalizationError

// or, where a failure needs no handling of its own
val canonical: String? = normalizeEmail(value, EmailPolicy.ByteStableV1).dataOrNull()?.canonical

To match on the subaddress as well as the mailbox, read both from one parse: normalizeEmailWithSubaddress(value, EmailSubaddressPolicy.ByteStableV1) returns the mailbox, byte-identical to normalizeEmail under ByteStableV1, and the subaddress (tag for user+tag@example.com, null without a +). The subaddress carries its own id, email.subaddress, so a stored tag token records what it is.

import io.github.aughtone.normalize.unicode.TextPolicy
import io.github.aughtone.normalize.unicode.UnicodeRelease
import io.github.aughtone.normalize.unicode.normalizeText

// ASCII rules only: text+trim+lower. Names no Unicode release, so it never goes stale.
val field = TextPolicy { ascii { trim(); lowercase() } }

// Unicode rules against frozen Unicode 17 data: text.u17+trim+casefold+nfc
val caseless = TextPolicy(UnicodeRelease.U17) { unicode { trim(); casefold(); nfc() } }

normalizeText(value, caseless)
    .onSuccess { normalized -> store(normalized.canonical, normalized.policyId, normalized.policyVersion) }
    .onFailure { failure -> log(failure.exception) }          // a typed, value-free TextNormalizationError

normalizeText is one configurable normalizer for general text fields. The rules are stripControl, trim, collapseSpace, removeSpace, lowercase, uppercase, casefold, the four normalization forms and nonEmpty, and each runs over the character set of the block it sits in: ascii { } touches only ASCII, unicode { } uses tables frozen from the named Unicode release and shipped with the library, never the platform's. A new OS release cannot change what your application produces, and a new Unicode release is a new policy (text.u18…), never a changed one.

Rules always run in one fixed order — strip control characters, trim, spaces, case, normalization form, then the non-empty check — however you write them, so the same rules always produce the same bytes and one configuration has one id. Writing them in a different order still works, and is reported two ways: at runtime through policy.warnings and TextPolicy.warningHandler (printed by default, replaceable or silenceable), and in the editor by the TextPolicyRuleOrder lint check that ships inside the :unicode Android artifact. The lint check needs no setup in a build with an Android target, where Android Studio underlines the builder and ./gradlew lint reports it; JVM-only, iOS and web builds get the runtime warning.

The id states the Unicode release once and only when a rule uses it: text+trim+lower is all ASCII, text.u17+trim+lower.ascii marks the one ASCII rule inside a Unicode policy. NfkcU17, NfkdU17 and the nfkc/nfkd rules are deliberately lossy — a ligature becomes its letters and cannot be turned back — so they are for search, not for a token you expect to round-trip. Named presets such as TextPolicy.NfcU17, TrimLowercase and CaselessU17 are conveniences for common configurations, nothing more.

This is not an identifier normalizer: emails, domains, phone numbers and handles have their own.

import io.github.aughtone.normalize.ipv4.Ipv4Policy
import io.github.aughtone.normalize.ipv4.block
import io.github.aughtone.normalize.ipv4.cidr
import io.github.aughtone.normalize.ipv4.normalizeIpv4Block
import io.github.aughtone.normalize.ipv4.normalizeIpv4Blocks
import io.github.aughtone.normalize.ipv4.normalizeIpv4Cidr

normalizeIpv4Block("192.0.2.57", Ipv4Policy.DottedQuad.block(24))       // "192.0.2.0/24"
normalizeIpv4Blocks("192.0.2.57", Ipv4Policy.DottedQuad, listOf(24, 16)) // "192.0.2.0/24", "192.0.0.0/16"
normalizeIpv4Cidr("192.0.2.0/24", Ipv4Policy.DottedQuad.cidr())         // "192.0.2.0/24"; host bits set is refused

Block derivation buckets an address into the network it falls in, and every prefix is its own identity (ipv4.dotted-quad+block-24). To match an address against a range from a list, run both through the same block policy: the range's network address, which CIDR input exposes, and the incoming address, at the same prefix. CIDR input comes strict (cidr, refusing host bits) or masked (cidrMasked, clearing them), and both write a network exactly as block derivation does.

IPv6 works the same way, and has modes for systems that need a different reading, each named in the id: unmap writes an IPv4-mapped address as IPv4 so it matches the IPv4 spelling of the same host, nat64 does the same for 64:ff9b::/96, and zone keeps a zone identifier. A block under unmap or nat64 carries both prefixes: Ipv6Policy.Rfc5952.unmap().block(24, 64).

These policies declare comparable forms (IpForms), so the matches they exist for are explicit rather than coincidental: an IPv4 address is comparable with its unmap or nat64 spelling in ipv4.address, and a derived block with a CIDR range in ipv4.network or ipv6.network. inet-aton only offers its forms, because its reading of 010 as 8 is an interpretation: opt in with Ipv4Policy.InetAton.withForms(setOf(IpForms.Ipv4Address)), which stores ipv4.inet-aton+form.ipv4.address.

import io.github.aughtone.normalize.mac.MacNotation
import io.github.aughtone.normalize.mac.MacPolicy
import io.github.aughtone.normalize.mac.formatMac
import io.github.aughtone.normalize.mac.normalizeMac

normalizeMac("00-00-5E-00-53-01", MacPolicy.Eui48)   // "00:00:5e:00:53:01"
normalizeMac("0000.5e00.5301", MacPolicy.Eui48)      // "00:00:5e:00:53:01"

val address = normalizeMac("0:0:5e:0:53:1", MacPolicy.Eui48).dataOrThrow()
formatMac(address, MacNotation.Ieee)                  // "00-00-5E-00-53-01", for display only

Every spelling of one address - colon, hyphen, Cisco dotted, bare, any case, leading zeros omitted - normalizes to lowercase colon pairs. The IEEE registry's notation is accepted but never produced, because two canonical notations could never match each other; formatMac renders any notation for display, and its output is not an identity to store. EUI-48 and EUI-64 are separate policies and neither is widened into the other.

import io.github.aughtone.normalize.uuid.UuidNotation
import io.github.aughtone.normalize.uuid.UuidPolicy
import io.github.aughtone.normalize.uuid.formatUuid
import io.github.aughtone.normalize.uuid.normalizeUuid

normalizeUuid("{919108F7-52D1-4320-9BAC-F847DB4148A8}", UuidPolicy.Hex)         // "919108f7-52d1-4320-9bac-f847db4148a8"
normalizeUuid("urn:uuid:919108f7-52d1-4320-9bac-f847db4148a8", UuidPolicy.Rfc9562)
normalizeUuid("f7089191d15220439bacf847db4148a8", UuidPolicy.Hex.guidBytes())   // a Windows GUID byte dump

Any case, braces, a urn:uuid: prefix and the bare 32-digit form all normalize to lowercase hyphenated text. UuidPolicy.Hex accepts any 128-bit value; UuidPolicy.Rfc9562 also requires RFC 9562's variant and version bits, and both write the comparable form uuid.

Windows GUIDs store their first three fields little-endian, so a GUID read from raw bytes and hex-encoded looks like a different UUID, and nothing in the text says which reading is meant. The guidBytes() mode is how a caller says the input is a byte dump: it swaps those fields back, and refuses braces and urn:uuid:, which only string forms carry. Because the result is only right if the caller is, the mode offers the uuid form rather than declaring it: opt in with withForms(setOf(UuidForms.Uuid)). formatUuid renders braces, a URN, uppercase, bare or GUID byte order for display.

import io.github.aughtone.normalize.ubilibet.DomainPolicy
import io.github.aughtone.normalize.ubilibet.normalizeDomain

normalizeDomain(value, DomainPolicy.AsciiU17)
    .onSuccess { normalized -> store(normalized.canonical) }   // "café.fr" -> "xn--caf-dma.fr"
    .onFailure { failure -> log(failure.exception) }          // a typed, value-free DomainNormalizationError

Every hostname goes through the same function, ASCII included: a second, simpler rule for ASCII names would produce identical bytes under a different policy identity, which is a mismatch waiting to happen. AsciiU17 applies every UTS-46 check; AsciiU17Lenient relaxes hyphen placement, the STD3 character restriction and DNS length, and keeps the bidi and joiner rules — those exist to stop a name that displays as one thing and resolves as another, which is not something leniency should buy.

To show a domain to a person, toUnicodeDomain(value, policy) converts it to U-labels under the same checks and reports per label which failed, so a display can show café.fr where a label validates and its A-label where it does not. The result is for display, not identity: it carries no policy id, and the canonical form to store and match is always the A-label from normalizeDomain.

import io.github.aughtone.normalize.phone.PhonePolicy
import io.github.aughtone.normalize.phone.normalizePhone

normalizePhone("+1 (212) 555-0123", PhonePolicy.E164).dataOrNull()?.canonical               // "+12125550123"
normalizePhone("(212) 555-0123", PhonePolicy.e164ForRegion("us")).dataOrNull()?.canonical   // "+12125550123"

The region travels on the policy, and nothing is ever guessed. E164 accepts only input carrying its own country code; e164ForRegion reads national-format input against a region you named. A guessed country code does not fail loudly — it produces a valid-looking token for a different number, and by then the input is gone. The region is part of the identity, because it records how national input was read, so phone.e164 and phone.e164+region-ca are different policies. Every phone policy writes the same E.164 number, though, and declares the comparable form phone.e164 (PhoneForms.E164): systems that read numbers with different regions, or leniently, match through comparability rather than by trusting that their strings agree.

import io.github.aughtone.normalize.confusables.ConfusablePolicy
import io.github.aughtone.normalize.confusables.normalizeSkeleton

val a = normalizeSkeleton("paypal", ConfusablePolicy.SkeletonU17).dataOrNull()?.canonical
val b = normalizeSkeleton("раypal", ConfusablePolicy.SkeletonU17).dataOrNull()?.canonical   // Cyrillic р and а
// equal canonical values: the second is a lookalike of the first

A skeleton is a check, never an account key. It is deliberately many-to-one — that is what makes a lookalike collide with its target — so two genuinely different users can share one. Derive your identity from the plain value, compute the skeleton beside it, and use a collision to flag something for review.

It is also a NormalizationStep, so a table-free normalizer can take it from a caller: normalizeUsername(value, UsernamePolicy.Basic, listOf(ConfusablePolicy.SkeletonU17)) produces the policy identity username.basic+skeleton.u17, which never matches the plain username.basic.

The id and version stored beside a hash resolve back to the policy that produced them, so new values can be normalized the same way as the ones already in a store — and so a policy can be named in a configuration file instead of hardcoded.

import io.github.aughtone.normalize.email.EmailPolicy
import io.github.aughtone.normalize.email.normalizeEmail
import io.github.aughtone.normalize.quodlibet.QuodlibetPolicies

// policyId and policyVersion were stored next to the hash when the first value was normalized
val policy = QuodlibetPolicies.resolve(policyId, policyVersion).dataOrThrow() as EmailPolicy
normalizeEmail(newValue, policy)

Matching across policies is explicit. Values from different policies never match by coincidence, but policies can declare a comparable form they both write, and a caller can opt into one a policy offers by naming it in the id (…+form.ipv4.address). Ask before matching:

import io.github.aughtone.normalize.common.Comparability
import io.github.aughtone.normalize.common.comparability

when (val result = policies.comparability(idA, versionA, idB, versionB).dataOrThrow()) {
    Comparability.SamePolicy -> match()
    is Comparability.InForm -> match()          // result.form names the declaration that allows it
    Comparability.NotComparable -> skip()
}

Resolution is explicit: combine the resolvers of the modules you depend on with +. There is no global registry and no startup registration, and an unknown id, an unknown link or a version this build does not carry fails loudly rather than resolving to something close — a nearly-right policy silently derives bytes that match nothing already stored.

Read Getting Started first, then AGENTS.md if you are an agent. The one rule that matters more than the rest: a published policy's canonical output never changes in place — see Normalization Suite Structure.

Apache License 2.0 — see LICENSE and NOTICE.md.