
Small modular internet I/O toolkit: multi-client UDP with NAT-traversal handshake, line-based URL connector, headless-browser screenshots, typed asynchronous HTTP stack with cache, rate budgets and hardened feed fetching.
A small Kotlin/JVM collection of internet I/O helpers, published as independent, single-concern artifacts — depend only on the one you need:
| Artifact | Package | Purpose | Third-party runtime deps |
|---|---|---|---|
io.github.spartanlaboratories:webtools-udp |
com.spartanlabs.webtools.udp |
Multi-client UDP connection layer + NAT-traversal handshake | none (slf4j-api only) |
io.github.spartanlaboratories:webtools-scraping |
com.spartanlabs.webtools.scraping |
Connector — open/read a URL line by line, plus one-shot get / skrape / image download
|
skrapeit, jsoup, unirest |
io.github.spartanlaboratories:webtools-browser |
com.spartanlabs.webtools.browser |
WebViewer — headless-browser screenshots |
selenium-java |
io.github.spartanlaboratories:webtools-http |
com.spartanlabs.webtools.http |
Kotlin Multiplatform (JVM target) — request/response types, a Url that reads like OkHttp's, a typed failure model, the transport and wall-clock ports of a suspend HTTP client stack, OkHttpTransport (the supplied transport: one request per hop, manual redirects, a capped decoded body, never a retry), the ResponseCache port with CaffeineResponseCache (the supplied cache: one load per key at a time, per-entry lifetimes, a weight bound, a per-call maxAge, failures never stored) and the pass-through ResponseCache.Disabled, the CallBudget port with WindowedCallBudget (the supplied budget: epoch-aligned rate windows with a boundary guard, per-lane floors, first-come-first-served waits that fail fast or are refunded and re-planned, pauses from Retry-After and reset hints, and a hard window that survives a restart through a BudgetStore), CallBudget.Unlimited, RateSignalsReader.Standard and RetryAfter.parse, and the Trusted/Untrusted outbound-URL policies with their transport settings (timeouts, size and redirect caps, concurrency), and the call facades UpstreamClient (declared Endpoints, a cache, a budget and a transport composed in one fixed order, with an owning JVM factory) and UrlFetcher (administrator-supplied URLs checked against the policy first) |
okhttp-jvm, caffeine, kotlinx-coroutines-core, slf4j-api |
io.github.spartanlaboratories:webtools-feeds |
com.spartanlabs.webtools.feeds |
Kotlin Multiplatform (JVM target) — RSS 0.90 to 2.0, RSS 1.0 and Atom 0.3 and 1.0 behind a hardened parser port (FeedParser) with Rome 2.1.0 as the supplied parser (RomeFeedParser: DOCTYPE refused before any parser runs, structural limits on depth, attributes, elements and namespaces, text/id/link/entry bounds), a feed model with a frozen entry id, and FeedFetcher, a stateless conditional GET over UrlFetcher that returns a 304 as a value and a permanent move as a fact |
webtools-http (and its dependencies), rome |
Every fallible operation returns kotlin.Result rather than throwing, so callers decide how
to recover. webtools-http and webtools-feeds are suspend APIs: every call that sends a request
or may wait (UpstreamClient.call, UrlFetcher.fetch, FeedFetcher.fetch, HttpTransport.execute,
ResponseCache.getOrLoad, CallBudget.acquire, flush, utilisation) suspends, the supplied
implementations never block the caller's thread, and there is no blocking variant. Building a component and close() do not suspend.
State boards: docs/dashboards/ shows the modules as built at a named commit, with planned systems drawn dashed.
Work boards: the repository work board shows every planning and development run, and each issue's board under docs/plans/ traces every part of a change through its gap, the options considered, the verdict and the as-built state, with the work in progress.
API reference and project site: https://spartanlaboratories.github.io/WebTools/ (landing page) and https://spartanlaboratories.github.io/WebTools/api/ (the Dokka reference of the five published modules).
dependencies {
implementation("io.github.spartanlaboratories:webtools-udp:2.0.1")
// and/or
implementation("io.github.spartanlaboratories:webtools-scraping:1.0.0")
implementation("io.github.spartanlaboratories:webtools-browser:1.0.0")
implementation("io.github.spartanlaboratories:webtools-http:1.0.0")
// tests only (Gradle): fakes, a virtual clock, .http fixtures and a recorder
testImplementation(testFixtures("io.github.spartanlaboratories:webtools-http:1.0.0"))
implementation("io.github.spartanlaboratories:webtools-feeds:1.0.0")
}Requires JDK 11 or newer (webtools-scraping and webtools-browser 1.0.0: JDK 23 or newer). Built with Kotlin 2.2.
webtools-http is a Kotlin Multiplatform module that publishes webtools-http and webtools-http-jvm;
a Gradle build resolves the right one itself, a Maven build depends on webtools-http-jvm. Its okhttp-jvm
5.5.0 dependency requires kotlin-stdlib 2.2.21, so a consumer's Kotlin standard library resolves to 2.2.21 or newer.
webtools-feeds is a Kotlin Multiplatform module that publishes webtools-feeds and webtools-feeds-jvm (a Maven build depends on webtools-feeds-jvm). It exposes
webtools-http as an api dependency and brings Rome 2.1.0 (with rome-utils and jdom2) at runtime; the fakes for its tests are webtools-http's test fixtures.
The test-fixtures variant of webtools-http (capability
io.github.spartanlaboratories:webtools-http-test-fixtures, a -test-fixtures.jar beside the module, JVM
only) holds the package com.spartanlabs.webtools.http.testing: VirtualClock, FakeHttpTransport with
RequestMatch, Step and Unmatched, Fixture and FixtureLoader for .http fixture files,
FixtureRecorder with RecordingGate and Redaction, and LoopbackTransports (the real OkHttpTransport
over a loopback mockwebserver3 server, with the policy's rules live), MockWebServer.webtoolsUrl,
Fixture.toMockResponse and MockWebServer.enqueue(fixture), LoopbackFetchers (a UrlFetcher over a loopback
server with the policy's rules live), and RecordingBudgetObserver and FakeBudgetStore
(a scripted budget store with failure injection and virtual-time delays). testFixtures(...) is Gradle-only: a
Maven build cannot select a variant by capability. The variant adds kotlinx-coroutines-test and mockwebserver3
to the test classpath only. Fixture files live in the consumer's own test resources; add
**/fixtures/**/*.http -text to its .gitattributes so a checkout does not rewrite their line endings.
Recording is off unless WEBTOOLS_HTTP_RECORD has a recognised value, and never runs where a CI variable
(CI, GITHUB_ACTIONS, ...) is set; a build that records declares the variable as an input of its test tasks,
or Gradle skips them as up to date when only the variable changed.
The Java 11 floor starts with webtools-udp 2.0.1: Java 11 bytecode (class-file major 55), compiled
against the Java 11 API, as are webtools-http and webtools-feeds. The earlier releases on Maven
Central — webtools-udp 1.0.0, 1.1.0 and 2.0.0, webtools-scraping 1.0.0 and webtools-browser
1.0.0 — are Java 23 bytecode and fail to load on JDK 22 or older with
UnsupportedClassVersionError. The Java 11 builds of the two archived modules are 1.0.0a in this
repository and have not been published. Building the repository needs a JDK 17 to 26; see Java floor.
webtools-udp 2.0.0 is a wire break for everything after the handshake — see Migrating to
2.0. Both ends of a connection must be on 2.0.0+; a 1.x ↔ 2.x pairing
fails the handshake cleanly instead of exchanging data either side can parse. See
docs/webtools-udp-protocol.md for the complete wire reference
and docs/webtools-udp-architecture.md for the component,
thread, and lifecycle picture.
The combined WebTools artifact is retired at its final version 2.0.1a and receives no
further releases. To move off it:
WebTools dependency with whichever of the three modules you actually use.com.spartanlabs.webtools.Foo → com.spartanlabs.webtools.{udp,scraping,browser}.Foo.WebTools carried an unused api dependency on GeneralTools; if you relied on it
transitively, depend on it directly now.webtools-browser no longer ships webdriver binaries — Selenium
Manager resolves the driver at
runtime (needs a local Chrome/Chromium install and, on first run, network access).| Type | Purpose |
|---|---|
MultiConnectionUDPServer |
Accepts handshakes from many clients on one common port and hands each its own Connection. Abstract — subclass and implement onClientConnect; optionally override admit to screen/refuse handshakes and validate an opaque credential, override onClientDisconnect, and set idleTimeoutMillis for idle-connection detection. |
MultiConnectionUDPClient |
The client-side counterpart: one socket, one owned listener thread, one dispatch thread — performs the handshake, then delivers the rest of the session via callback. channel(mode) is the recommended entry point (see Reliable-ordered channel); the older send / start / startBytes (deprecated, still fully functional — see Migrating to 2.0) also accept/deliver raw ByteArray datagrams via their ByteArray overloads. startKeepAlive / stopKeepAlive run an opt-in, library-timed keepalive; startProbe / stopProbe / linkQuality run an opt-in RTT / packet-loss probe. |
Connection |
Interface for one named connection to a peer. channel(mode) (see below) is the recommended entry point; actuate / actuateBytes / push(String) / push(ByteArray) are deprecated but fully functional (see Migrating to 2.0); terminate / keepAlive / startKeepAlive / stopKeepAlive / startProbe / stopProbe / linkQuality are unchanged control-plane members. |
UDPConnection |
The production Connection: a socket-free handle to one multiplexed client of a MultiConnectionUDPServer; owns no socket. |
DeliveryMode |
Which guarantee a UdpChannel provides: UNRELIABLE (fire-and-forget, the 0x90 frame) or RELIABLE_ORDERED (acked, retransmitted, exactly-once in order, the 0xA0/0xA1 frames — head-of-line blocking is inherent). |
UdpChannel |
A delivery-mode-scoped view of one connection: send(bytes/message) and actuate/actuateBytes to bind the inbound handler for that mode. Returned by Connection.channel(mode) / MultiConnectionUDPClient.channel(mode). See Reliable-ordered channel. |
ReliableWindowFullException / ReliableMessageTooLargeException
|
The two typed reliable-send failures, both subtypes of the open ReliableSendFailure; never thrown, only ever a Result.failure payload. See Reliable-ordered channel. |
LinkQuality |
An immutable snapshot from the opt-in probe: smoothed rttMillis, rttVarianceMillis (jitter proxy), windowed packetLossRatio, and probesSent / probesDelivered. |
UDPSendReceiveServer |
A bound send/receive UDP socket pair with an async receive loop — receives datagrams up to 65507 bytes (configurable via receiveBufferBytes), truncating larger ones. No DeliveryMode / UdpChannel surface — see its KDoc. |
HandshakeWireFormat |
The handshake-bootstrap wire format: Iam <name> [<credential>], the accepted REGISTERED <version> reply, REFUSED <reason>. Everything after the handshake lives in TransportWireFormat. |
DatagramType |
The 1-byte tag carried as byte 0 of every post-REGISTERED datagram (KEEPALIVE 0x80, PROBE_PING 0x81, PROBE_PONG 0x82, UNRELIABLE 0x90, RELIABLE_DATA 0xA0, RELIABLE_ACK 0xA1); the listener switches on it before touching a payload. |
TransportWireFormat |
Builds/parses the framed post-handshake datagrams: keepaliveDatagram, probePingDatagram / probePongDatagram / probeSequenceOf, unreliableDatagram / unreliablePayloadOf / unreliableChannelOf. |
Admission / HandshakeRefusedException
|
admit's return type (Admitted / Refused(reason)); the typed failure a refused client's handshake surfaces. |
IncompatibleProtocolException |
Typed handshake() failure when the server's REGISTERED reply announces a different wire-protocol major than this build speaks (e.g. a 1.x server). |
resolveLocalAddress() |
Best-effort lookup of this machine's outward-facing local address. |
API stability. The whole webtools-udp 2.x public surface — including DeliveryMode,
DatagramType, DisconnectReason, UdpChannel, ReliableSendFailure and its subtypes, and
every other public type — is Stable Core: breaking changes only in a major. DeliveryMode,
DatagramType, and DisconnectReason may each gain a new entry in a minor release; keep an
else branch in any when over one of them. The eight members deprecated in 2.0.0 (see
Migrating to 2.0) stay fully functional — still compiling, deprecated at
WARNING only — for all of 2.x, and leave the public surface only at 3.0.0.
| Type | Purpose |
|---|---|
Connector |
Single-connection web client: open a URL and read it line by line; plus one-shot get, skrape, and image download helpers. |
For suspend calls, timeouts and size caps, an outbound-URL policy, response caching and a call budget, use
webtools-http instead.
| Type | Purpose |
|---|---|
WebViewer |
Headless-Chrome screenshot utility — screenshot(url) returns a Result<BufferedImage>, getPage(url) a Result<File>. Driver provisioned by Selenium Manager. |
| Type | Purpose |
|---|---|
UpstreamClient |
Calls the endpoints of one upstream API from a base URL: the outbound-URL policy's pre-check, then the response cache, then the call budget inside the cache's loader, then the transport. Each call is declared once as an Endpoint; CallSpec carries the per-call path, query, headers, lane and waits. See HTTP client stack. |
UrlFetcher |
Fetches administrator-supplied URLs: the pre-check, the budget and the transport, no cache. Every status is the response except a throttle. |
OutboundUrlPolicy |
Trusted (a known upstream: https only unless cleartext is allowed, optional host pins) or Untrusted (a URL a third party supplied: lenient host forms, reserved names, denied addresses, other ports and sensitive headers refused), each carrying its TransportSettings. See Outbound-URL policies. |
OkHttpTransport |
The supplied HttpTransport: one request per hop, redirects followed by hand and judged hop by hop, a capped decoded body, never a retry. |
CaffeineResponseCache / ResponseCache.Disabled
|
The supplied ResponseCache (one load per key at a time, per-entry lifetimes, a weight bound, failures never stored) and the pass-through that stores nothing. |
WindowedCallBudget / CallBudget.Unlimited
|
The supplied CallBudget (epoch-aligned rate windows, lane floors, fail-fast waits, pauses from the server's signals, a hard window kept across a restart through a BudgetStore) and the budget that grants everything. |
HttpRequest / HttpResponse / HttpHeaders / Url
|
The value types. A header added with addSensitive is treated as a credential (refused under Untrusted, dropped on a cross-origin redirect); Url.redacted() is the form the stack logs. |
HttpFailure |
The failure model carried in Result: TransportFailure (nothing usable came back), UpstreamFailure (RateLimited, UpstreamError), BudgetExhausted, DecodeFailure and Closed. |
ExperimentalForInheritanceWebToolsApi |
The opt-in marker for implementing one of the stack's seams. |
API stability. Everything a consumer calls, passes, configures or matches is Stable Core,
the supplied implementations (OkHttpTransport, CaffeineResponseCache, WindowedCallBudget,
RateSignalsReader.Standard, …) included: none of them needs an opt-in. Implementing, delegating to or
subclassing a seam (HttpTransport, ResponseCache, CallBudget, Permit, RateSignalsReader, BudgetStore,
BudgetObserver, BudgetEvent, WallClock, and the failure groups HttpFailure, TransportFailure
and UpstreamFailure) is Experimental: it needs @OptIn(ExperimentalForInheritanceWebToolsApi::class)
and may change in a minor release. A seam graduates to @SupportedExtension in a feature release once a
consumer has built against it. The failure groups and the reason vocabularies (RefusalReason,
ExhaustedReason, …) may gain members in a feature release; keep an else branch in any when over
one of them. The test fixtures are Stable Core with source compatibility only.
| Type | Purpose |
|---|---|
FeedFetcher |
A stateless conditional GET over UrlFetcher: a 304 is FeedFetch.NotModified, a feed FeedFetch.Fetched. On the JVM, FeedFetcher.untrusted(userAgent) builds and owns an Untrusted UrlFetcher. See Feeds. |
RomeFeedParser / FeedParser
|
The supplied parser over Rome 2.1.0 (a DOCTYPE refused before any parser runs, structural limits) and the port behind it. |
Feed / FeedEntry / FeedEnclosure
|
The model; an entry's id is never blank and never changes for the same entry. |
FeedValidators / FeedLimits
|
The ETag and Last-Modified pair the caller stores between polls; the bounds a parser applies, which can only be tightened. |
FeedFailure |
NotAFeed, Malformed, Refused and LimitExceeded, under HttpFailure. |
API stability. Everything a consumer calls, passes, configures or matches is Stable Core,
RomeFeedParser included. Implementing FeedParser or subclassing FeedFailure is Experimental, under
webtools-http's opt-in marker. FeedFailure may gain leaves, and its reason vocabularies (NotAFeedReason,
FeedRefusal, FeedLimit) may gain members, in a feature release; keep an else branch.
webtools-http composes a response cache, a call budget and a transport behind two facades.
UpstreamClient calls the endpoints of one upstream API; UrlFetcher fetches URLs an administrator
supplies. Both are AutoCloseable, both return kotlin.Result, and neither ever retries: a throttle, a transport
failure and a 5xx are each returned once. docs/webtools-http-architecture.md
describes the stack as built, webtools-feeds included: its components, the call path, where each outbound-URL
check runs, the failure model, the threads, ownership and closing, what is logged, the stability tiers, the test
support and the known limitations.
val client = UpstreamClient(Url.parse("https://api.example.com/v2/").getOrThrow(), "my-app/1.0 (+https://example.com/contact)") {
headers = HttpHeaders { addSensitive("Authorization", "Bearer $token") } // default headers; credentials are never printed
budget(listOf(RateWindow("minute", 60, 1.minutes) { margin = 10 })) // built and owned by the client
}
// Declare each call once: a stable id, a decoder, how long to keep the answer, which status means "nothing there".
val player = Endpoint("player", ::decodePlayer) {
ttl = 10.minutes
misses(404)
}
val result: Result<Lookup<Player>> = client.call(player) {
path("players", id) // raw segments; each is encoded, so "a/b" stays one segment
query("region", "eu")
maxAge = 1.minutes // accept a cached answer no older than this
}
result.fold(
onSuccess = { lookup -> when (lookup) { is Lookup.Found -> show(lookup.value); Lookup.Missing -> showNotFound() } },
onFailure = { failure -> when (failure) {
is UpstreamFailure.RateLimited -> backOff(failure.retryAfter) // wait this long, then call
is HttpFailure.BudgetExhausted -> backOff(failure.wait)
else -> report(failure) // keep an else: new failure types may be added
} },
)
client.flush() // the exact final write of a budget with a persistent store
client.close() // closes what the client built (cache, budget, transport), never what it was handedGET is cached.ttl defaults to zero (in-flight calls coalesce, nothing is stored); missTtl defaults to
the smaller of ttl and 60 seconds; cost defaults to 1 and lane to Lane.Interactive.Found or Missing on success. A 429 (or what the client's rateSignals reader reports as a
throttle) is RateLimited, whose retryAfter is the longer of the server's Retry-After and the time the budget
stays paused; a 2xx the decoder rejects is DecodeFailure; any other status is UpstreamError; the budget's refusal is
BudgetExhausted; the transport's and the policy's are TransportFailure leaves. Failures are never cached. Once a
permit is granted, only an InvalidRequest gives it back: every other failure the transport returns, a redirect or
an address its policy refuses included, leaves it spent.defaultMaxWait (5 seconds; CallSpec.maxWait per call) for a permit, then
fails with BudgetExhausted carrying the wait it would have needed. The worst-case latency of a call is that wait
plus the transport's call timeout.Trusted, pinned to the base URL's host, https only. Pass
policy = OutboundUrlPolicy.trusted { … } to change it. The JVM factory raises OkHttp's concurrency limits to the
budget's burst so a granted permit never queues in OkHttp; pass your own transport to the common factory
UpstreamClient(transport, baseUrl) { … } to size it yourself.cache(CaffeineResponseCache { maxWeight = 5_000 }) and budget(myBudget) are
caller-owned: the client never closes them.com.spartanlabs.webtools.http (UpstreamClient,
UrlFetcher, OkHttpTransport, CaffeineResponseCache, WindowedCallBudget, and GuardedDns at DEBUG), never a header value, a body, a
query string or a decoded value. A URL is printed with Url.redacted(), which hides the query and shows the path,
and an endpoint id is printed as given: keep credentials out of path segments, the base URL's path and endpoint ids
(#93).OutboundUrlPolicy is a closed pair. Trusted, for an upstream you chose, checks the scheme (https only unless
allowCleartext), optionally pins hosts and filters no address. Untrusted, for a URL a third party supplied, also
refuses lenient host forms (127.1, 2130706433), single-label names, reserved names (localhost, internal, …), private, loopback,
link-local and other special-purpose addresses, whether written as a literal or resolved from a name, ports other
than 80, 443, 8080 and 8443, and any request carrying a sensitive header; it never uses a proxy. Its parameters can
never admit a denied address or name (extraDeniedRanges and extraDeniedHostSuffixes add denials; allowedPorts
sets the port list; mixedAnswers chooses between refusing a mixed answer and using its allowed addresses). Each policy
carries its TransportSettings: Trusted 30 s per call, 10 s per socket operation, 5 MiB, 5 redirects, 16
concurrent requests (5 per host); Untrusted 15 s, 5 s, 2 MiB, 3 redirects, 8 (2 per host). Change one with
OutboundUrlPolicy.untrusted { settings { callTimeout = 20.seconds } }.
UrlFetcher("my-app/1.0 (+https://example.com/contact)").use { fetcher ->
fetcher.fetch(HttpRequest(HttpMethod.Get, url)) { lane = Lane.Tracker }.fold(
onSuccess = { response -> handle(response) }, // every status, a 304 included, is a response
onFailure = { failure -> report(failure) }, // RefusedByPolicy, RateLimited, BudgetExhausted, a transport failure …
)
}UrlFetcher uses an Untrusted policy (private, loopback and link-local addresses refused, a short list of ports,
no sensitive header) and has no cache and no decoder. Pass a policy of your own
(UrlFetcher(userAgent, OutboundUrlPolicy.untrusted { extraDeniedRanges += … })) to add your deployment's own denied
ranges, and budget = … in the lambda to share a budget. Over a transport of your own (a FakeHttpTransport in a
test) use the common factory UrlFetcher(transport, policy) { … }: policy is the pre-check's, the transport
enforces its own, and the fetcher never closes the transport.
FakeHttpTransport, VirtualClock and LoopbackFetchers (all in testFixtures) cover the stack without a network:
runTest {
val clock = virtualClock()
val fake = FakeHttpTransport(clock.timeSource).on(RequestMatch.get("/v2/players/7"), Step.respond(200, body = """{"name":"Ada"}"""))
UpstreamClient(fake, baseUrl) { timeSource = clock.timeSource; wallClock = clock.wallClock }.use { client ->
assertEquals(Lookup.Found(Player("Ada")), client.call(player) { path("players", "7") }.getOrThrow())
assertEquals(1, fake.calls.size)
}
}For a real exchange over loopback, start a mockwebserver3.MockWebServer on 127.0.0.1 and use
LoopbackFetchers.untrusted("test") (a UrlFetcher with every Untrusted rule live except the loopback ones) or
LoopbackTransports.trusted(…) with the common UpstreamClient factory.
webtools-feeds parses RSS and Atom behind a hardened port and fetches feeds with a stateless conditional GET over UrlFetcher. It keeps no state: you store the validators it returns and
hand them back on the next poll. docs/webtools-http-architecture.md describes the
fetch path and the hardening layers (§4.5) and the module's known limitations (§13).
FeedFetcher.untrusted("my-app/1.0 (+https://example.com/contact)").use { fetcher ->
val stored: FeedValidators = loadValidators(feedId) // FeedValidators.None on the first poll
fetcher.fetch(feedUrl, stored).fold(
onSuccess = { fetch ->
saveValidators(feedId, fetch.validators) // after every response
fetch.permanentlyMovedTo?.let { moved -> considerMoving(feedId, moved) }
when (fetch) {
is FeedFetch.Fetched -> store(fetch.feed.entries) // new items: entries whose id you have not seen, never by date
is FeedFetch.NotModified -> Unit
}
},
onFailure = { failure -> report(failure) },
)
}finalUrl changes (a validator from one server must not reach another); deciding whether to adopt permanentlyMovedTo (it is a fact
about the redirect chain: adopt it only when the chain's first request was https, or after it was reported on several polls); polling, back-off, 410, the first poll and "new items".Result: the stack's own (RefusedByPolicy, RateLimited, BudgetExhausted, transport failures, UpstreamError for any status that is not a feed, 304 to an unconditional request
included) pass through unchanged; a document that is not a feed is FeedFailure.NotAFeed (a web page, an empty or non-XML body, an unknown format), Malformed, Refused (a DOCTYPE or another markup declaration, an
encoding outside the set the parser reads, a prolog longer than 64 KiB, or a prolog that decodes two ways) or LimitExceeded. Keep an else when you match.UrlFetcher's Untrusted policy and a feed may hold 1,000 entries by default (FeedLimits; a value can only be tightened): a larger feed, a podcast with a
long back catalogue for one, fails with BodyTooLarge or LimitExceeded(Entries). https to http redirects are refused by default.id is never blank and never changes for the same entry: the guid or Atom id, else the link as written, else a digest. Links are absolute http/https or null. summary and content are
untrusted markup. Dates are for display; never decide "new" by date.maxNamespaceUris (64) new URIs per document. Provide about 128 MB of heap
headroom for the default two concurrent parses of one fetcher; the figure grows with the sum of every fetcher's maxConcurrentParses in the process.close() returns Closed("feed fetcher"). Closing a fetcher built by FeedFetcher.untrusted closes its transport, so a fetch still on the network returns Closed("transport"), and a
fetch racing close() may return the composed UrlFetcher's Closed("fetcher"): match HttpFailure.Closed. A fetcher over a UrlFetcher you built never closes it, and a parse already running is never
interrupted.RomeFeedParser is the default. A FeedParser of your own is Experimental to implement (@OptIn(ExperimentalForInheritanceWebToolsApi::class)); pass it with parser = … in the factory's lambda.FeedFetcher logs under com.spartanlabs.webtools.feeds.FeedFetcher and prints a feed URL as scheme, host and port only. The UrlFetcher it composes, and the transport under it, print Url.redacted(),
which shows the path: a secret in a feed URL's path appears in their lines (#93).FeedFetcher(UrlFetcher(FakeHttpTransport()…, OutboundUrlPolicy.untrusted())) { parseDispatcher = … }; the pre-check refuses localhost and IP literals, so give the fake a
public-looking host such as https://feeds.example.test/rss. For a real exchange over loopback build the UrlFetcher with LoopbackFetchers.untrusted(…) and pass it to the common factory.MultiConnectionUDPServer multiplexes all client traffic over one socket
(COMMON_LISTEN_PORT, 9998). The handshake bootstrap stays plain UTF-8 text; every datagram
after it is framed — byte 0 is a DatagramType tag the listener switches on before it ever
looks at a payload:
This section summarises the wire; docs/webtools-udp-protocol.md
is the complete, self-contained wire reference (every datagram's byte layout, the as-built
quirks, and the malformed-input handling), and
docs/webtools-udp-architecture.md covers the threads,
locks, and lifecycles that move these bytes.
| Direction | Message | Sent to |
|---|---|---|
| client → server | Iam <name> |
COMMON_LISTEN_PORT (9998) |
| client → server |
Iam <name> <credential> (optional opaque, whitespace-free token) |
COMMON_LISTEN_PORT (9998) |
| server → client |
REGISTERED <version> (e.g. REGISTERED 2) |
the source address and port of the client's Iam datagram |
| server → client |
REFUSED <reason> (handshake refused; bare REFUSED if no reason) |
the source address and port of the client's Iam datagram |
| server → client |
REFUSED malformed (an Iam with a missing or empty name, or a present-but-empty credential; the reason text is informative) |
the source address and port of the malformed Iam datagram |
| client ↔ server |
0x90 <channel> <payload…> — application data (UNRELIABLE) |
port 9998, from/to the same socket the client sent Iam from |
| client ↔ server |
0x80 — keepalive (KEEPALIVE, ~20 s idle cadence) |
port 9998, from the same socket |
| client ↔ server |
0x81 <seq> / 0x82 <seq> — link-quality probe (PROBE_PING / PROBE_PONG; transport-consumed) |
port 9998, same socket |
| client ↔ server |
0xA0 <channel><seq><ack><ackbits><payload> — reliable-ordered data (RELIABLE_DATA) |
port 9998, same socket |
| client ↔ server |
0xA1 <channel><ack><ackbits> — standalone reliable ack (RELIABLE_ACK; no payload) |
port 9998, same socket |
Byte 0 of a post-handshake datagram is always >= 0x80 and the listener never falls back to a
text match, so any payload bytes ride an 0x90 frame intact — see Binary application
payloads. The server's accepted reply carries the wire-protocol
major it speaks (REGISTERED 2 for 2.x); a client on a different major fails handshake()
with a typed IncompatibleProtocolException rather than exchanging framed data the other side
can't parse — see Migrating to 2.0.
Every server → client datagram is addressed to the client's observed post-NAT source, never
to anything in a payload, so the data path traverses NAT and the server binds no
per-client ports (it is hostable behind a single-port container or L4 UDP load balancer). A
retransmitted Iam from the same source is answered with another REGISTERED <version> and
does not re-register. <credential> (tokens[2], optional) is passed to admit verbatim; any token
after it is ignored.
The server splits the line on single spaces, so two spaces in a row make an empty token. An Iam
whose <name> is missing or empty, or whose <credential> slot is present but empty (Iam x,
Iam bob tok), is malformed: the server answers it REFUSED malformed — from any source,
registered or not — without calling admit, registering anything, or firing onClientConnect. A
source that already holds a registration keeps it. A missing credential (Iam bob) is not
malformed. The reason malformed is informative text, not a contract; clients must not branch on
it. The protocol reference has the details, including
the replies an unverified source address can draw.
Before a new handshake is accepted the server calls admit(name, peer, credential) →
Admission; returning Admission.Refused(reason) replies REFUSED <reason> to the client
(surfaced client-side as a typed HandshakeRefusedException), registers nothing, and fires
no onClientConnect. This replaces the old "call Connection.terminate() after the fact"
workaround for a refuse-at-connect decision. admit is only consulted for a first well-formed
Iam from an unknown origin (never for a malformed line, which the server refuses itself), and
runs before any same-name supersede, so a refused newcomer cannot evict a legitimate incumbent.
A fresh, admitted Iam under a name that is already registered under a different
origin (e.g. after a NAT rebind) supersedes that stale registration; the old
origin is no longer addressed by pushToAll. Connection.terminate() fully
deregisters the connection, not just its message handler - e.g. an
application-level refusal discovered after the handshake already completed
should call it so the refused client is not addressed by future broadcasts.
The client must send an 0x80 keepalive datagram on that socket every ~20 s of idle time to
hold its NAT mapping open. The client may either call MultiConnectionUDPClient.sendKeepAlive()
on its own timer, or call MultiConnectionUDPClient.startKeepAlive() once and let the library
time it (idle-aware — application traffic defers the next keepalive). Connection.keepAlive()
is the server-side one-shot equivalent and Connection.startKeepAlive() its library-timed form.
The server drops an inbound 0x80 without dispatching it, though when idle detection is enabled
an inbound 0x80 — like any inbound datagram — refreshes that client's server-side liveness
timestamp. A server-side keepalive is server → client and refreshes cone NATs only — the
authoritative keepalive is the client's. See Scheduled keepalive.
A server behind symmetric NAT still needs a rendezvous/relay (out of scope). Background:
issue #1,
docs/plans/1-nat-traversal/architecture.md,
docs/plans/1-nat-traversal/plan-tier-2.md.
Post-handshake application data is not limited to text. connection.channel(DeliveryMode.UNRELIABLE).send(bytes)
(server → client) / client.channel(DeliveryMode.UNRELIABLE).send(bytes) (client → server) —
or the older Connection.push(ByteArray) / MultiConnectionUDPClient.send(ByteArray) these are
implemented over — and MultiConnectionUDPServer.pushToAll(ByteArray) frame the payload as an
0x90 unreliable datagram and put it on the wire verbatim — no encoding, no trim, no
reserved first byte. channel(DeliveryMode.UNRELIABLE).actuateBytes(handler) (or the older
Connection.actuateBytes / MultiConnectionUDPClient.startBytes / MultiConnectionUDPServer.startBytes)
registers an inbound handler that receives an exact-length, undecoded, untrimmed copy of the
payload (the frame's 0x90 tag and channel byte already stripped). The text and bytes handlers
are mutually exclusive per connection per mode — the last actuate / actuateBytes call on
that UdpChannel wins; a reliable-ordered handler bound via channel(DeliveryMode.RELIABLE_ORDERED)
is independent of both — see Reliable-ordered channel.
Because control traffic (Iam, keepalive, probe) and application data are separated by the
DatagramType tag rather than by matching text, any payload bytes survive intact — a
binary blob that happens to decode-and-trim to KA or that starts with 0x81 is delivered to
start / actuate exactly as sent, with no lead-byte workaround needed.
Every receive path in this module — MultiConnectionUDPServer, MultiConnectionUDPClient,
and UDPSendReceiveServer — accepts datagrams up to 65507 bytes (the maximum UDP
payload over IPv4) whole; a larger payload is truncated, not rejected. Each constructor
takes an optional receiveBufferBytes (512..65507, default 65507) to shrink the
per-instance receive buffer when datagrams are known to be small; a datagram larger than
the configured size is truncated and a WARN is logged. For real-network use keep frames
under the path MTU (~1200 bytes) to avoid IP fragmentation and the loss that comes with
it — application payloads carry 2 bytes of framing overhead (0x90 + channel).
By default the server tracks no per-connection idle time and never signals that a client has
gone silent (a crashed / powered-off / out-of-range client stays addressable until an
explicit Connection.terminate() or a same-name supersede). Pass a positive
idleTimeoutMillis to opt in: a dedicated mcups-liveness daemon thread then reports any
connection with no inbound datagram (application data or a keepalive) within the threshold via
onClientDisconnect(connection, reason):
val server = object : MultiConnectionUDPServer(idleTimeoutMillis = 60_000) {
override fun onClientConnect(connection: Connection) { /* ... */ }
override fun onClientDisconnect(connection: Connection, reason: DisconnectReason) {
when (reason) {
DisconnectReason.TIMEOUT -> startReconnectGraceWindow(connection) // do NOT terminate
DisconnectReason.SUPERSEDED -> rebindSession(connection)
DisconnectReason.TERMINATED -> releaseSession(connection)
}
}
}onClientDisconnect runs on the dispatch thread (mcups-dispatch), not the caller's. It is
notify-only: TIMEOUT does not remove the registration or call terminate() — the
connection stays addressable by pushToAll so the application can run its own grace window
and call terminate() when ready (which then produces a second TERMINATED call). A
same-name supersede fires SUPERSEDED. stop() teardown fires nothing. With the default
idleTimeoutMillis = 0 there is no extra thread and no per-datagram cost.
By default the server accepts every well-formed Iam: one whose name is non-empty and whose
credential slot, if present, is non-empty. The server itself refuses any other Iam with
REFUSED malformed before admit is consulted, so admit never sees an empty name. Override
admit(name, peer, credential): Admission to screen a handshake before it is registered —
validate the credential, enforce a capacity gate, reject a banned name:
val server = object : MultiConnectionUDPServer() {
override fun admit(name: String, peer: InetSocketAddress, credential: String): Admission =
when {
isBanned(name) -> Admission.Refused("banned")
!tokenValid(name, credential) -> Admission.Refused("invalid credential")
connections.size >= capacity -> Admission.Refused("server full")
else -> Admission.Admitted
}
override fun onClientConnect(connection: Connection) { /* ... */ }
}val client = MultiConnectionUDPClient(serverAddress)
client.handshake("alice", credential = base64UrlToken).fold(
onSuccess = { client.channel(DeliveryMode.UNRELIABLE).actuate { msg -> /* ... */ } },
onFailure = { ex -> if (ex is HandshakeRefusedException) showError(ex.reason) },
)The credential is one whitespace-free token, opaque to the library (base64url-encode a
structured or binary value), passed to admit verbatim; an absent credential arrives as the
empty string (a present-but-empty slot is malformed and never reaches admit). It rides in cleartext — the library gives you a channel, not
confidentiality. Back-compat is total in every direction: an old client (Iam <name>)
against a new server registers with an empty credential and the default admit; a new
client (Iam <name> <cred>) against an old server still registers (the old server ignores
the trailing token — so auth is not enforced until both ends are on 1.4.0+). admit
runs inline on the listener thread, so keep it fast and local.
Both sides have an opt-in, idle-aware library-managed keepalive so a consumer never
hand-rolls the ~20 s idle timer around the one-shot primitive:
// client
client.startKeepAlive() // default 20 s; or startKeepAlive(intervalMillis)
client.stopKeepAlive() // also done by client.stop()
// server, per connection
override fun onClientConnect(connection: Connection) {
connection.startKeepAlive() // server → client keepalive; also stopped by terminate() / stop()
}It is idle-aware: an 0x80 keepalive goes out only after intervalMillis of output silence
on that path, so application traffic defers it (a scheduled keepalive may lag up to one
quarter-interval, max 5 s, past the interval). Each side lazily creates one daemon
ScheduledExecutorService (mcupc-keepalive / mcups-keepalive) on the first startKeepAlive
call and shuts it in stop() — a consumer that never opts in pays for no thread. The one-shot
sendKeepAlive() / keepAlive() primitives are unchanged and still own no timer; both emit the
same 0x80 datagram whether sent one-shot or scheduled. Server → client keepalives refresh
only endpoint-independent (cone) NAT mappings; the authoritative keepalive is still the
client's own.
An opt-in transport-level probe on either side measures round-trip time and packet loss without the consumer hand-rolling an application ping. Off by default:
// client
client.startProbe() // default 1 s; or startProbe(intervalMillis), >= 250 ms
client.stopProbe() // also done by client.stop()
val q = client.linkQuality() // null until the first probe reply comes back
// q.rttMillis, q.rttVarianceMillis (jitter proxy), q.packetLossRatio, q.probesSent, q.probesDelivered
// server, per connection
override fun onClientConnect(connection: Connection) {
connection.startProbe() // server → client probe; also stopped by terminate() / stop()
}Each armed side sends one tiny 0x81 probe (an 8-byte big-endian sequence) per interval and
the peer echoes it back as 0x82; the RTT is computed against the prober's own monotonic clock
(EWMA-smoothed, RFC 6298 SRTT/RTTVAR). intervalMillis must be at least
TransportWireFormat.MIN_PROBE_INTERVAL_MILLIS (250 ms); a lower value fails the call rather
than being silently rounded up. The responder always answers an inbound 0x81 (a
client unconditionally, a server for a registered origin), so the peer can measure even if this
side never opted in. 0x81 / 0x82 are consumed by the transport and never reach start /
actuate. Each side lazily creates one daemon ScheduledExecutorService
(mcupc-probe / mcups-probe) on the first startProbe and shuts it in stop().
packetLossRatio is lagging and windowed (it needs a few seconds to first resolve and
reacts over tens of seconds) — a link-health gauge, not a fast congestion trigger. The probe
requires both ends on 1.6.0+ (and, on the framed 2.x transport, both ends on 2.0.0+);
against an incompatible peer no 0x82 comes back and packetLossRatio climbs toward 1.0.
Both sides can open a second delivery plane alongside the default unreliable one — acked, retransmitted, delivered exactly once, in send order:
// either side
val reliable = connection.channel(DeliveryMode.RELIABLE_ORDERED) // or client.channel(...)
reliable.actuate { message -> /* decoded, trimmed text, delivered once, in order */ }
reliable.send("ability-cast").fold(
onSuccess = { /* accepted for reliable delivery - buffered and sequenced, not "on the wire" yet */ },
onFailure = { ex ->
when (ex) {
is ReliableWindowFullException -> Unit // routine backpressure - retry, coalesce, or drop
is ReliableMessageTooLargeException -> Unit // programming/config mistake - split or raise the cap
}
},
)Result.success from a reliable send means accepted for reliable delivery — buffered,
sequenced, and retransmitted until acked — never "already on the wire". A full in-flight window
fails immediately with ReliableWindowFullException (there is no internal absorb queue); an
oversize message fails with ReliableMessageTooLargeException — v1 does not fragment. Both
extend the open ReliableSendFailure, so a caller can is-check one type or catch the
supertype. The default cap is 1024 bytes, configurable per server/client instance up to a
hard 8192-byte maximum (reliableMaxMessageBytes constructor parameter).
The reliable and unreliable planes have independent sequence spaces and independent handlers —
binding one via channel(...).actuate* never disturbs the other — but delivery for both shares
the same per-connection dispatch executor, so head-of-line blocking is inherent: a missing
reliable message stalls delivery of every reliable message sent after it, and also delays
unreliable delivery to that same connection. Use the reliable channel for discrete events
(chat, ability casts, inventory changes) and the unreliable one for frequent state snapshots
where a stale or dropped update is harmless.
The channel is created lazily. On the client, the first reliable send or the first inbound
0xA0/0xA1 on channel 0x00 from the server creates it — obtaining the handle via
channel(RELIABLE_ORDERED) or binding its handler alone does not. On the server, the first
reliable send to a peer, the first startReliable/actuate* binding on that peer's channel,
or the first inbound reliable datagram on channel 0x00 from a registered peer creates it. (A
datagram on any other channel is dropped with a WARN and creates nothing.) Either way it arms one
shared mcups-retransmit /
mcupc-retransmit daemon thread per side (created only once a reliable channel is actually used
on that side). See docs/webtools-udp-architecture.md for
the complete creation/teardown picture. A terminate(), a same-name supersede, or stop() on
either side discards any un-acked data and resets the channel; a send after teardown fails its
Result. Both ends must be on webtools-udp 2.0.0+ — a pre-2.0.0 peer never emits or answers
0xA0/0xA1.
Limits in 2.0. Stated loudly because it is the biggest limitation of this feature: there
is no congestion control — no cwnd, no pacing, no loss-triggered rate reduction. Under
sustained loss the channel retransmits at RTO-backoff cadence up to the in-flight window, then
fails sends; this is adequate for a low-rate discrete-event stream and inadequate for a
genuinely congested path. In addition: no fragmentation — an oversize message fails
immediately with ReliableMessageTooLargeException rather than being split; one reliable
channel per connection — there is no channel id a consumer can choose; and no session
resume — a terminate(), a same-name supersede (NAT rebind), or stop() discards all
in-flight and buffered reliable data and resets the channel, and a fresh registration always
gets a fresh channel. See docs/webtools-udp-protocol.md for
the wire-level detail behind each of these.
Server-wide conveniences mirror the unreliable ones: MultiConnectionUDPServer.startReliable /
pushToAllReliable. Unlike pushToAll, pushToAllReliable attempts every peer even if an
earlier one's window is full, reporting only the first failure — a single busy peer must not
stop a broadcast reaching everyone else.
MultiConnectionUDPClient performs the handshake and then owns a background
listener thread and dispatch thread for the rest of the session, so a consumer
never hand-rolls the receive loop or the "one socket for everything" invariant:
val client = MultiConnectionUDPClient(serverAddress)
client.handshake("alice").getOrThrow()
val unreliable = client.channel(DeliveryMode.UNRELIABLE)
unreliable.actuate { message -> /* handle inbound data; runs on client's dispatch thread */ }
unreliable.send("hello")
// client.start { ... } / client.send("hello") - the pre-2.0-channel spelling, deprecated but
// still fully functional; channel(UNRELIABLE) is the recommended replacement (see Migrating to 2.0)
client.startKeepAlive() // library times it; no caller timer needed
// client.sendKeepAlive() is still available for callers who want to drive the cadence themselves
// or, for a binary protocol:
unreliable.actuateBytes { bytes -> /* exact datagram body, no decode, no trim */ }
unreliable.send(byteArrayOf(0x01, 0x02, 0x03)) // any bytes, no reserved first byte needed
// ...
client.stop()Inbound keepalive datagrams (a server may send one via Connection.keepAlive()) are dropped
automatically and never reach the actuate callback.
webtools-udp 2.0.0 breaks the post-handshake wire format: every datagram after the
handshake now leads with a 1-byte DatagramType tag (§ UDP transport
protocol). Both ends of a connection must be on 2.0.0+ — a
1.x peer's unframed keepalive/probe/application datagrams are dropped as unrecognized, and a
2.x client against a 1.x server fails handshake() with a typed
IncompatibleProtocolException rather than exchanging data neither side can parse correctly.
HandshakeWireFormat: KEEPALIVE_TOKEN, DEFAULT_KEEPALIVE_INTERVAL_MILLIS,
isKeepAlive, PROBE_REQUEST_VERB, PROBE_REPLY_VERB, DEFAULT_PROBE_INTERVAL_MILLIS,
probeRequestMessage, probeReplyMessage, isProbeRequest, isProbeReply, probeToken.
Their role moves to TransportWireFormat in binary form.HandshakeWireFormat.isRegistered now requires the exact REGISTERED 2 reply, not a
bare REGISTERED; registeredMessage() / registeredProtocolVersion(reply) are new.DatagramType tag, there is no longer a reserved first byte to avoid —
any binary payload, including one that used to collide with KA/PING/PONG, rides an
0x90 frame intact. See Binary application payloads.reliableMaxMessageBytes on both MultiConnectionUDPServer and
MultiConnectionUDPClient: the reliable-channel payload cap, default 1024 bytes, validated to
1..8192. An oversize reliable send fails with ReliableMessageTooLargeException.startProbe
(on Connection and on MultiConnectionUDPClient) now fails with IllegalArgumentException
for an intervalMillis below TransportWireFormat.MIN_PROBE_INTERVAL_MILLIS (250 ms) instead
of silently running at 250 ms, and sends exactly one probe per interval — 1.6.0 probed up to
four times too often (exactly 4× at the 1 s default), so probesSent and packetLossRatio
differ from 1.6.0 at the same settings. linkQuality() now counts a probe as lost once it
is overdue as of the call.MultiConnectionUDPClient.start / startBytes call now rebinds the handler
instead of starting a second listener: the last call to start, startBytes, or
channel(DeliveryMode.UNRELIABLE).actuate* wins, and only the first of them starts the
listener thread.Deprecations in 2.0. The public reliable channel API (connection.channel(mode) /
client.channel(mode), see Reliable-ordered channel) is additive,
but it also gives every per-Connection primitive a channel-shaped replacement, so eight
existing members now carry @Deprecated(WARNING) with a working ReplaceWith quick-fix — four
on Connection (push(String), push(ByteArray), actuate, actuateBytes) and four on
MultiConnectionUDPClient (send(String), send(ByteArray), start, startBytes). All eight
remain fully functional in 2.0.0 — this is a warning, not a break — and are slated to
leave the public surface at a future 3.0.0 (demoted, not deleted: the implementations survive
internally). MultiConnectionUDPServer.start / startBytes / pushToAll are not
deprecated and are not going anywhere: channel(...) is per-Connection, so there is no
server-wide replacement to point a ReplaceWith at.
The repo is a Gradle multi-module build; the modules are :webtools-udp, :webtools-scraping,
:webtools-browser, :webtools-http (Kotlin Multiplatform), :webtools-http-commoncheck
(unpublished), :webtools-feeds (Kotlin Multiplatform) and :webtools-feeds-commoncheck (unpublished). The configuration they share lives in the included build-logic/ build (see below).
./gradlew build # compile + full test suite, every module
./gradlew :webtools-udp:build # one module only
./gradlew test # every test set, one JVM per module
./gradlew gatingTest # gating - fast pre-commit checks (all modules)
./gradlew componentTest # component - isolated component behaviour
./gradlew integrationTest # integration - real sockets / external interfaces
./gradlew deterministicTest # deterministic - pure input->output mappings
./gradlew e2eTest # e2e - full-stack flows
./gradlew nonfunctionalTest # nonfunctional - robustness / security properties
./gradlew uatTest # uat - manual acceptance (mostly @Disabled)
./gradlew verifyJavaFloor # the Java 11 floor of every published module's jars, metadata and runtime classpath (part of check)
./gradlew :webtools-http-commoncheck:build # compile webtools-http's commonMain for js (no Node.js needed)
./gradlew :webtools-feeds-commoncheck:build # compile webtools-feeds' commonMain for js (no Node.js needed)
./gradlew :build-logic:test # build-logic's pre-commit gate: its gating, component and deterministic checks
./gradlew :build-logic:fixturesPublicationCheck # publish a copy of webtools-http to a temporary repository and resolve its test fixtures from it (also part of webtools-http's check)
./gradlew :build-logic:integrationTest :build-logic:e2eTest :build-logic:nonfunctionalTest # on demand: TestKit checks
./gradlew dokkaGeneratePublicationHtml # the API reference of the five published modules, aggregated in build/dokka/html; fails on a broken KDoc link or an undocumented public declaration (the CI "docs" job runs it)Each published module's javadoc jar is its Dokka output, and the root project aggregates the five modules into one
reference. The theme is docs/dokka/styles/dokka-styles.css, with the logo in docs/brand/, and webtools-http's test
fixtures are part of the reference. The Dokka versions in settings.gradle.kts (pluginManagement.plugins) and
build-logic/build.gradle.kts must agree.
./gradlew build and check also compile every *-commoncheck module for js: an unpublished module
that compiles a multiplatform module's commonMain so a JVM API in common code is a compile error. No
Node.js is needed.
Each module organises its tests by the project's named test sets, in the package
com.spartanlabs.testing.<set>.webtools.<name>, where <name> is the module's name without webtools-
(udp, http, feeds, …): a JVM module under <module>/src/test/kotlin/, a multiplatform module under
<module>/src/commonTest/kotlin/ and <module>/src/jvmTest/kotlin/, both of which run on the JVM target. A
set's task selects its tests by package (com.spartanlabs.testing.<set>.*) from the module's JVM test
compilation, so any set can be run or gated on its own; a test class outside those packages runs under
test only. Shared test helpers live under testing/support/ and are not published. The published test
fixtures of webtools-http are a separate source set, webtools-http/src/jvmTestFixtures (package
com.spartanlabs.webtools.http.testing), whose own tests sit in com.spartanlabs.testing.<set>.webtools.http.testing;
webtools-feeds' tests use them through testFixtures(project(":webtools-http")).
Every JVM module compiles to Java 11 bytecode against the Java 11 API, which is what the install
section's Requires JDK 11 or newer rests on. Building needs a JDK 17 to 26, the range Gradle
9.7.1 runs on, and the tests of every set run on that JDK too; no JDK 11 is involved. The convention
plugins set jvmTarget, -Xjdk-release and the Java compatibility from one constant,
WebToolsBuild.JAVA_FLOOR, so the result does not depend on which of those JDKs builds.
verifyJavaFloor, part of check, fails the build when a built jar holds a class above Java 11
bytecode, when a library variant of the generated Gradle Module Metadata declares another
org.gradle.jvm.version, or when a runtime dependency holds such a class; on success it writes a
one-line summary to <module>/build/reports/java-floor/verifyJavaFloor.txt. Running
./gradlew verifyJavaFloor checks every library module without running any module's tests.
build-logic/ is a Gradle build included through pluginManagement { includeBuild("build-logic") }
in settings.gradle.kts. It holds the build configuration every module shares as four convention
plugins, which a module applies by id in its plugins {} block:
| Plugin id | Configures |
|---|---|
webtools.published-library |
the Maven Central publishing skeleton: signing, the shared POM metadata and Dokka (theme, source links, failOnWarning, reportUndocumented) |
webtools.kotlin-jvm-library |
a Kotlin/JVM library: the Java floor and verifyJavaFloor, test dependencies, JUnit Platform, the test tasks of each set, javadoc jar from Dokka; applies webtools.published-library
|
webtools.kotlin-multiplatform-library |
a Kotlin Multiplatform library with a jvm() target (used by webtools-http and webtools-feeds): the Java floor, the TargetJvmVersion attribute and verifyJavaFloor, javadoc jar from Dokka; applies binary-compatibility-validator — commit api/<module>.api — and webtools.published-library
|
webtools.kotlin-common-check |
an unpublished, check-only js module that compiles a sibling's commonMain (no publishing, no tests, no Node.js, no Java floor) |
Change shared build behaviour there, not in a module's build file: a module's build.gradle.kts
adds its dependencies, its coordinates, its POM name and description, and any configuration that
belongs to it alone (for example webtools-udp's version and its UDP-port lock, or webtools-http's
test-fixtures variant).
build-logic's own tests use the same set packages, and each of its sets with tests has a task
of the same name (./gradlew :build-logic:gatingTest, …). ./gradlew build runs none of them, with one
exception: webtools-http's check (so build, check and :webtools-http:check) runs
:build-logic:fixturesPublicationCheck, which publishes a copy of the module with a throwaway signing key
to a temporary repository and resolves its test fixtures from three build shapes. It takes about a minute
when its inputs changed and is UP-TO-DATE otherwise; its first run needs the network. It is wired only
when WebTools is the root build, so a build that includes WebTools never runs it.
:build-logic:test runs the sets that start no nested build, in seconds.
:build-logic:integrationTest, :build-logic:e2eTest and :build-logic:nonfunctionalTest run
nested Gradle builds against generated projects; their first run downloads the Kotlin compiler and
the test dependencies into build-logic/build/testkit/home, so it needs the network.
A small Kotlin/JVM collection of internet I/O helpers, published as independent, single-concern artifacts — depend only on the one you need:
| Artifact | Package | Purpose | Third-party runtime deps |
|---|---|---|---|
io.github.spartanlaboratories:webtools-udp |
com.spartanlabs.webtools.udp |
Multi-client UDP connection layer + NAT-traversal handshake | none (slf4j-api only) |
io.github.spartanlaboratories:webtools-scraping |
com.spartanlabs.webtools.scraping |
Connector — open/read a URL line by line, plus one-shot get / skrape / image download
|
skrapeit, jsoup, unirest |
io.github.spartanlaboratories:webtools-browser |
com.spartanlabs.webtools.browser |
WebViewer — headless-browser screenshots |
selenium-java |
io.github.spartanlaboratories:webtools-http |
com.spartanlabs.webtools.http |
Kotlin Multiplatform (JVM target) — request/response types, a Url that reads like OkHttp's, a typed failure model, the transport and wall-clock ports of a suspend HTTP client stack, OkHttpTransport (the supplied transport: one request per hop, manual redirects, a capped decoded body, never a retry), the ResponseCache port with CaffeineResponseCache (the supplied cache: one load per key at a time, per-entry lifetimes, a weight bound, a per-call maxAge, failures never stored) and the pass-through ResponseCache.Disabled, the CallBudget port with WindowedCallBudget (the supplied budget: epoch-aligned rate windows with a boundary guard, per-lane floors, first-come-first-served waits that fail fast or are refunded and re-planned, pauses from Retry-After and reset hints, and a hard window that survives a restart through a BudgetStore), CallBudget.Unlimited, RateSignalsReader.Standard and RetryAfter.parse, and the Trusted/Untrusted outbound-URL policies with their transport settings (timeouts, size and redirect caps, concurrency), and the call facades UpstreamClient (declared Endpoints, a cache, a budget and a transport composed in one fixed order, with an owning JVM factory) and UrlFetcher (administrator-supplied URLs checked against the policy first) |
okhttp-jvm, caffeine, kotlinx-coroutines-core, slf4j-api |
io.github.spartanlaboratories:webtools-feeds |
com.spartanlabs.webtools.feeds |
Kotlin Multiplatform (JVM target) — RSS 0.90 to 2.0, RSS 1.0 and Atom 0.3 and 1.0 behind a hardened parser port (FeedParser) with Rome 2.1.0 as the supplied parser (RomeFeedParser: DOCTYPE refused before any parser runs, structural limits on depth, attributes, elements and namespaces, text/id/link/entry bounds), a feed model with a frozen entry id, and FeedFetcher, a stateless conditional GET over UrlFetcher that returns a 304 as a value and a permanent move as a fact |
webtools-http (and its dependencies), rome |
Every fallible operation returns kotlin.Result rather than throwing, so callers decide how
to recover. webtools-http and webtools-feeds are suspend APIs: every call that sends a request
or may wait (UpstreamClient.call, UrlFetcher.fetch, FeedFetcher.fetch, HttpTransport.execute,
ResponseCache.getOrLoad, CallBudget.acquire, flush, utilisation) suspends, the supplied
implementations never block the caller's thread, and there is no blocking variant. Building a component and close() do not suspend.
State boards: docs/dashboards/ shows the modules as built at a named commit, with planned systems drawn dashed.
Work boards: the repository work board shows every planning and development run, and each issue's board under docs/plans/ traces every part of a change through its gap, the options considered, the verdict and the as-built state, with the work in progress.
API reference and project site: https://spartanlaboratories.github.io/WebTools/ (landing page) and https://spartanlaboratories.github.io/WebTools/api/ (the Dokka reference of the five published modules).
dependencies {
implementation("io.github.spartanlaboratories:webtools-udp:2.0.1")
// and/or
implementation("io.github.spartanlaboratories:webtools-scraping:1.0.0")
implementation("io.github.spartanlaboratories:webtools-browser:1.0.0")
implementation("io.github.spartanlaboratories:webtools-http:1.0.0")
// tests only (Gradle): fakes, a virtual clock, .http fixtures and a recorder
testImplementation(testFixtures("io.github.spartanlaboratories:webtools-http:1.0.0"))
implementation("io.github.spartanlaboratories:webtools-feeds:1.0.0")
}Requires JDK 11 or newer (webtools-scraping and webtools-browser 1.0.0: JDK 23 or newer). Built with Kotlin 2.2.
webtools-http is a Kotlin Multiplatform module that publishes webtools-http and webtools-http-jvm;
a Gradle build resolves the right one itself, a Maven build depends on webtools-http-jvm. Its okhttp-jvm
5.5.0 dependency requires kotlin-stdlib 2.2.21, so a consumer's Kotlin standard library resolves to 2.2.21 or newer.
webtools-feeds is a Kotlin Multiplatform module that publishes webtools-feeds and webtools-feeds-jvm (a Maven build depends on webtools-feeds-jvm). It exposes
webtools-http as an api dependency and brings Rome 2.1.0 (with rome-utils and jdom2) at runtime; the fakes for its tests are webtools-http's test fixtures.
The test-fixtures variant of webtools-http (capability
io.github.spartanlaboratories:webtools-http-test-fixtures, a -test-fixtures.jar beside the module, JVM
only) holds the package com.spartanlabs.webtools.http.testing: VirtualClock, FakeHttpTransport with
RequestMatch, Step and Unmatched, Fixture and FixtureLoader for .http fixture files,
FixtureRecorder with RecordingGate and Redaction, and LoopbackTransports (the real OkHttpTransport
over a loopback mockwebserver3 server, with the policy's rules live), MockWebServer.webtoolsUrl,
Fixture.toMockResponse and MockWebServer.enqueue(fixture), LoopbackFetchers (a UrlFetcher over a loopback
server with the policy's rules live), and RecordingBudgetObserver and FakeBudgetStore
(a scripted budget store with failure injection and virtual-time delays). testFixtures(...) is Gradle-only: a
Maven build cannot select a variant by capability. The variant adds kotlinx-coroutines-test and mockwebserver3
to the test classpath only. Fixture files live in the consumer's own test resources; add
**/fixtures/**/*.http -text to its .gitattributes so a checkout does not rewrite their line endings.
Recording is off unless WEBTOOLS_HTTP_RECORD has a recognised value, and never runs where a CI variable
(CI, GITHUB_ACTIONS, ...) is set; a build that records declares the variable as an input of its test tasks,
or Gradle skips them as up to date when only the variable changed.
The Java 11 floor starts with webtools-udp 2.0.1: Java 11 bytecode (class-file major 55), compiled
against the Java 11 API, as are webtools-http and webtools-feeds. The earlier releases on Maven
Central — webtools-udp 1.0.0, 1.1.0 and 2.0.0, webtools-scraping 1.0.0 and webtools-browser
1.0.0 — are Java 23 bytecode and fail to load on JDK 22 or older with
UnsupportedClassVersionError. The Java 11 builds of the two archived modules are 1.0.0a in this
repository and have not been published. Building the repository needs a JDK 17 to 26; see Java floor.
webtools-udp 2.0.0 is a wire break for everything after the handshake — see Migrating to
2.0. Both ends of a connection must be on 2.0.0+; a 1.x ↔ 2.x pairing
fails the handshake cleanly instead of exchanging data either side can parse. See
docs/webtools-udp-protocol.md for the complete wire reference
and docs/webtools-udp-architecture.md for the component,
thread, and lifecycle picture.
The combined WebTools artifact is retired at its final version 2.0.1a and receives no
further releases. To move off it:
WebTools dependency with whichever of the three modules you actually use.com.spartanlabs.webtools.Foo → com.spartanlabs.webtools.{udp,scraping,browser}.Foo.WebTools carried an unused api dependency on GeneralTools; if you relied on it
transitively, depend on it directly now.webtools-browser no longer ships webdriver binaries — Selenium
Manager resolves the driver at
runtime (needs a local Chrome/Chromium install and, on first run, network access).| Type | Purpose |
|---|---|
MultiConnectionUDPServer |
Accepts handshakes from many clients on one common port and hands each its own Connection. Abstract — subclass and implement onClientConnect; optionally override admit to screen/refuse handshakes and validate an opaque credential, override onClientDisconnect, and set idleTimeoutMillis for idle-connection detection. |
MultiConnectionUDPClient |
The client-side counterpart: one socket, one owned listener thread, one dispatch thread — performs the handshake, then delivers the rest of the session via callback. channel(mode) is the recommended entry point (see Reliable-ordered channel); the older send / start / startBytes (deprecated, still fully functional — see Migrating to 2.0) also accept/deliver raw ByteArray datagrams via their ByteArray overloads. startKeepAlive / stopKeepAlive run an opt-in, library-timed keepalive; startProbe / stopProbe / linkQuality run an opt-in RTT / packet-loss probe. |
Connection |
Interface for one named connection to a peer. channel(mode) (see below) is the recommended entry point; actuate / actuateBytes / push(String) / push(ByteArray) are deprecated but fully functional (see Migrating to 2.0); terminate / keepAlive / startKeepAlive / stopKeepAlive / startProbe / stopProbe / linkQuality are unchanged control-plane members. |
UDPConnection |
The production Connection: a socket-free handle to one multiplexed client of a MultiConnectionUDPServer; owns no socket. |
DeliveryMode |
Which guarantee a UdpChannel provides: UNRELIABLE (fire-and-forget, the 0x90 frame) or RELIABLE_ORDERED (acked, retransmitted, exactly-once in order, the 0xA0/0xA1 frames — head-of-line blocking is inherent). |
UdpChannel |
A delivery-mode-scoped view of one connection: send(bytes/message) and actuate/actuateBytes to bind the inbound handler for that mode. Returned by Connection.channel(mode) / MultiConnectionUDPClient.channel(mode). See Reliable-ordered channel. |
ReliableWindowFullException / ReliableMessageTooLargeException
|
The two typed reliable-send failures, both subtypes of the open ReliableSendFailure; never thrown, only ever a Result.failure payload. See Reliable-ordered channel. |
LinkQuality |
An immutable snapshot from the opt-in probe: smoothed rttMillis, rttVarianceMillis (jitter proxy), windowed packetLossRatio, and probesSent / probesDelivered. |
UDPSendReceiveServer |
A bound send/receive UDP socket pair with an async receive loop — receives datagrams up to 65507 bytes (configurable via receiveBufferBytes), truncating larger ones. No DeliveryMode / UdpChannel surface — see its KDoc. |
HandshakeWireFormat |
The handshake-bootstrap wire format: Iam <name> [<credential>], the accepted REGISTERED <version> reply, REFUSED <reason>. Everything after the handshake lives in TransportWireFormat. |
DatagramType |
The 1-byte tag carried as byte 0 of every post-REGISTERED datagram (KEEPALIVE 0x80, PROBE_PING 0x81, PROBE_PONG 0x82, UNRELIABLE 0x90, RELIABLE_DATA 0xA0, RELIABLE_ACK 0xA1); the listener switches on it before touching a payload. |
TransportWireFormat |
Builds/parses the framed post-handshake datagrams: keepaliveDatagram, probePingDatagram / probePongDatagram / probeSequenceOf, unreliableDatagram / unreliablePayloadOf / unreliableChannelOf. |
Admission / HandshakeRefusedException
|
admit's return type (Admitted / Refused(reason)); the typed failure a refused client's handshake surfaces. |
IncompatibleProtocolException |
Typed handshake() failure when the server's REGISTERED reply announces a different wire-protocol major than this build speaks (e.g. a 1.x server). |
resolveLocalAddress() |
Best-effort lookup of this machine's outward-facing local address. |
API stability. The whole webtools-udp 2.x public surface — including DeliveryMode,
DatagramType, DisconnectReason, UdpChannel, ReliableSendFailure and its subtypes, and
every other public type — is Stable Core: breaking changes only in a major. DeliveryMode,
DatagramType, and DisconnectReason may each gain a new entry in a minor release; keep an
else branch in any when over one of them. The eight members deprecated in 2.0.0 (see
Migrating to 2.0) stay fully functional — still compiling, deprecated at
WARNING only — for all of 2.x, and leave the public surface only at 3.0.0.
| Type | Purpose |
|---|---|
Connector |
Single-connection web client: open a URL and read it line by line; plus one-shot get, skrape, and image download helpers. |
For suspend calls, timeouts and size caps, an outbound-URL policy, response caching and a call budget, use
webtools-http instead.
| Type | Purpose |
|---|---|
WebViewer |
Headless-Chrome screenshot utility — screenshot(url) returns a Result<BufferedImage>, getPage(url) a Result<File>. Driver provisioned by Selenium Manager. |
| Type | Purpose |
|---|---|
UpstreamClient |
Calls the endpoints of one upstream API from a base URL: the outbound-URL policy's pre-check, then the response cache, then the call budget inside the cache's loader, then the transport. Each call is declared once as an Endpoint; CallSpec carries the per-call path, query, headers, lane and waits. See HTTP client stack. |
UrlFetcher |
Fetches administrator-supplied URLs: the pre-check, the budget and the transport, no cache. Every status is the response except a throttle. |
OutboundUrlPolicy |
Trusted (a known upstream: https only unless cleartext is allowed, optional host pins) or Untrusted (a URL a third party supplied: lenient host forms, reserved names, denied addresses, other ports and sensitive headers refused), each carrying its TransportSettings. See Outbound-URL policies. |
OkHttpTransport |
The supplied HttpTransport: one request per hop, redirects followed by hand and judged hop by hop, a capped decoded body, never a retry. |
CaffeineResponseCache / ResponseCache.Disabled
|
The supplied ResponseCache (one load per key at a time, per-entry lifetimes, a weight bound, failures never stored) and the pass-through that stores nothing. |
WindowedCallBudget / CallBudget.Unlimited
|
The supplied CallBudget (epoch-aligned rate windows, lane floors, fail-fast waits, pauses from the server's signals, a hard window kept across a restart through a BudgetStore) and the budget that grants everything. |
HttpRequest / HttpResponse / HttpHeaders / Url
|
The value types. A header added with addSensitive is treated as a credential (refused under Untrusted, dropped on a cross-origin redirect); Url.redacted() is the form the stack logs. |
HttpFailure |
The failure model carried in Result: TransportFailure (nothing usable came back), UpstreamFailure (RateLimited, UpstreamError), BudgetExhausted, DecodeFailure and Closed. |
ExperimentalForInheritanceWebToolsApi |
The opt-in marker for implementing one of the stack's seams. |
API stability. Everything a consumer calls, passes, configures or matches is Stable Core,
the supplied implementations (OkHttpTransport, CaffeineResponseCache, WindowedCallBudget,
RateSignalsReader.Standard, …) included: none of them needs an opt-in. Implementing, delegating to or
subclassing a seam (HttpTransport, ResponseCache, CallBudget, Permit, RateSignalsReader, BudgetStore,
BudgetObserver, BudgetEvent, WallClock, and the failure groups HttpFailure, TransportFailure
and UpstreamFailure) is Experimental: it needs @OptIn(ExperimentalForInheritanceWebToolsApi::class)
and may change in a minor release. A seam graduates to @SupportedExtension in a feature release once a
consumer has built against it. The failure groups and the reason vocabularies (RefusalReason,
ExhaustedReason, …) may gain members in a feature release; keep an else branch in any when over
one of them. The test fixtures are Stable Core with source compatibility only.
| Type | Purpose |
|---|---|
FeedFetcher |
A stateless conditional GET over UrlFetcher: a 304 is FeedFetch.NotModified, a feed FeedFetch.Fetched. On the JVM, FeedFetcher.untrusted(userAgent) builds and owns an Untrusted UrlFetcher. See Feeds. |
RomeFeedParser / FeedParser
|
The supplied parser over Rome 2.1.0 (a DOCTYPE refused before any parser runs, structural limits) and the port behind it. |
Feed / FeedEntry / FeedEnclosure
|
The model; an entry's id is never blank and never changes for the same entry. |
FeedValidators / FeedLimits
|
The ETag and Last-Modified pair the caller stores between polls; the bounds a parser applies, which can only be tightened. |
FeedFailure |
NotAFeed, Malformed, Refused and LimitExceeded, under HttpFailure. |
API stability. Everything a consumer calls, passes, configures or matches is Stable Core,
RomeFeedParser included. Implementing FeedParser or subclassing FeedFailure is Experimental, under
webtools-http's opt-in marker. FeedFailure may gain leaves, and its reason vocabularies (NotAFeedReason,
FeedRefusal, FeedLimit) may gain members, in a feature release; keep an else branch.
webtools-http composes a response cache, a call budget and a transport behind two facades.
UpstreamClient calls the endpoints of one upstream API; UrlFetcher fetches URLs an administrator
supplies. Both are AutoCloseable, both return kotlin.Result, and neither ever retries: a throttle, a transport
failure and a 5xx are each returned once. docs/webtools-http-architecture.md
describes the stack as built, webtools-feeds included: its components, the call path, where each outbound-URL
check runs, the failure model, the threads, ownership and closing, what is logged, the stability tiers, the test
support and the known limitations.
val client = UpstreamClient(Url.parse("https://api.example.com/v2/").getOrThrow(), "my-app/1.0 (+https://example.com/contact)") {
headers = HttpHeaders { addSensitive("Authorization", "Bearer $token") } // default headers; credentials are never printed
budget(listOf(RateWindow("minute", 60, 1.minutes) { margin = 10 })) // built and owned by the client
}
// Declare each call once: a stable id, a decoder, how long to keep the answer, which status means "nothing there".
val player = Endpoint("player", ::decodePlayer) {
ttl = 10.minutes
misses(404)
}
val result: Result<Lookup<Player>> = client.call(player) {
path("players", id) // raw segments; each is encoded, so "a/b" stays one segment
query("region", "eu")
maxAge = 1.minutes // accept a cached answer no older than this
}
result.fold(
onSuccess = { lookup -> when (lookup) { is Lookup.Found -> show(lookup.value); Lookup.Missing -> showNotFound() } },
onFailure = { failure -> when (failure) {
is UpstreamFailure.RateLimited -> backOff(failure.retryAfter) // wait this long, then call
is HttpFailure.BudgetExhausted -> backOff(failure.wait)
else -> report(failure) // keep an else: new failure types may be added
} },
)
client.flush() // the exact final write of a budget with a persistent store
client.close() // closes what the client built (cache, budget, transport), never what it was handedGET is cached.ttl defaults to zero (in-flight calls coalesce, nothing is stored); missTtl defaults to
the smaller of ttl and 60 seconds; cost defaults to 1 and lane to Lane.Interactive.Found or Missing on success. A 429 (or what the client's rateSignals reader reports as a
throttle) is RateLimited, whose retryAfter is the longer of the server's Retry-After and the time the budget
stays paused; a 2xx the decoder rejects is DecodeFailure; any other status is UpstreamError; the budget's refusal is
BudgetExhausted; the transport's and the policy's are TransportFailure leaves. Failures are never cached. Once a
permit is granted, only an InvalidRequest gives it back: every other failure the transport returns, a redirect or
an address its policy refuses included, leaves it spent.defaultMaxWait (5 seconds; CallSpec.maxWait per call) for a permit, then
fails with BudgetExhausted carrying the wait it would have needed. The worst-case latency of a call is that wait
plus the transport's call timeout.Trusted, pinned to the base URL's host, https only. Pass
policy = OutboundUrlPolicy.trusted { … } to change it. The JVM factory raises OkHttp's concurrency limits to the
budget's burst so a granted permit never queues in OkHttp; pass your own transport to the common factory
UpstreamClient(transport, baseUrl) { … } to size it yourself.cache(CaffeineResponseCache { maxWeight = 5_000 }) and budget(myBudget) are
caller-owned: the client never closes them.com.spartanlabs.webtools.http (UpstreamClient,
UrlFetcher, OkHttpTransport, CaffeineResponseCache, WindowedCallBudget, and GuardedDns at DEBUG), never a header value, a body, a
query string or a decoded value. A URL is printed with Url.redacted(), which hides the query and shows the path,
and an endpoint id is printed as given: keep credentials out of path segments, the base URL's path and endpoint ids
(#93).OutboundUrlPolicy is a closed pair. Trusted, for an upstream you chose, checks the scheme (https only unless
allowCleartext), optionally pins hosts and filters no address. Untrusted, for a URL a third party supplied, also
refuses lenient host forms (127.1, 2130706433), single-label names, reserved names (localhost, internal, …), private, loopback,
link-local and other special-purpose addresses, whether written as a literal or resolved from a name, ports other
than 80, 443, 8080 and 8443, and any request carrying a sensitive header; it never uses a proxy. Its parameters can
never admit a denied address or name (extraDeniedRanges and extraDeniedHostSuffixes add denials; allowedPorts
sets the port list; mixedAnswers chooses between refusing a mixed answer and using its allowed addresses). Each policy
carries its TransportSettings: Trusted 30 s per call, 10 s per socket operation, 5 MiB, 5 redirects, 16
concurrent requests (5 per host); Untrusted 15 s, 5 s, 2 MiB, 3 redirects, 8 (2 per host). Change one with
OutboundUrlPolicy.untrusted { settings { callTimeout = 20.seconds } }.
UrlFetcher("my-app/1.0 (+https://example.com/contact)").use { fetcher ->
fetcher.fetch(HttpRequest(HttpMethod.Get, url)) { lane = Lane.Tracker }.fold(
onSuccess = { response -> handle(response) }, // every status, a 304 included, is a response
onFailure = { failure -> report(failure) }, // RefusedByPolicy, RateLimited, BudgetExhausted, a transport failure …
)
}UrlFetcher uses an Untrusted policy (private, loopback and link-local addresses refused, a short list of ports,
no sensitive header) and has no cache and no decoder. Pass a policy of your own
(UrlFetcher(userAgent, OutboundUrlPolicy.untrusted { extraDeniedRanges += … })) to add your deployment's own denied
ranges, and budget = … in the lambda to share a budget. Over a transport of your own (a FakeHttpTransport in a
test) use the common factory UrlFetcher(transport, policy) { … }: policy is the pre-check's, the transport
enforces its own, and the fetcher never closes the transport.
FakeHttpTransport, VirtualClock and LoopbackFetchers (all in testFixtures) cover the stack without a network:
runTest {
val clock = virtualClock()
val fake = FakeHttpTransport(clock.timeSource).on(RequestMatch.get("/v2/players/7"), Step.respond(200, body = """{"name":"Ada"}"""))
UpstreamClient(fake, baseUrl) { timeSource = clock.timeSource; wallClock = clock.wallClock }.use { client ->
assertEquals(Lookup.Found(Player("Ada")), client.call(player) { path("players", "7") }.getOrThrow())
assertEquals(1, fake.calls.size)
}
}For a real exchange over loopback, start a mockwebserver3.MockWebServer on 127.0.0.1 and use
LoopbackFetchers.untrusted("test") (a UrlFetcher with every Untrusted rule live except the loopback ones) or
LoopbackTransports.trusted(…) with the common UpstreamClient factory.
webtools-feeds parses RSS and Atom behind a hardened port and fetches feeds with a stateless conditional GET over UrlFetcher. It keeps no state: you store the validators it returns and
hand them back on the next poll. docs/webtools-http-architecture.md describes the
fetch path and the hardening layers (§4.5) and the module's known limitations (§13).
FeedFetcher.untrusted("my-app/1.0 (+https://example.com/contact)").use { fetcher ->
val stored: FeedValidators = loadValidators(feedId) // FeedValidators.None on the first poll
fetcher.fetch(feedUrl, stored).fold(
onSuccess = { fetch ->
saveValidators(feedId, fetch.validators) // after every response
fetch.permanentlyMovedTo?.let { moved -> considerMoving(feedId, moved) }
when (fetch) {
is FeedFetch.Fetched -> store(fetch.feed.entries) // new items: entries whose id you have not seen, never by date
is FeedFetch.NotModified -> Unit
}
},
onFailure = { failure -> report(failure) },
)
}finalUrl changes (a validator from one server must not reach another); deciding whether to adopt permanentlyMovedTo (it is a fact
about the redirect chain: adopt it only when the chain's first request was https, or after it was reported on several polls); polling, back-off, 410, the first poll and "new items".Result: the stack's own (RefusedByPolicy, RateLimited, BudgetExhausted, transport failures, UpstreamError for any status that is not a feed, 304 to an unconditional request
included) pass through unchanged; a document that is not a feed is FeedFailure.NotAFeed (a web page, an empty or non-XML body, an unknown format), Malformed, Refused (a DOCTYPE or another markup declaration, an
encoding outside the set the parser reads, a prolog longer than 64 KiB, or a prolog that decodes two ways) or LimitExceeded. Keep an else when you match.UrlFetcher's Untrusted policy and a feed may hold 1,000 entries by default (FeedLimits; a value can only be tightened): a larger feed, a podcast with a
long back catalogue for one, fails with BodyTooLarge or LimitExceeded(Entries). https to http redirects are refused by default.id is never blank and never changes for the same entry: the guid or Atom id, else the link as written, else a digest. Links are absolute http/https or null. summary and content are
untrusted markup. Dates are for display; never decide "new" by date.maxNamespaceUris (64) new URIs per document. Provide about 128 MB of heap
headroom for the default two concurrent parses of one fetcher; the figure grows with the sum of every fetcher's maxConcurrentParses in the process.close() returns Closed("feed fetcher"). Closing a fetcher built by FeedFetcher.untrusted closes its transport, so a fetch still on the network returns Closed("transport"), and a
fetch racing close() may return the composed UrlFetcher's Closed("fetcher"): match HttpFailure.Closed. A fetcher over a UrlFetcher you built never closes it, and a parse already running is never
interrupted.RomeFeedParser is the default. A FeedParser of your own is Experimental to implement (@OptIn(ExperimentalForInheritanceWebToolsApi::class)); pass it with parser = … in the factory's lambda.FeedFetcher logs under com.spartanlabs.webtools.feeds.FeedFetcher and prints a feed URL as scheme, host and port only. The UrlFetcher it composes, and the transport under it, print Url.redacted(),
which shows the path: a secret in a feed URL's path appears in their lines (#93).FeedFetcher(UrlFetcher(FakeHttpTransport()…, OutboundUrlPolicy.untrusted())) { parseDispatcher = … }; the pre-check refuses localhost and IP literals, so give the fake a
public-looking host such as https://feeds.example.test/rss. For a real exchange over loopback build the UrlFetcher with LoopbackFetchers.untrusted(…) and pass it to the common factory.MultiConnectionUDPServer multiplexes all client traffic over one socket
(COMMON_LISTEN_PORT, 9998). The handshake bootstrap stays plain UTF-8 text; every datagram
after it is framed — byte 0 is a DatagramType tag the listener switches on before it ever
looks at a payload:
This section summarises the wire; docs/webtools-udp-protocol.md
is the complete, self-contained wire reference (every datagram's byte layout, the as-built
quirks, and the malformed-input handling), and
docs/webtools-udp-architecture.md covers the threads,
locks, and lifecycles that move these bytes.
| Direction | Message | Sent to |
|---|---|---|
| client → server | Iam <name> |
COMMON_LISTEN_PORT (9998) |
| client → server |
Iam <name> <credential> (optional opaque, whitespace-free token) |
COMMON_LISTEN_PORT (9998) |
| server → client |
REGISTERED <version> (e.g. REGISTERED 2) |
the source address and port of the client's Iam datagram |
| server → client |
REFUSED <reason> (handshake refused; bare REFUSED if no reason) |
the source address and port of the client's Iam datagram |
| server → client |
REFUSED malformed (an Iam with a missing or empty name, or a present-but-empty credential; the reason text is informative) |
the source address and port of the malformed Iam datagram |
| client ↔ server |
0x90 <channel> <payload…> — application data (UNRELIABLE) |
port 9998, from/to the same socket the client sent Iam from |
| client ↔ server |
0x80 — keepalive (KEEPALIVE, ~20 s idle cadence) |
port 9998, from the same socket |
| client ↔ server |
0x81 <seq> / 0x82 <seq> — link-quality probe (PROBE_PING / PROBE_PONG; transport-consumed) |
port 9998, same socket |
| client ↔ server |
0xA0 <channel><seq><ack><ackbits><payload> — reliable-ordered data (RELIABLE_DATA) |
port 9998, same socket |
| client ↔ server |
0xA1 <channel><ack><ackbits> — standalone reliable ack (RELIABLE_ACK; no payload) |
port 9998, same socket |
Byte 0 of a post-handshake datagram is always >= 0x80 and the listener never falls back to a
text match, so any payload bytes ride an 0x90 frame intact — see Binary application
payloads. The server's accepted reply carries the wire-protocol
major it speaks (REGISTERED 2 for 2.x); a client on a different major fails handshake()
with a typed IncompatibleProtocolException rather than exchanging framed data the other side
can't parse — see Migrating to 2.0.
Every server → client datagram is addressed to the client's observed post-NAT source, never
to anything in a payload, so the data path traverses NAT and the server binds no
per-client ports (it is hostable behind a single-port container or L4 UDP load balancer). A
retransmitted Iam from the same source is answered with another REGISTERED <version> and
does not re-register. <credential> (tokens[2], optional) is passed to admit verbatim; any token
after it is ignored.
The server splits the line on single spaces, so two spaces in a row make an empty token. An Iam
whose <name> is missing or empty, or whose <credential> slot is present but empty (Iam x,
Iam bob tok), is malformed: the server answers it REFUSED malformed — from any source,
registered or not — without calling admit, registering anything, or firing onClientConnect. A
source that already holds a registration keeps it. A missing credential (Iam bob) is not
malformed. The reason malformed is informative text, not a contract; clients must not branch on
it. The protocol reference has the details, including
the replies an unverified source address can draw.
Before a new handshake is accepted the server calls admit(name, peer, credential) →
Admission; returning Admission.Refused(reason) replies REFUSED <reason> to the client
(surfaced client-side as a typed HandshakeRefusedException), registers nothing, and fires
no onClientConnect. This replaces the old "call Connection.terminate() after the fact"
workaround for a refuse-at-connect decision. admit is only consulted for a first well-formed
Iam from an unknown origin (never for a malformed line, which the server refuses itself), and
runs before any same-name supersede, so a refused newcomer cannot evict a legitimate incumbent.
A fresh, admitted Iam under a name that is already registered under a different
origin (e.g. after a NAT rebind) supersedes that stale registration; the old
origin is no longer addressed by pushToAll. Connection.terminate() fully
deregisters the connection, not just its message handler - e.g. an
application-level refusal discovered after the handshake already completed
should call it so the refused client is not addressed by future broadcasts.
The client must send an 0x80 keepalive datagram on that socket every ~20 s of idle time to
hold its NAT mapping open. The client may either call MultiConnectionUDPClient.sendKeepAlive()
on its own timer, or call MultiConnectionUDPClient.startKeepAlive() once and let the library
time it (idle-aware — application traffic defers the next keepalive). Connection.keepAlive()
is the server-side one-shot equivalent and Connection.startKeepAlive() its library-timed form.
The server drops an inbound 0x80 without dispatching it, though when idle detection is enabled
an inbound 0x80 — like any inbound datagram — refreshes that client's server-side liveness
timestamp. A server-side keepalive is server → client and refreshes cone NATs only — the
authoritative keepalive is the client's. See Scheduled keepalive.
A server behind symmetric NAT still needs a rendezvous/relay (out of scope). Background:
issue #1,
docs/plans/1-nat-traversal/architecture.md,
docs/plans/1-nat-traversal/plan-tier-2.md.
Post-handshake application data is not limited to text. connection.channel(DeliveryMode.UNRELIABLE).send(bytes)
(server → client) / client.channel(DeliveryMode.UNRELIABLE).send(bytes) (client → server) —
or the older Connection.push(ByteArray) / MultiConnectionUDPClient.send(ByteArray) these are
implemented over — and MultiConnectionUDPServer.pushToAll(ByteArray) frame the payload as an
0x90 unreliable datagram and put it on the wire verbatim — no encoding, no trim, no
reserved first byte. channel(DeliveryMode.UNRELIABLE).actuateBytes(handler) (or the older
Connection.actuateBytes / MultiConnectionUDPClient.startBytes / MultiConnectionUDPServer.startBytes)
registers an inbound handler that receives an exact-length, undecoded, untrimmed copy of the
payload (the frame's 0x90 tag and channel byte already stripped). The text and bytes handlers
are mutually exclusive per connection per mode — the last actuate / actuateBytes call on
that UdpChannel wins; a reliable-ordered handler bound via channel(DeliveryMode.RELIABLE_ORDERED)
is independent of both — see Reliable-ordered channel.
Because control traffic (Iam, keepalive, probe) and application data are separated by the
DatagramType tag rather than by matching text, any payload bytes survive intact — a
binary blob that happens to decode-and-trim to KA or that starts with 0x81 is delivered to
start / actuate exactly as sent, with no lead-byte workaround needed.
Every receive path in this module — MultiConnectionUDPServer, MultiConnectionUDPClient,
and UDPSendReceiveServer — accepts datagrams up to 65507 bytes (the maximum UDP
payload over IPv4) whole; a larger payload is truncated, not rejected. Each constructor
takes an optional receiveBufferBytes (512..65507, default 65507) to shrink the
per-instance receive buffer when datagrams are known to be small; a datagram larger than
the configured size is truncated and a WARN is logged. For real-network use keep frames
under the path MTU (~1200 bytes) to avoid IP fragmentation and the loss that comes with
it — application payloads carry 2 bytes of framing overhead (0x90 + channel).
By default the server tracks no per-connection idle time and never signals that a client has
gone silent (a crashed / powered-off / out-of-range client stays addressable until an
explicit Connection.terminate() or a same-name supersede). Pass a positive
idleTimeoutMillis to opt in: a dedicated mcups-liveness daemon thread then reports any
connection with no inbound datagram (application data or a keepalive) within the threshold via
onClientDisconnect(connection, reason):
val server = object : MultiConnectionUDPServer(idleTimeoutMillis = 60_000) {
override fun onClientConnect(connection: Connection) { /* ... */ }
override fun onClientDisconnect(connection: Connection, reason: DisconnectReason) {
when (reason) {
DisconnectReason.TIMEOUT -> startReconnectGraceWindow(connection) // do NOT terminate
DisconnectReason.SUPERSEDED -> rebindSession(connection)
DisconnectReason.TERMINATED -> releaseSession(connection)
}
}
}onClientDisconnect runs on the dispatch thread (mcups-dispatch), not the caller's. It is
notify-only: TIMEOUT does not remove the registration or call terminate() — the
connection stays addressable by pushToAll so the application can run its own grace window
and call terminate() when ready (which then produces a second TERMINATED call). A
same-name supersede fires SUPERSEDED. stop() teardown fires nothing. With the default
idleTimeoutMillis = 0 there is no extra thread and no per-datagram cost.
By default the server accepts every well-formed Iam: one whose name is non-empty and whose
credential slot, if present, is non-empty. The server itself refuses any other Iam with
REFUSED malformed before admit is consulted, so admit never sees an empty name. Override
admit(name, peer, credential): Admission to screen a handshake before it is registered —
validate the credential, enforce a capacity gate, reject a banned name:
val server = object : MultiConnectionUDPServer() {
override fun admit(name: String, peer: InetSocketAddress, credential: String): Admission =
when {
isBanned(name) -> Admission.Refused("banned")
!tokenValid(name, credential) -> Admission.Refused("invalid credential")
connections.size >= capacity -> Admission.Refused("server full")
else -> Admission.Admitted
}
override fun onClientConnect(connection: Connection) { /* ... */ }
}val client = MultiConnectionUDPClient(serverAddress)
client.handshake("alice", credential = base64UrlToken).fold(
onSuccess = { client.channel(DeliveryMode.UNRELIABLE).actuate { msg -> /* ... */ } },
onFailure = { ex -> if (ex is HandshakeRefusedException) showError(ex.reason) },
)The credential is one whitespace-free token, opaque to the library (base64url-encode a
structured or binary value), passed to admit verbatim; an absent credential arrives as the
empty string (a present-but-empty slot is malformed and never reaches admit). It rides in cleartext — the library gives you a channel, not
confidentiality. Back-compat is total in every direction: an old client (Iam <name>)
against a new server registers with an empty credential and the default admit; a new
client (Iam <name> <cred>) against an old server still registers (the old server ignores
the trailing token — so auth is not enforced until both ends are on 1.4.0+). admit
runs inline on the listener thread, so keep it fast and local.
Both sides have an opt-in, idle-aware library-managed keepalive so a consumer never
hand-rolls the ~20 s idle timer around the one-shot primitive:
// client
client.startKeepAlive() // default 20 s; or startKeepAlive(intervalMillis)
client.stopKeepAlive() // also done by client.stop()
// server, per connection
override fun onClientConnect(connection: Connection) {
connection.startKeepAlive() // server → client keepalive; also stopped by terminate() / stop()
}It is idle-aware: an 0x80 keepalive goes out only after intervalMillis of output silence
on that path, so application traffic defers it (a scheduled keepalive may lag up to one
quarter-interval, max 5 s, past the interval). Each side lazily creates one daemon
ScheduledExecutorService (mcupc-keepalive / mcups-keepalive) on the first startKeepAlive
call and shuts it in stop() — a consumer that never opts in pays for no thread. The one-shot
sendKeepAlive() / keepAlive() primitives are unchanged and still own no timer; both emit the
same 0x80 datagram whether sent one-shot or scheduled. Server → client keepalives refresh
only endpoint-independent (cone) NAT mappings; the authoritative keepalive is still the
client's own.
An opt-in transport-level probe on either side measures round-trip time and packet loss without the consumer hand-rolling an application ping. Off by default:
// client
client.startProbe() // default 1 s; or startProbe(intervalMillis), >= 250 ms
client.stopProbe() // also done by client.stop()
val q = client.linkQuality() // null until the first probe reply comes back
// q.rttMillis, q.rttVarianceMillis (jitter proxy), q.packetLossRatio, q.probesSent, q.probesDelivered
// server, per connection
override fun onClientConnect(connection: Connection) {
connection.startProbe() // server → client probe; also stopped by terminate() / stop()
}Each armed side sends one tiny 0x81 probe (an 8-byte big-endian sequence) per interval and
the peer echoes it back as 0x82; the RTT is computed against the prober's own monotonic clock
(EWMA-smoothed, RFC 6298 SRTT/RTTVAR). intervalMillis must be at least
TransportWireFormat.MIN_PROBE_INTERVAL_MILLIS (250 ms); a lower value fails the call rather
than being silently rounded up. The responder always answers an inbound 0x81 (a
client unconditionally, a server for a registered origin), so the peer can measure even if this
side never opted in. 0x81 / 0x82 are consumed by the transport and never reach start /
actuate. Each side lazily creates one daemon ScheduledExecutorService
(mcupc-probe / mcups-probe) on the first startProbe and shuts it in stop().
packetLossRatio is lagging and windowed (it needs a few seconds to first resolve and
reacts over tens of seconds) — a link-health gauge, not a fast congestion trigger. The probe
requires both ends on 1.6.0+ (and, on the framed 2.x transport, both ends on 2.0.0+);
against an incompatible peer no 0x82 comes back and packetLossRatio climbs toward 1.0.
Both sides can open a second delivery plane alongside the default unreliable one — acked, retransmitted, delivered exactly once, in send order:
// either side
val reliable = connection.channel(DeliveryMode.RELIABLE_ORDERED) // or client.channel(...)
reliable.actuate { message -> /* decoded, trimmed text, delivered once, in order */ }
reliable.send("ability-cast").fold(
onSuccess = { /* accepted for reliable delivery - buffered and sequenced, not "on the wire" yet */ },
onFailure = { ex ->
when (ex) {
is ReliableWindowFullException -> Unit // routine backpressure - retry, coalesce, or drop
is ReliableMessageTooLargeException -> Unit // programming/config mistake - split or raise the cap
}
},
)Result.success from a reliable send means accepted for reliable delivery — buffered,
sequenced, and retransmitted until acked — never "already on the wire". A full in-flight window
fails immediately with ReliableWindowFullException (there is no internal absorb queue); an
oversize message fails with ReliableMessageTooLargeException — v1 does not fragment. Both
extend the open ReliableSendFailure, so a caller can is-check one type or catch the
supertype. The default cap is 1024 bytes, configurable per server/client instance up to a
hard 8192-byte maximum (reliableMaxMessageBytes constructor parameter).
The reliable and unreliable planes have independent sequence spaces and independent handlers —
binding one via channel(...).actuate* never disturbs the other — but delivery for both shares
the same per-connection dispatch executor, so head-of-line blocking is inherent: a missing
reliable message stalls delivery of every reliable message sent after it, and also delays
unreliable delivery to that same connection. Use the reliable channel for discrete events
(chat, ability casts, inventory changes) and the unreliable one for frequent state snapshots
where a stale or dropped update is harmless.
The channel is created lazily. On the client, the first reliable send or the first inbound
0xA0/0xA1 on channel 0x00 from the server creates it — obtaining the handle via
channel(RELIABLE_ORDERED) or binding its handler alone does not. On the server, the first
reliable send to a peer, the first startReliable/actuate* binding on that peer's channel,
or the first inbound reliable datagram on channel 0x00 from a registered peer creates it. (A
datagram on any other channel is dropped with a WARN and creates nothing.) Either way it arms one
shared mcups-retransmit /
mcupc-retransmit daemon thread per side (created only once a reliable channel is actually used
on that side). See docs/webtools-udp-architecture.md for
the complete creation/teardown picture. A terminate(), a same-name supersede, or stop() on
either side discards any un-acked data and resets the channel; a send after teardown fails its
Result. Both ends must be on webtools-udp 2.0.0+ — a pre-2.0.0 peer never emits or answers
0xA0/0xA1.
Limits in 2.0. Stated loudly because it is the biggest limitation of this feature: there
is no congestion control — no cwnd, no pacing, no loss-triggered rate reduction. Under
sustained loss the channel retransmits at RTO-backoff cadence up to the in-flight window, then
fails sends; this is adequate for a low-rate discrete-event stream and inadequate for a
genuinely congested path. In addition: no fragmentation — an oversize message fails
immediately with ReliableMessageTooLargeException rather than being split; one reliable
channel per connection — there is no channel id a consumer can choose; and no session
resume — a terminate(), a same-name supersede (NAT rebind), or stop() discards all
in-flight and buffered reliable data and resets the channel, and a fresh registration always
gets a fresh channel. See docs/webtools-udp-protocol.md for
the wire-level detail behind each of these.
Server-wide conveniences mirror the unreliable ones: MultiConnectionUDPServer.startReliable /
pushToAllReliable. Unlike pushToAll, pushToAllReliable attempts every peer even if an
earlier one's window is full, reporting only the first failure — a single busy peer must not
stop a broadcast reaching everyone else.
MultiConnectionUDPClient performs the handshake and then owns a background
listener thread and dispatch thread for the rest of the session, so a consumer
never hand-rolls the receive loop or the "one socket for everything" invariant:
val client = MultiConnectionUDPClient(serverAddress)
client.handshake("alice").getOrThrow()
val unreliable = client.channel(DeliveryMode.UNRELIABLE)
unreliable.actuate { message -> /* handle inbound data; runs on client's dispatch thread */ }
unreliable.send("hello")
// client.start { ... } / client.send("hello") - the pre-2.0-channel spelling, deprecated but
// still fully functional; channel(UNRELIABLE) is the recommended replacement (see Migrating to 2.0)
client.startKeepAlive() // library times it; no caller timer needed
// client.sendKeepAlive() is still available for callers who want to drive the cadence themselves
// or, for a binary protocol:
unreliable.actuateBytes { bytes -> /* exact datagram body, no decode, no trim */ }
unreliable.send(byteArrayOf(0x01, 0x02, 0x03)) // any bytes, no reserved first byte needed
// ...
client.stop()Inbound keepalive datagrams (a server may send one via Connection.keepAlive()) are dropped
automatically and never reach the actuate callback.
webtools-udp 2.0.0 breaks the post-handshake wire format: every datagram after the
handshake now leads with a 1-byte DatagramType tag (§ UDP transport
protocol). Both ends of a connection must be on 2.0.0+ — a
1.x peer's unframed keepalive/probe/application datagrams are dropped as unrecognized, and a
2.x client against a 1.x server fails handshake() with a typed
IncompatibleProtocolException rather than exchanging data neither side can parse correctly.
HandshakeWireFormat: KEEPALIVE_TOKEN, DEFAULT_KEEPALIVE_INTERVAL_MILLIS,
isKeepAlive, PROBE_REQUEST_VERB, PROBE_REPLY_VERB, DEFAULT_PROBE_INTERVAL_MILLIS,
probeRequestMessage, probeReplyMessage, isProbeRequest, isProbeReply, probeToken.
Their role moves to TransportWireFormat in binary form.HandshakeWireFormat.isRegistered now requires the exact REGISTERED 2 reply, not a
bare REGISTERED; registeredMessage() / registeredProtocolVersion(reply) are new.DatagramType tag, there is no longer a reserved first byte to avoid —
any binary payload, including one that used to collide with KA/PING/PONG, rides an
0x90 frame intact. See Binary application payloads.reliableMaxMessageBytes on both MultiConnectionUDPServer and
MultiConnectionUDPClient: the reliable-channel payload cap, default 1024 bytes, validated to
1..8192. An oversize reliable send fails with ReliableMessageTooLargeException.startProbe
(on Connection and on MultiConnectionUDPClient) now fails with IllegalArgumentException
for an intervalMillis below TransportWireFormat.MIN_PROBE_INTERVAL_MILLIS (250 ms) instead
of silently running at 250 ms, and sends exactly one probe per interval — 1.6.0 probed up to
four times too often (exactly 4× at the 1 s default), so probesSent and packetLossRatio
differ from 1.6.0 at the same settings. linkQuality() now counts a probe as lost once it
is overdue as of the call.MultiConnectionUDPClient.start / startBytes call now rebinds the handler
instead of starting a second listener: the last call to start, startBytes, or
channel(DeliveryMode.UNRELIABLE).actuate* wins, and only the first of them starts the
listener thread.Deprecations in 2.0. The public reliable channel API (connection.channel(mode) /
client.channel(mode), see Reliable-ordered channel) is additive,
but it also gives every per-Connection primitive a channel-shaped replacement, so eight
existing members now carry @Deprecated(WARNING) with a working ReplaceWith quick-fix — four
on Connection (push(String), push(ByteArray), actuate, actuateBytes) and four on
MultiConnectionUDPClient (send(String), send(ByteArray), start, startBytes). All eight
remain fully functional in 2.0.0 — this is a warning, not a break — and are slated to
leave the public surface at a future 3.0.0 (demoted, not deleted: the implementations survive
internally). MultiConnectionUDPServer.start / startBytes / pushToAll are not
deprecated and are not going anywhere: channel(...) is per-Connection, so there is no
server-wide replacement to point a ReplaceWith at.
The repo is a Gradle multi-module build; the modules are :webtools-udp, :webtools-scraping,
:webtools-browser, :webtools-http (Kotlin Multiplatform), :webtools-http-commoncheck
(unpublished), :webtools-feeds (Kotlin Multiplatform) and :webtools-feeds-commoncheck (unpublished). The configuration they share lives in the included build-logic/ build (see below).
./gradlew build # compile + full test suite, every module
./gradlew :webtools-udp:build # one module only
./gradlew test # every test set, one JVM per module
./gradlew gatingTest # gating - fast pre-commit checks (all modules)
./gradlew componentTest # component - isolated component behaviour
./gradlew integrationTest # integration - real sockets / external interfaces
./gradlew deterministicTest # deterministic - pure input->output mappings
./gradlew e2eTest # e2e - full-stack flows
./gradlew nonfunctionalTest # nonfunctional - robustness / security properties
./gradlew uatTest # uat - manual acceptance (mostly @Disabled)
./gradlew verifyJavaFloor # the Java 11 floor of every published module's jars, metadata and runtime classpath (part of check)
./gradlew :webtools-http-commoncheck:build # compile webtools-http's commonMain for js (no Node.js needed)
./gradlew :webtools-feeds-commoncheck:build # compile webtools-feeds' commonMain for js (no Node.js needed)
./gradlew :build-logic:test # build-logic's pre-commit gate: its gating, component and deterministic checks
./gradlew :build-logic:fixturesPublicationCheck # publish a copy of webtools-http to a temporary repository and resolve its test fixtures from it (also part of webtools-http's check)
./gradlew :build-logic:integrationTest :build-logic:e2eTest :build-logic:nonfunctionalTest # on demand: TestKit checks
./gradlew dokkaGeneratePublicationHtml # the API reference of the five published modules, aggregated in build/dokka/html; fails on a broken KDoc link or an undocumented public declaration (the CI "docs" job runs it)Each published module's javadoc jar is its Dokka output, and the root project aggregates the five modules into one
reference. The theme is docs/dokka/styles/dokka-styles.css, with the logo in docs/brand/, and webtools-http's test
fixtures are part of the reference. The Dokka versions in settings.gradle.kts (pluginManagement.plugins) and
build-logic/build.gradle.kts must agree.
./gradlew build and check also compile every *-commoncheck module for js: an unpublished module
that compiles a multiplatform module's commonMain so a JVM API in common code is a compile error. No
Node.js is needed.
Each module organises its tests by the project's named test sets, in the package
com.spartanlabs.testing.<set>.webtools.<name>, where <name> is the module's name without webtools-
(udp, http, feeds, …): a JVM module under <module>/src/test/kotlin/, a multiplatform module under
<module>/src/commonTest/kotlin/ and <module>/src/jvmTest/kotlin/, both of which run on the JVM target. A
set's task selects its tests by package (com.spartanlabs.testing.<set>.*) from the module's JVM test
compilation, so any set can be run or gated on its own; a test class outside those packages runs under
test only. Shared test helpers live under testing/support/ and are not published. The published test
fixtures of webtools-http are a separate source set, webtools-http/src/jvmTestFixtures (package
com.spartanlabs.webtools.http.testing), whose own tests sit in com.spartanlabs.testing.<set>.webtools.http.testing;
webtools-feeds' tests use them through testFixtures(project(":webtools-http")).
Every JVM module compiles to Java 11 bytecode against the Java 11 API, which is what the install
section's Requires JDK 11 or newer rests on. Building needs a JDK 17 to 26, the range Gradle
9.7.1 runs on, and the tests of every set run on that JDK too; no JDK 11 is involved. The convention
plugins set jvmTarget, -Xjdk-release and the Java compatibility from one constant,
WebToolsBuild.JAVA_FLOOR, so the result does not depend on which of those JDKs builds.
verifyJavaFloor, part of check, fails the build when a built jar holds a class above Java 11
bytecode, when a library variant of the generated Gradle Module Metadata declares another
org.gradle.jvm.version, or when a runtime dependency holds such a class; on success it writes a
one-line summary to <module>/build/reports/java-floor/verifyJavaFloor.txt. Running
./gradlew verifyJavaFloor checks every library module without running any module's tests.
build-logic/ is a Gradle build included through pluginManagement { includeBuild("build-logic") }
in settings.gradle.kts. It holds the build configuration every module shares as four convention
plugins, which a module applies by id in its plugins {} block:
| Plugin id | Configures |
|---|---|
webtools.published-library |
the Maven Central publishing skeleton: signing, the shared POM metadata and Dokka (theme, source links, failOnWarning, reportUndocumented) |
webtools.kotlin-jvm-library |
a Kotlin/JVM library: the Java floor and verifyJavaFloor, test dependencies, JUnit Platform, the test tasks of each set, javadoc jar from Dokka; applies webtools.published-library
|
webtools.kotlin-multiplatform-library |
a Kotlin Multiplatform library with a jvm() target (used by webtools-http and webtools-feeds): the Java floor, the TargetJvmVersion attribute and verifyJavaFloor, javadoc jar from Dokka; applies binary-compatibility-validator — commit api/<module>.api — and webtools.published-library
|
webtools.kotlin-common-check |
an unpublished, check-only js module that compiles a sibling's commonMain (no publishing, no tests, no Node.js, no Java floor) |
Change shared build behaviour there, not in a module's build file: a module's build.gradle.kts
adds its dependencies, its coordinates, its POM name and description, and any configuration that
belongs to it alone (for example webtools-udp's version and its UDP-port lock, or webtools-http's
test-fixtures variant).
build-logic's own tests use the same set packages, and each of its sets with tests has a task
of the same name (./gradlew :build-logic:gatingTest, …). ./gradlew build runs none of them, with one
exception: webtools-http's check (so build, check and :webtools-http:check) runs
:build-logic:fixturesPublicationCheck, which publishes a copy of the module with a throwaway signing key
to a temporary repository and resolves its test fixtures from three build shapes. It takes about a minute
when its inputs changed and is UP-TO-DATE otherwise; its first run needs the network. It is wired only
when WebTools is the root build, so a build that includes WebTools never runs it.
:build-logic:test runs the sets that start no nested build, in seconds.
:build-logic:integrationTest, :build-logic:e2eTest and :build-logic:nonfunctionalTest run
nested Gradle builds against generated projects; their first run downloads the Kotlin compiler and
the test dependencies into build-logic/build/testkit/home, so it needs the network.