
MediaWiki bots and tools: read and edit any wiki, parse wikitext byte-for-byte, run dry-run bots with resumable state, SPARQL, EventStreams, and offline testkit.
A Kotlin Multiplatform library for MediaWiki bots and tools: read and edit any wiki, parse wikitext that writes back byte for byte, and run bots that are dry runs until told otherwise. On the JVM, macOS, Linux, Windows and JavaScript, with an npm package and a command-line tool.
commonMain code.Flows that page only as far as you collect.{{nobots}}; resumable state, diffs and skip logs.api.php. Wikibase entities, SPARQL, EventStreams, Wikimedia page views, PetScan,
PagePile and XML dumps, read on every core.kwikibot-testkit gives your bot a fake wiki.docs/comparison.md sets kwikibot beside pywikibot, mwclient, mwn, m3api and three wikitext parsers, feature by feature, with parsing speed, memory and start-up.
import com.fenakhay.kwikibot.client.Family
import com.fenakhay.kwikibot.client.WikiClient
import com.fenakhay.kwikibot.client.WikiConfig
import com.fenakhay.kwikibot.model.LangCode
import com.fenakhay.kwikibot.net.UserAgent
import com.fenakhay.kwikibot.wikitext.Wikitext
suspend fun main() {
val config = WikiConfig(UserAgent("MyBot", "1.0", "https://en.wiktionary.org/wiki/User:MyBot"))
WikiClient(config).use { client ->
val wiki = client.wiki(LangCode("en"), Family.WIKTIONARY)
val page = wiki.pages.content(wiki.ref("volcano")) ?: return
val code = Wikitext.parse(page.text)
println("${page.ref.title} at revision ${page.revisionId}")
println("${code.templates().size} templates, ${code.wikilinks().size} links")
for (template in code.templates("col")) {
println(template.parameters.drop(1).map { it.value.text.trim() })
}
}
}volcano at revision 93733515
287 templates, 15 links
[hypervolcano, paleovolcano, pneumonoultramicroscopicsilicovolcanoconiosis, vog, volcanist, …]
The user agent is required, and its contact must be somewhere the wiki's operators can reach you: Wikimedia blocks requests without one. Reading needs no account, and parsing touches no network.
kwikibot is on Maven Central. Gradle picks the build for each platform from one set of
coordinates, so a multiplatform project depends on it once, in commonMain:
kotlin {
sourceSets {
commonMain.dependencies {
implementation("com.fenakhay.kwikibot:kwikibot:1.4.0")
}
commonTest.dependencies {
implementation("com.fenakhay.kwikibot:kwikibot-testkit:1.4.0")
}
}
}A JVM-only project uses the same coordinates:
dependencies {
implementation("com.fenakhay.kwikibot:kwikibot:1.4.0")
testImplementation("com.fenakhay.kwikibot:kwikibot-testkit:1.4.0")
}With a version catalog:
[versions]
kwikibot = "1.4.0"
[libraries]
kwikibot = { module = "com.fenakhay.kwikibot:kwikibot", version.ref = "kwikibot" }
kwikibot-testkit = { module = "com.fenakhay.kwikibot:kwikibot-testkit", version.ref = "kwikibot" }implementation(libs.kwikibot)Maven does not read Gradle's metadata, so a Maven project names the JVM artifact, which ends in
-jvm:
<dependency>
<groupId>com.fenakhay.kwikibot</groupId>
<artifactId>kwikibot-jvm</artifactId>
<version>1.4.0</version>
</dependency>| Needs | |
|---|---|
| Kotlin/JVM project | Kotlin 2.4 or newer, and JDK 21 or newer to run. Gradle resolves kotlin-stdlib to 2.5.0-RC, the version kwikibot is built with |
| Kotlin/Native or Kotlin/JS project | Kotlin 2.5.0-RC or newer: Kotlin 2.4 cannot read klibs built by 2.5 |
| npm package | Node 22.3 or newer |
| A wiki | MediaWiki 1.39 or newer, see Compatibility |
On the JVM, the library needs one more thing from you: it logs through SLF4J and ships no
binding. Without one, SLF4J prints a warning and drops every line, retries and logins included. A
command-line bot adds runtimeOnly("org.slf4j:slf4j-simple:2.0.20"). See Logging.
On JavaScript, wiki.signatures() reads a wiki's time zone, such as Europe/Paris, through the
runtime's Intl, which Node and browsers ship, so nothing loads a time zone database.
kotlinx-datetime's own TimeZone.of knows no named zone there until @js-joda/timezone is
loaded, which matters only to code that calls it.
kwikibot brings in every module but the test kit. Each is also published on its own, for a
program that wants less: a tool that only parses wikitext has no reason to pull in Ktor.
| Module | What it holds |
|---|---|
kwikibot |
Everything below except the test kit |
kwikibot-model |
Value types with no I/O: Title, Namespace, PageRef, PageContent, Revision, the sealed WikiError and EditOutcome, signature timestamps |
kwikibot-wikitext |
The wikitext parser and editor, with no dependencies of its own |
kwikibot-net |
The HTTP transport: user agent, credentials and login, tokens, throttle, maxlag, retries, response cache |
kwikibot-protocol |
Reading api.php responses: errors and warnings, site info, paraminfo, continuation, page decoding |
kwikibot-client |
WikiClient, the Wiki handle and its services, EventStreams, SPARQL, page views, and XML dumps |
kwikibot-wikibase |
Entities, statements and data values |
kwikibot-bot |
botRun, botMain, page sources and filters, {{nobots}}, stop pages, run state, diffs, logs, plural forms, prompts |
kwikibot-fleet |
Many bots in one process: Bot, Fleet, fleetMain, shared sessions and a shared stop page, schedules. Not in the umbrella: depend on it to use it |
kwikibot-testkit |
FakeWiki, FakePageService, MockTransport and FailOnDeprecation, for testing a bot |
Depend on every module at the same version. Sources and KDoc are published with each artifact, so
the IDE has documentation to show and a debugger has code to step into. On Maven, kwikibot-js is
the umbrella's Kotlin/JS artifact, for Kotlin projects; the npm package is a separate thing,
below.
Every module builds for every platform below, with one API. Where they differ:
| JVM | macOS, Linux, Windows | Node | Browser | |
|---|---|---|---|---|
| HTTP engine | OkHttp | Darwin, Curl, WinHTTP | fetch | fetch |
Files: DiskCache, file uploads and downloads, BotConfig, RunState, diff and skip logs |
✓ | ✓ | ✓ | – |
XML dumps (XmlDump, DumpPage) |
✓ | ✓ | ✓, on worker threads | From bytes, on one thread |
botMain, TerminalPrompt, environment variables |
✓ | ✓ | ✓ | – |
Tested by ./gradlew check and CI |
✓ | ✓ | ✓ | the npm package, in headless Chrome |
Paths are kotlinx.io.files.Path. The file APIs
fail in a browser, which has no filesystem; there, environment variables read as unset and a
prompt reads no answer. On JavaScript only the
request timeout applies, since fetch has no connect or socket timeout, and every request also
carries Api-User-Agent, because a browser will not send a page's own User-Agent. From a page on
a site other than the wiki's, requests carry origin=*, which MediaWiki answers as it would a
logged-out reader. The KDoc of each connection setting says which engines honour it.
val config = WikiConfig(
userAgent = UserAgent("MyBot", "1.0", "https://en.wiktionary.org/wiki/User:MyBot"),
throttle = Throttle(read = 100.milliseconds, write = 10.seconds),
)
WikiClient(config, Credentials.fromEnvironment()).use { client ->
val wiktionary = client.wiki(LangCode("en"), Family.WIKTIONARY)
val wikidata = client.wiki(ApiEndpoint("www.wikidata.org"))
val yours = client.wiki(ApiEndpoint("wiki.example.org", scriptPath = "/w"))
}A WikiClient holds the connections, the login and the pacing, and hands out a Wiki for each
site it opens. Opening one logs in if there are credentials and reads the site's namespaces and
settings. wiki.identity says who the wiki thinks you are.
Family names the Wikimedia projects: Wikipedia, Wiktionary, Wikisource, Wikiquote, Wikibooks,
Wikinews, Wikiversity, Wikivoyage, Commons, Wikidata, Meta, mediawiki.org and test.wikipedia.org.
Any other wiki is an ApiEndpoint. When all you have is a URL, ApiDetector.discover finds its
api.php, which is what kwikibot detect does.
| For | Where it comes from | |
|---|---|---|
Credentials.BotPassword(account, botName, password) |
Most bots |
Special:BotPasswords on the wiki. account is the account name alone; kwikibot adds @botName itself |
Credentials.OAuth2(accessToken) |
A bot that runs as its owner and never asks anyone else to log in | An owner-only OAuth 2 consumer, proposed at Special:OAuthConsumerRegistration on Meta. The token is shown once |
Credentials.OAuth1(consumerKey, consumerSecret, accessToken, accessSecret) |
The same, for a tool or consumer that only has OAuth 1.0a | An owner-only OAuth 1.0a consumer, proposed at Special:OAuthConsumerRegistration on Meta. The four values are shown once |
Credentials.Anonymous |
Reading | The default |
Give a bot password the grants "High-volume (bot) access" and "Edit existing pages", and whatever
else the bot does, such as creating, moving or uploading. Without the first, even a flagged account
logs in without the bot and apihighlimits rights: its edits are not marked as a bot's, and a
query names at most 50 pages instead of 500.
An OAuth 2 token goes on every request as a bearer header, and the client keeps the session
cookies the wiki sets and sends them back with it, which Wikimedia asks of owner-only tokens so that
the account's own rate limit applies. Cookies are kept per client, so each account needs a
WikiClient of its own.
An OAuth 1.0a consumer needs no login round trip either. Every request is signed with HMAC-SHA1, as
RFC 5849 says, and the signature goes in an
Authorization: OAuth header. The signature covers the method, the URL and every query and form
parameter, but not a multipart body, so an upload is signed too. Each attempt at a request gets a
nonce and a timestamp of its own, so a retry is signed afresh. The two secrets only key the signature
and are never sent; a response cache keeps the account apart by a digest of the consumer key and the
access token. The wiki checks the signature against its own $wgServer and the path it was asked for,
so a proxy in front of it that rewrites either breaks it.
val credentials = Credentials.OAuth1(
consumerKey = System.getenv("KWIKIBOT_OAUTH_CONSUMER_KEY"),
consumerSecret = System.getenv("KWIKIBOT_OAUTH_CONSUMER_SECRET"),
accessToken = System.getenv("KWIKIBOT_OAUTH_ACCESS_TOKEN"),
accessSecret = System.getenv("KWIKIBOT_OAUTH_ACCESS_SECRET"),
)
WikiClient(config, credentials).use { client -> client.wiki(endpoint) }Keep secrets out of source and out of shell history. Credentials.fromEnvironment() reads
KWIKIBOT_OAUTH_TOKEN, or else KWIKIBOT_OAUTH_CONSUMER_KEY, KWIKIBOT_OAUTH_CONSUMER_SECRET,
KWIKIBOT_OAUTH_ACCESS_TOKEN and KWIKIBOT_OAUTH_ACCESS_SECRET, or else KWIKIBOT_ACCOUNT,
KWIKIBOT_BOT_NAME and KWIKIBOT_PASSWORD; KWIKIBOT_ACCOUNT is the username of either kind of
OAuth. With none of them set it gives Anonymous, and with some but not all of a group it fails
rather than guess. An OAuth 2 token and any of the OAuth 1.0a variables set together fail too, since
nothing says which was meant. The message names the variables, never what they hold. That is
for a program that builds its own WikiClient; botMain and the kwikibot tool take the account
from kwikibot.toml, which names the variable to read a secret from instead of holding it.
Every request from a logged-in client asserts its account. When the wiki ends the session, the
client logs back in and repeats the request instead of carrying on anonymously, up to
WikiConfig.relogins times in ten minutes.
val page = wiki.pages.content(wiki.ref("volcano")) ?: return
val updated = Wikitext.parse(page.text)
.mapTemplates("col") { it.withParameter("title", "terms derived from volcano") }
val outcome = wiki.pages.edit(page.ref) {
text = updated.serialize()
summary = "titling the derived terms"
baseRevision = page.revisionId
}
when (outcome) {
is EditOutcome.Saved -> println("saved as ${outcome.revision}")
is EditOutcome.NoChange -> println("already done")
is EditOutcome.Refused -> println("refused: ${outcome.detail}")
}The diff on the wiki is the parameter that changed and nothing else. baseRevision is what lets
the wiki refuse the save if somebody else edited the page since it was read; without it, the later
edit silently wins.
Every refusal a wiki can return is a case of EditOutcome.Refused rather than an exception:
Conflict, Protected, Filtered, RateLimited, CaptchaRequired, PermissionDenied,
PageStateChanged and Rejected, each saying whether trying again could help. Failures that are
not about the edit, such as a wiki that cannot be reached, are a sealed WikiError.
wiki.pages also moves, deletes, undeletes, protects, rolls back, undoes, purges and watches, and
edits sections: section = "new" leaves a message on a talk page. testActions asks whether an
edit would be allowed before one is computed, and on a wiki with pending changes,
wiki.extensions.flagged(refs) says whether an edit is live or queued.
wiki.lists.categoryMembers(wiki.ref("Category:English lemmas"), namespaces = setOf(Namespace.MAIN))
.take(10)
.collect { println(it.title) }Category members, backlinks, transclusions, links, search, prefix search, all pages, special
pages, watchlists, contributions, recent changes, logs and page histories are all cold Flows.
They fetch the next batch only when collection asks for it, so take(10) costs one request.
inNamespaces, titleMatching, subpageDepthAtMost and distinctPages compose on top of any of
them, filtering what arrives, so a narrow filter can page through a whole category before it finds
ten. A listing's own parameters, such as namespaces and startSortKey, filter on the wiki
instead. intersect and excluding read their second flow in full before the first.
val target = wiki.ref("File:Volcano.jpg")
val uploaded = wiki.files.upload(Path("volcano.jpg"), target, comment = "a volcano", text = "== Summary ==")
val fetched = wiki.files.uploadFromUrl("https://photos.example.org/volcano.jpg", target, comment = "a volcano")
for (outcome in listOf(uploaded, fetched)) {
when (outcome) {
is UploadOutcome.Uploaded -> println("uploaded as ${outcome.title}")
is UploadOutcome.Warned ->
if (!outcome.isDuplicateOnly) outcome.fileKey?.let { wiki.files.publishStashed(it, target) }
is UploadOutcome.Refused -> println("refused: ${outcome.detail}")
}
}upload sends a large file in chunks, and one whose answer is lost carries on where it stopped.
uploadFromUrl has the wiki fetch the file itself, which it does only with $wgAllowCopyUploads on,
from a host in $wgCopyUploadsDomains, for an account with the upload_by_url right; each of those
missing is an error rather than an outcome, since no file would fare differently. Either way a
warning is an outcome: the file waits under its fileKey, and publishStashed publishes it without
sending it again. info, usage, globalUsage, duplicatesOf and download read what is there.
val refused = wiki.users.setPreferences(mapOf("userjs-mybot-last-run" to "2026-11-04", "rclimit" to "500"))
val lastRun = wiki.users.preferences()["userjs-mybot-last-run"]
val drift = wiki.meta.serverTime() - Clock.System.now()
when (val sent = wiki.users.email("Example", "Your bot request", "It has been approved.")) {
is EmailOutcome.Sent -> println("sent")
is EmailOutcome.Undeliverable -> println("not sent: ${sent.reason}")
}
val global = wiki.extensions.globalUser("Example")
println("${global?.home}: ${global?.editCount} edits")setPreferences answers with the preferences the wiki refused, since MediaWiki saves the rest and only
warns about those; null puts one back to its default, and resetPreferences puts back a whole kind.
email sends a real email from the account to a person, so it is paced as a write and wikis rate-limit it.
An account that cannot be emailed is an outcome, and a sender who may not send is an error. serverTime
is the wiki's clock, asked for every time. globalUser reads a CentralAuth account, or null when there is
none, and lists its local accounts only when MERGED or UNATTACHED is asked for, since an account that has
been around for years has hundreds.
wiki.paramInfo reads the wiki's own description of its API: the limits an account gets, the
values a parameter takes, and which of those are deprecated. That is how a renamed value goes out
under the new name to a wiki that knows it and under the old name to one that does not.
requireRight, requireExtension and requireVersion fail at the start of a run instead of
halfway through. The extension services (GeoData, PageImages, TextExtracts, Linter, Echo, Thanks,
FlaggedRevs, AbuseFilter, PageTriage, ProofreadPage, CentralAuth and the rest) refuse when their
extension is absent.
val code = Wikitext.parse("==English==\n===Noun===\n{{en-noun}}\n\n{{col|en|vog|tephra}}\n")
code.templates("col").single().value("2")
code.wikilinks()
val english = code.outline().find("English", level = 2)
val edited = code
.mapTemplates("col") { it.withParameter("4", "lahar") }
.addCategory("Volcanoes")
edited.serialize()Wikitext.parse never fails: no input is rejected, and markup that does not parse, such as an
unclosed template or a stray bracket, stays literal text, which is what MediaWiki renders. Nor
does any input nest too deep: templates, links or tags nested a hundred thousand levels deep parse,
compare and write back like any other page, on every platform and whatever the thread's stack. Nodes
are immutable values, every edit returns a new Markup, and every node that was not touched keeps
the exact text it was parsed from.
The parser finds templates, parameters and headings where MediaWiki does, and ends HTML tags where MediaWiki's paragraphs and lists end them. It is checked against MediaWiki's own parse of 288 fragments, and against 230 real pages from six wikis that must write back byte for byte. Parsing stays linear in the length of the page, also on the pathological shapes the tests throw at it.
What a wiki is configured with changes how its wikitext reads, so a wiki can say:
val options = wiki.parseOptions()
val rules = wiki.titleRules()
val code = Wikitext.parseExact(page.text, options)
code.templates(listOf("l", "link"), rules)
code.ranges()[code.templates().first()]parseOptions() holds the wiki's extension tags and URL schemes, and titleRules() its namespace
names, first-letter case and parser functions. With the rules, templates finds {{l}},
{{Template:l}} and {{subst:l}} alike, and treats {{L}} as the same template only where the
wiki does, which Wiktionary does not. Without rules, names are compared ignoring the case of their
first letter. Template.key is the page a template transcludes, resolved as MediaWiki resolves
it. parseExact throws if the tree would not write back as given, for a bot about to save an edit
built on it. ranges() says where each node, parameter and attribute sits in the text.
Wikitext.parse(text) without options uses the settings of Wikimedia's wikis.
Templates edit themselves: mapTemplates, withParameter and withoutParameter. Extension
functions in com.fenakhay.kwikibot.wikitext.ops add section outlines and section editing,
categories and language links, and replaceText, which changes prose and leaves templates, link
targets, comments, tag attributes and the contents of tags such as <pre> alone unless its
TextScope says otherwise. Isbn validates ISBNs, CosmeticChanges applies named passes with
nothing on by default, and wiki.signatures() reads signature timestamps.
What the parser does not do is expand templates. It reads the page as written, so markup that a
template produces, such as a table row or a category, is not in the tree. wiki.pages.expandText
and wiki.renderer ask the wiki for the expanded or rendered result.
val page = Wikitext.parse(text)
val table = page.tables().first()
val heights = table.column("Height").map { it?.text }
val etna = table.rows[1].cells[0]
val renamed = etna.withContent("[[Mount Etna]]")
val longer = renamed.tables().first().withRow("Fuji", "3,776 m")
val written =
TableBuilder()
.sortable()
.caption("Volcanoes")
.header("Name", "Height")
.row("Etna", "3,357 m")
.build()tables() reads the {| … |} tables of a page as MediaWiki's own table pass does: markup at the
start of a line, || and !! for further cells, attributes before a cell's first |, {{!}} as a
pipe, cells over several lines, rows with no |-, and tables left open or never opened. A table
nested in a cell is reached from that cell's tables(). Each Cell has its content as Markup
and its text, and a table its grid with rowspan and colspan laid out, its headers and
column(name). The view is worked out over the parsed nodes when asked for, so parsing costs the
same as before. It is checked against MediaWiki's rendering of the corpus's tables, the limits below
aside, and against every table on its real pages.
Every edit, withContent, withAttributes, withRow, withRowAt, withoutRow and withColumn,
returns the whole page with only that part of the table rewritten, every other node the instance it
was, and new rows in the table's own style. Text that would read as table markup, such as a || b,
is escaped; what cannot be, and markup that would change the table some other way, is refused with
an IllegalArgumentException. TableBuilder writes new tables with the same escaping.
A table, row or cell that a template writes, such as {{table start}}, is not in the page as
written, and so not among its tables. HTML <table> markup is read as tags, not as a table. A
|| inside bold text or a link is read as part of it, where MediaWiki would split the cell there.
val report = wiki.botRun(wiki.stopPage()) {
source(wiki.lists.transclusions(wiki.ref("Template:old")))
transform { page ->
val code = Wikitext.parse(page.text)
if (code.templates("old").isEmpty()) skip("no {{old}} left")
Edit(code.mapTemplates("old") { it.copy(name = Markup.of("new")) }.serialize(), "old → new")
}
exclusionPolicy = BotPolicy(wiki.identity.name)
stopPolicy = StopPolicy.page(wiki.pages, wiki.ref("User:MyBot/Stop"))
onOutcome = { outcome ->
if (outcome is PageOutcome.Pending) {
println(Diffs.unified(outcome.before, outcome.edit.text, outcome.ref.title.toString()))
}
}
}
println("${report.pending} pages would change, ${report.skipped} skipped")botRun reads pages from a source, hands each to the transform, and saves what comes back. It
computes edits without sending them until dryRun = false, so a new bot shows you its diffs before
it touches a page. It reads ahead with bounded concurrency and writes one page at a time. Before
saving on a Wikimedia wiki, read Running on Wikimedia wikis.
The transform returns an Edit, or null to leave the page alone, which is the normal answer:
most pages a bot looks at need nothing done. skip(reason) leaves it alone with a reason, which
onOutcome and the skip log receive and the report counts. The transform runs on
Dispatchers.Default, several pages at once, so anything it shares needs its own protection. An
exception from it, other than a WikiError, ends the run.
A saving run checks the stop policy before it starts and before every save, and fails closed.
StopPolicy.page lets the bot go on only while the page says false, so create the page with that
text before the first saving run: a missing or empty page stops the bot. A dry run does not read
it. With an exclusionPolicy, a page whose {{bots}} or {{nobots}} excludes the bot is skipped
before the transform runs.
fun main(args: Array<String>) = botMain(args) {
val report = wiki.botRun(options.stopPolicy(wiki)) {
options.applyTo(this, wiki)
config.applyTo(this)
source(PageSourceSpec.parseAll(arguments).pages(wiki))
transform { page -> Edit(page.text.replace("{{old", "{{new"), "old → new") }
exclusionPolicy = BotPolicy(wiki.identity.name)
onOutcome = options.runLog(wiki)::record
}
if (report.clean && !report.stopped) 0 else 1
}oldtonew transcludes:Template:old --diff-log diffs.log --skip-log skips.jsonl
oldtonew transcludes:Template:old --save --limit 20 --stop-page User:MyBot/StopbotMain needs a kwikibot.toml, and logs in as the account its [login] or [oauth] names,
also for a dry run, unless botMain(args, anonymous = true) allows reading without one. With this
file, the bot password is read from KWIKIBOT_PASSWORD:
[bot]
name = "MyBot"
contact = "https://en.wiktionary.org/wiki/User:MyBot"
[wiki]
lang = "en"
family = "wiktionary"
[login]
account = "MyBot"
botName = "oldtonew"It reads the common flags (--save, --limit, --stop-page, --diff-log, --skip-log,
--state, --resume, --trial, --read-delay, --write-delay and the rest) and hands everything
else to the bot as arguments, unchecked, so a mistyped flag reaches the bot rather than an error.
Page sources are the bot's to read: PageSourceSpec.parseAll takes cat:English lemmas,
transcludes:Template:old, page:volcano, search:…, file:titles.txt and the rest, and
options.runLog(wiki) writes the diff and skip logs the flags name. botMain refuses --save
unless the account is in the bot group or --trial says the run is a trial, says on standard error
which account it runs as, and exits with the status the block returns. When the bot cannot start,
the reason is one line on standard error and the status is 1, or 2 for a common flag with a missing
or bad value.
There is no command base class to subclass, because how a program starts is the program's own
business. botMain is one way to assemble the pieces; CommonOptions.parse, PageSourceSpec,
RunLog, Progress and reportTo are there for a bot that wants its own. The
example bot runs on en.wiktionary and assembles its run itself,
without botMain, in one file.
A run can last hours, and the client and botRun are built for that:
429, readonly and database errors are retried with backoff, waiting as long
as the wiki's Retry-After asks when it asks for longer. Retries are logged at INFO, and
WikiConfig.listener hears every retry, every pause the wiki asks for and every re-login.nocreate. If someone edited the page in the meantime, the
wiki merges the two edits or, when it cannot, refuses the save. It also refuses a save to a page
deleted since it was read.state = RunState(dir) records each finished page, the revision it was decided on and each save
as they happen. resume = true carries on from there. The record assumes the same transform, so
after changing the bot, resume into a fresh directory; one directory belongs to one wiki.readBatch = 50, one request for fifty pages
instead of one each.RunLog writes unified diffs and JSON Lines skip records, and Progress keeps the rate, the time
left and what the run would change (or has saved) on one line that fits the terminal.kwikibot-fleet is for a project with several bots. Each implements one interface, and a fleet runs them in
lanes: at the same time as each other, one at a time within a bot, with a failure staying in its run.
class OldTemplateBot : Bot {
override val id = "oldtemplate"
override val description = "Takes {{old}} off the pages that use it."
override val schedule = Schedule.cron("0 3 * * *", "Europe/Berlin")
override suspend fun run(context: RunContext) {
val wiki = context.wiki
wiki.botRun(context.stopPolicy) {
context.options.applyTo(this, wiki)
source(wiki.lists.transclusions(wiki.ref("Template:old")))
transform { page -> Edit(page.text.replace("{{old}}", ""), "removing {{old}}") }
maxEdits = 500
onOutcome = context.outcomes
}
}
}
fun main(args: Array<String>) = fleetMain(args, listOf(OldTemplateBot(), NewsletterBot()))That program takes list, run <bot>, run --all and serve, which keeps running the bots with a
schedule, along with the common flags and kwikibot.toml. --all means every bot only straight after
run; after a bot's name it is that bot's own argument. Bots on one account share one login and one
budget: a second login over a bot password session is an error, and requests are paced per account, so
different accounts do not hold each other back. A bot naming an account logs in only with the
credentials under KWIKIBOT_<ACCOUNT>_, OAUTH_TOKEN, the four OAUTH_CONSUMER_KEY, OAUTH_CONSUMER_SECRET,
OAUTH_ACCESS_TOKEN and OAUTH_ACCESS_SECRET, or ACCOUNT, BOT_NAME and PASSWORD, and its
run fails if they are not there, or if it is told to save as an account outside the bot group without
--trial. They all answer to one stop page, read once a minute however many there are, and a run can
ask a person a question while the other lanes go on, with one question on screen at a time.
| Exit status | When |
|---|---|
| 0 |
list, or every run ended DONE and none stopped early |
| 1 | A run ended FAILED, CANCELLED or DECLINED, or stopped early, each with a line saying why; or the bot could not start |
| 2 | A bad command line |
Fleet is the host underneath, for embedding: start, cancel, runs and events, knowing nothing
of terminals or storage, so a dashboard or a store of past runs can sit on top. Nothing is kept once the
process ends. A bot's tests need no fleet: bot.run(RecordingContext(FakeWiki(pages))) writes down what
it said.
val wikidata = client.wiki(ApiEndpoint("www.wikidata.org"))
val douglasAdams = wikidata.wikibase().entity(EntityId("Q42")) ?: return
println(douglasAdams.label("en"))
val instanceOf = douglasAdams.best(EntityId("P31"))?.value
if (instanceOf is DataValue.EntityRef) println(instanceOf.id)Items, properties, lexemes and MediaInfo arrive as a sealed Entity, with statements that keep
novalue and somevalue distinct. A data value survives a round trip even when its type is one
this library has never heard of, so an edit preserves what it did not touch.
WikiHttpClient.create().use { http ->
val sparql = SparqlClient(http, config.userAgent)
val volcanoes = sparql.entityIds("SELECT ?item WHERE { ?item wdt:P31 wd:Q8072 } LIMIT 10")
EventStreams(http, config.userAgent).recentChanges()
.filter { it.wiki == "enwiktionary" }
.take(5)
.collect { println(it.title) }
}SPARQL runs against the Wikidata Query Service or any other endpoint; Commons' needs an OAuth
token, given as SparqlAuth.wcqs(token). Each event carries an offset, and passed back as
since, it lets a listener that restarts carry on where it stopped. PetScan and PagePile are page
sources, for the working lists editors assemble by hand. XmlDump streams a dump, plain,
gzip or bzip2, as a Sequence of pages on one thread, as a Flow in the dump's order on every core,
or fastest of all through scan, which hands out pages in no order:
XmlDump.pages(Path("enwiktionary-latest-pages-articles.xml.bz2"), namespaces = setOf(0))
.filter { "{{col|" in it.text }
.forEach { println(it.title) }
XmlDump.scan(Path("enwiktionary-latest-pages-articles.xml.bz2"), namespaces = setOf(0)) { page ->
if ("{{col|" in page.text) matches.add(page.title)
}bzip2 is decoded block by block on every thread, with or without a multistream index. gzip has no
boundaries to split on, so its decompression takes one thread and only the parsing shares out. A
truncated or corrupt dump throws DumpFormatException after yielding every whole page before the damage.
WikiHttpClient.create().use { http ->
val pageviews = Pageviews(http, config.userAgent)
val january = Instant.parse("2024-01-01T00:00:00Z")
val end = Instant.parse("2024-01-31T00:00:00Z")
val einstein = pageviews.perArticle("en.wikipedia", "Albert Einstein", january, end)
println(einstein.sumOf { it.views })
val wikipedia = client.wiki(ApiEndpoint("en.wikipedia.org"))
val volcanoes = wikipedia.lists.categoryMembers(wikipedia.ref("Category:Stratovolcanoes")).take(50).toList()
wikipedia.pageviews(pageviews, volcanoes, january, end, PageviewGranularity.MONTHLY)
.entries
.sortedByDescending { (_, counts) -> counts.sumOf { it.views } }
.take(10)
.forEach { (page, counts) -> println("${page.title}: ${counts.sumOf { it.views }}") }
pageviews.top("en.wikipedia", january).take(5).forEach { println("${it.rank}. ${it.title}") }
}Pageviews reads the Wikimedia REST API's pageviews: a page's views by day or month with
perArticle, a whole project's by hour, day or month with aggregate, and a project's most read
pages of a day or a month with top, for people unless agent says otherwise. The counts start in
July 2015, Pageviews.FIRST_DAY, and cover Wikimedia's wikis only. A page or a period with no data
is an empty list. wiki.pageviews counts a wiki's own pages under the project its server names,
en.wikipedia for en.wikipedia.org, and refuses any other wiki before sending anything, since the
API would answer that it has no data. A page is one request, so ranking a category of five thousand
pages is five thousand requests: wiki.pageviews asks four at a time, within the client's own limit
on requests in flight, three by default, and waits out a 429 or a 5xx as its RetryPolicy says,
Retry-After included. For many pages, a client of its own with a lower limit paces it further.
WikiHttpClient.create().use { http ->
val liftWing = LiftWing(http, config.userAgent)
EventStreams(http, config.userAgent).recentChanges()
.filter { it.matches("enwiki", setOf(0)) && it.type == "edit" && !it.isBot }
.mapNotNull { event -> event.revisionId?.let { event.title to RevisionId(it) } }
.collect { (title, revision) ->
val damaging = liftWing.damaging("enwiki", revision).scoreOrNull() ?: return@collect
if (damaging.trueProbability > 0.9) println("$title may be damaging: $revision")
}
}LiftWing asks Wikimedia's machine-learning service, which replaced ORES, about revisions:
damaging and goodfaith for the 33 wikis with those models, revertRisk for any Wikipedia,
language-agnostic unless RevertRiskModel.MULTILINGUAL is asked for, and articleQuality and
draftQuality where a wiki has them. predict(model, body) answers any other model's JSON as it is.
Every score is advisory, a classifier's guess from what editors did before, so a bot that acts on one
should use a cautious threshold and leave the last word to a person. A revision LiftWing cannot score,
deleted, hidden, missing, or for revert risk the first of its page, is a LiftWingResult.Unscorable
saying why, and a model or wiki LiftWing does not have throws WikiError.Api with the code
nosuchmodel. A wiki's own revisions are scored a few at a time, under the prefix and language its
site information gives, and any wiki that is not Wikimedia's is refused before anything is sent:
val wikipedia = client.wiki(ApiEndpoint("en.wikipedia.org"))
wikipedia.revertRisk(liftWing, listOf(RevisionId(1383620073))).forEach { (revision, result) ->
when (result) {
is LiftWingResult.Scored -> println("$revision: ${result.score.probability}")
is LiftWingResult.Unscorable -> println("$revision: ${result.reason}")
}
}Without a token, LiftWing allows 50,000 requests an hour and 15 a second, counted by address or user
agent. An OAuth 2.0 access token doubles the hourly limit and raises the other to 100 a second: pass it
as LiftWing(http, userAgent, authorization = { token }), on a client of its own. A 429 or a 5xx is
waited out as the RetryPolicy says, Retry-After included.
class VolcanoBotTest {
@Test
fun savesTheNewText() = runTest {
val pages = FakePageService("volcano" to "old")
val wiki = FakeWiki(pages)
val report = wiki.botRun(StopPolicy.NONE) {
source(flowOf(wiki.ref("volcano")))
transform { page -> Edit(page.text + " more", "adding") }
dryRun = false
}
assertEquals(1, report.saved)
assertEquals("old more", pages.text("volcano"))
}
}FakeWiki is a Wiki over pages held in memory, so a bot's own tests run in milliseconds and
need no network. Only the pages are faked: any other service throws unless the test passes one in,
so a test gives the run a fixed source, as above, or a lists of its own. That is deliberate: a
fake that answered every listing with nothing would let a bot that does nothing pass.
FakePageService records every edit and can be told to refuse some. MockTransport answers raw
API requests from JSON, for testing at the level of the protocol, and FailOnDeprecation turns any
deprecation warning a wiki sends into a test failure.
kwikibot init-config writes a commented starter file. kwikibot looks for one in the working
directory, then the home directory, then at $KWIKIBOT_CONFIG, then in
$XDG_CONFIG_HOME/kwikibot (or ~/.config/kwikibot), then in %APPDATA%\kwikibot.
| Section | Keys |
|---|---|
[bot] |
name, version, contact: the user agent |
[wiki] |
lang and family, or server and scriptPath
|
[throttle] |
read, write
|
[login] |
account, botName, passwordEnv, optional
|
[oauth] |
tokenEnv, username, optional; used in place of [login] when its token is set |
[oauth1] |
consumerKeyEnv, consumerSecretEnv, accessTokenEnv, accessSecretEnv, username, optional; used in place of [login] when its variables are set, and refused beside an [oauth] whose token is set |
[run] |
readConcurrency, writeConcurrency, readBatch
|
[http] |
maxConcurrentRequests, maxIdleConnections, timeout
|
[retry] |
maxRetries, initialDelay, maxDelay, relogins
|
[cache] |
path, ttl
|
| top level | maxlag |
Durations are written "100ms" or "10s". An unknown key is a warning, not an error. Without a
[wiki] section the wiki is en.wiktionary, and [oauth] is used in place of [login] only when
its token is set. The file holds no secrets: passwordEnv and tokenEnv name the environment
variables to read them from, KWIKIBOT_PASSWORD and KWIKIBOT_OAUTH_TOKEN unless they say
otherwise, and [oauth1] names four, KWIKIBOT_OAUTH_CONSUMER_KEY, KWIKIBOT_OAUTH_CONSUMER_SECRET,
KWIKIBOT_OAUTH_ACCESS_TOKEN and KWIKIBOT_OAUTH_ACCESS_SECRET unless it says otherwise. Since the working and home directories are searched first, $KWIKIBOT_CONFIG does not
override a file in either.
The library logs through kotlin-logging, each line under
the name of the class that logs it, such as com.fenakhay.kwikibot.net.auth.LoginManager. The
names are the same on every platform. Where the lines go is not:
org.slf4j:slf4j-simple:2.0.20.INFO up.INFO up.A command-line bot on macOS, Linux or Windows usually wants its log on standard error, out of the
way of what it prints. In nativeMain, with io.github.oshai:kotlin-logging:8.0.4 as a
dependency of your own:
@OptIn(ExperimentalForeignApi::class)
object StandardError : Appender {
override fun log(loggingEvent: KLoggingEvent) {
fputs(KotlinLoggingConfiguration.direct.formatter.formatMessage(loggingEvent) + "\n", stderr)
}
}
fun quietLogging() {
KotlinLoggingConfiguration.loggerFactory = DirectLoggerFactory
KotlinLoggingConfiguration.direct.logLevel = Level.WARN
KotlinLoggingConfiguration.direct.appender = StandardError
}DirectLoggerFactory is already the default on Linux and Windows; macOS needs it set.
kotlin-logging also prints a line of its own when it starts. logStartupMessage = false turns it
off, as does KOTLIN_LOGGING_STARTUP_MESSAGE=false in the environment.
A bot on a Wikimedia wiki runs under the bot policy of that wiki, and under Wikimedia's User-Agent policy, API etiquette and rate limits. Before saving anything:
botMain refuses --save for an account outside the bot group unless --trial says this is
one: --save --trial --limit 20. A trial needs --limit, which counts pages taken from the
source, not edits, so --limit 20 works through twenty pages however many it changes.kwikibot whoami prints
bot: true once the flag is there.false until someone changes it, and an
operator who answers on the bot's talk page.What kwikibot does without being asked, and where to change it:
| Default | Setting | |
|---|---|---|
| User agent | Required, with a contact | WikiConfig.userAgent |
maxlag |
5 seconds | WikiConfig.maxlag |
| Pacing | At least 100 ms between reads and 10 s between writes | WikiConfig.throttle |
| Retries | 5, waiting 1, 2, 4, 8 and 16 s, or as long as Retry-After asks |
WikiConfig.retry |
| Session loss | Logs back in, up to 3 times in ten minutes | WikiConfig.relogins |
| Saving | A dry run |
dryRun, --save
|
| Requests in flight | 3 at a time, to every wiki of a client together |
HttpSettings.maxConcurrentRequests, --http-connections
|
| Concurrency of a run | 4 pages read and 1 written at a time |
readConcurrency, writeConcurrency
|
{{nobots}} |
Not checked | exclusionPolicy |
| Stop page | None |
stopPolicy, --stop-page
|
| Edit conflicts | Read again and transformed again, up to 2 times | conflictRetries |
Wikimedia asks for no more than three requests at a time, and a client sends no more than that
however many pages a run works on: a request with no room waits for one in flight to be answered.
Raise maxConcurrentRequests only for a wiki whose operators allow more. Wikimedia also holds an
account without a bot flag to a number of requests a minute that the default read pacing can exceed.
A request over the limit is answered 429, and kwikibot waits as long as the answer asks. Every
wiki opened on one WikiClient shares one throttle, so a pause Wikidata asks for holds back
Wiktionary too.
npm install kwikibotThe package is CommonJS with TypeScript definitions, and needs Node 26 or newer. Its everyday names,
such as connect, BotPassword and parse, come from "kwikibot", and the rest from the area they
belong to, such as kwikibot/wikitext or kwikibot/fleet:
import { connect } from "kwikibot";
const contact = "https://en.wiktionary.org/wiki/User:MyBot";
const wiki = await connect("en.wiktionary.org", contact);
const page = await wiki.page("volcano");
for (const template of page.wikitext().templates("col")) {
console.log(template.parameters.map((p) => p.value));
}
let seen = 0;
for await (const title of wiki.categoryMembers("English lemmas")) {
console.log(title);
if (++seen === 10) break;
}
wiki.close();page resolves to null for a page that does not exist. Listings are async iterables that fetch
as the loop asks, so break ends the paging. A bot run takes a transform that returns the page's
new text, or null to leave the page alone, and may return a promise of either:
import { BotPassword, connect, parse } from "kwikibot";
const login = new BotPassword("MyBot", "task", process.env.MYBOT_PASSWORD);
const bot = await connect("en.wikipedia.org", contact, login);
const report = await bot.run(
["Mount Etna", "Mount Vesuvius"],
"adding [[Category:Volcanoes]]",
(text) => {
const code = parse(text);
if (code.categories().includes("Volcanoes")) return null;
return code.addCategory("Volcanoes").toString();
},
false,
(change) => console.log(change.diff),
);
console.log(`${report.pending} pages would change`);
bot.close();The run above only computes its edits; true as the fourth argument saves them. A page the wiki
refuses is a problem in the report. A transform that throws, or returns anything but text or
null, stops the run, and its promise rejects with that error.
A run from JavaScript takes an array of titles; to run over a listing, collect it first with
Array.fromAsync. It has no stop page and does not check {{nobots}}, so a transform that should
respect it returns null when code.templates("nobots").length is not zero. Pacing is fixed at
100 ms between reads and 10 s between saves.
connect takes the wiki's host name, and a fourth argument for a scriptPath other than /w;
detect(url, contact) finds both from any page URL. Requests go out as
kwikibot/<version> with your contact; the bot's own name cannot be set.
The package covers the parser (parse), titles and signature times, reading and editing pages,
category members, search, transclusions and history, Wikibase entities, SPARQL (sparql) and bot
runs. Revision ids are numbers and lists are arrays. Everything else in the library is for Kotlin.
It runs in browsers too, through a bundler, where logging in works only from a page on the wiki's
own site (see Platforms). It logs to the console from INFO up, the login included,
and the level cannot be set from JavaScript. The package is built from the
kwikibot-js directory.
kwikibot answers the first question of any misbehaving bot: is it me or the wiki? It is a
Kotlin/Native binary, so it needs no JDK and starts in milliseconds.
brew install fenakhay/tap/kwikibotscoop bucket add kwikibot https://github.com/fenakhay/kwikibot
scoop install kwikibotDebian and RPM packages for x64 Linux are attached to each
release, as are plain archives for macOS on Apple
Silicon, Linux and Windows. SHA256SUMS covers every file.
$ kwikibot --help
Usage: kwikibot [<options>] <command> [<args>]...
Talk to a MediaWiki wiki.
Commands:
whoami Show the account the wiki sees.
get Print the wikitext of a page.
login Log in and report the result.
init-config Write a starter kwikibot.toml.
config Show the configuration in effect.
detect Find the api.php of the wiki at a URL.
version Show the library and runtime versions.
$ kwikibot detect https://en.wiktionary.org
api: https://en.wiktionary.org/w/api.php
server: en.wiktionary.org
scriptPath: /w
whoami and login say what the wiki makes of your credentials, config shows the configuration
actually in effect, and version gives you the line to paste into a bug report. The commands that
talk to a wiki take --lang, --family, --contact, --config, --account and --bot-name.
With the last two the password is read from KWIKIBOT_PASSWORD, and without them the login comes
from kwikibot.toml. Without a kwikibot.toml, --contact is required. It logs to standard
error.
| MediaWiki | |
|---|---|
Wikimedia production, the weekly -wmf builds |
Supported. The live tests read from it, and the record of its API is checked weekly. |
| The current LTS and the latest release, 1.43 and 1.46 today | Supported. CI starts each in Docker and runs LocalWikiTest against it, failing on any warning the wiki gives. |
| 1.39 to 1.42 | Supported, not tested in CI. A value MediaWiki has since renamed is sent the way the wiki takes it. Before 1.41, opening a wiki draws one harmless warning, for a site property the release does not have. |
| Older than 1.39 | Not supported. Opening one logs a warning instead of refusing. |
The library checks a version number only where it cannot ask the wiki. A renamed value is chosen
from the wiki's paraminfo instead, so a third-party wiki on an old release is handled the same
way as Wikimedia's weekly builds.
kwikibot looks for them in three places:
deprecation.
WikiConfig.onWarning hears every warning and by default logs each one once, deprecations at
WARN. kwikibot avoids deprecated values, so a deprecation warning means something needs fixing.api-surface.tsv records the API of five Wikimedia wikis, and the build checks every value
kwikibot sends against it. A weekly job compares the file with the wikis and reports what
changed, marking anything kwikibot uses.When MediaWiki deprecates the value behind one of kwikibot's enum entries, the entry is marked
@Deprecated with its replacement, and keeps working because it is sent as the replacement. When
the value is gone from every reference wiki, the deprecation becomes an error, and the entry is
removed at the next major release. Every message has the same form:
MediaWiki 1.46 (T319141) deprecated parse prop=sections; use TOC_DATA. Deprecated since kwikibot 1.2.0.
Kotlin's ABI validation
records the public API of every published module, in */api/*.api for the JVM and
*/api/*.klib.api for the other platforms, and the build fails on any change it does not expect.
CHANGELOG.md lists every release's changes, its deprecations and what is pending
removal. 1.4.0, the release that made kwikibot multiplatform, breaks compatibility with 1.3 despite
its number; the changelog says how.
The build runs on JDK 25, which has to be installed; the published JVM jars target Java 21. Gradle
downloads the rest on the first build: the Kotlin/Native toolchain into ~/.konan, and Node.js 26
and Yarn for the JavaScript tests.
./gradlew checkThat runs every module's tests on the JVM, on Node and as a native binary for the machine it runs on, and the npm package's in headless Chrome as well, which has to be installed. It also runs detekt, Spotless, the ABI check and the coverage gates. Coverage is measured on the JVM, which runs every common test. The macOS targets build only on a Mac. A Mac compiles and links the Linux and Windows tests without running them; CI runs them on Linux and Windows.
Tests never touch the network: they replay recorded API responses. The live tests run on the JVM,
read from production wikis and write only to test.wikipedia.org sandbox pages:
./gradlew liveTestThe same tests can run against a throwaway MediaWiki in Docker, where they may write anywhere. Any
tag of the official image works, such as lts, latest or 1.39:
bash .github/mediawiki/start.sh lts
KWIKI_MEDIAWIKI=http://localhost:8080 ./gradlew :kwikibot-client:liveTest --tests '*LocalWikiTest'
docker rm -f kwikibot-mediawikistart.sh also lets the wiki upload from URLs on localhost, the container itself, and gives a sysop
the upload_by_url right, so LocalWikiTest has the wiki fetch a file it serves.
LocalOAuth1Test logs in to that wiki with OAuth 1.0a, edits and uploads, and checks that a wrong secret
is refused. The image does not bundle the OAuth extension, so oauth.sh clones the branch matching the
release into the container, installs it and prints an owner-only consumer for Admin as variables:
bash .github/mediawiki/start.sh lts
eval "$(bash .github/mediawiki/oauth.sh 8080)"
KWIKI_MEDIAWIKI=http://localhost:8080 ./gradlew :kwikibot-client:liveTest --tests '*LocalOAuth1Test'
docker rm -f kwikibot-mediawikiThe parser has benchmarks on the JVM and as a native macOS binary, over the same 230 real pages the round-trip test replays, with a committed baseline:
./gradlew :kwikibot-benchmarks:compareToBaselineThat fails if anything got slower by more than its error bars. The committed baseline comes from
one machine, so run recordBaseline on yours before comparing. Memory is measured separately, with
measureAllocations for the bytes allocated per character parsed and runRetention for how much
of a finished run is still held, since a benchmark that runs faster can still be the one that will
not fit.
MPL-2.0. See LICENSE.
A Kotlin Multiplatform library for MediaWiki bots and tools: read and edit any wiki, parse wikitext that writes back byte for byte, and run bots that are dry runs until told otherwise. On the JVM, macOS, Linux, Windows and JavaScript, with an npm package and a command-line tool.
commonMain code.Flows that page only as far as you collect.{{nobots}}; resumable state, diffs and skip logs.api.php. Wikibase entities, SPARQL, EventStreams, Wikimedia page views, PetScan,
PagePile and XML dumps, read on every core.kwikibot-testkit gives your bot a fake wiki.docs/comparison.md sets kwikibot beside pywikibot, mwclient, mwn, m3api and three wikitext parsers, feature by feature, with parsing speed, memory and start-up.
import com.fenakhay.kwikibot.client.Family
import com.fenakhay.kwikibot.client.WikiClient
import com.fenakhay.kwikibot.client.WikiConfig
import com.fenakhay.kwikibot.model.LangCode
import com.fenakhay.kwikibot.net.UserAgent
import com.fenakhay.kwikibot.wikitext.Wikitext
suspend fun main() {
val config = WikiConfig(UserAgent("MyBot", "1.0", "https://en.wiktionary.org/wiki/User:MyBot"))
WikiClient(config).use { client ->
val wiki = client.wiki(LangCode("en"), Family.WIKTIONARY)
val page = wiki.pages.content(wiki.ref("volcano")) ?: return
val code = Wikitext.parse(page.text)
println("${page.ref.title} at revision ${page.revisionId}")
println("${code.templates().size} templates, ${code.wikilinks().size} links")
for (template in code.templates("col")) {
println(template.parameters.drop(1).map { it.value.text.trim() })
}
}
}volcano at revision 93733515
287 templates, 15 links
[hypervolcano, paleovolcano, pneumonoultramicroscopicsilicovolcanoconiosis, vog, volcanist, …]
The user agent is required, and its contact must be somewhere the wiki's operators can reach you: Wikimedia blocks requests without one. Reading needs no account, and parsing touches no network.
kwikibot is on Maven Central. Gradle picks the build for each platform from one set of
coordinates, so a multiplatform project depends on it once, in commonMain:
kotlin {
sourceSets {
commonMain.dependencies {
implementation("com.fenakhay.kwikibot:kwikibot:1.4.0")
}
commonTest.dependencies {
implementation("com.fenakhay.kwikibot:kwikibot-testkit:1.4.0")
}
}
}A JVM-only project uses the same coordinates:
dependencies {
implementation("com.fenakhay.kwikibot:kwikibot:1.4.0")
testImplementation("com.fenakhay.kwikibot:kwikibot-testkit:1.4.0")
}With a version catalog:
[versions]
kwikibot = "1.4.0"
[libraries]
kwikibot = { module = "com.fenakhay.kwikibot:kwikibot", version.ref = "kwikibot" }
kwikibot-testkit = { module = "com.fenakhay.kwikibot:kwikibot-testkit", version.ref = "kwikibot" }implementation(libs.kwikibot)Maven does not read Gradle's metadata, so a Maven project names the JVM artifact, which ends in
-jvm:
<dependency>
<groupId>com.fenakhay.kwikibot</groupId>
<artifactId>kwikibot-jvm</artifactId>
<version>1.4.0</version>
</dependency>| Needs | |
|---|---|
| Kotlin/JVM project | Kotlin 2.4 or newer, and JDK 21 or newer to run. Gradle resolves kotlin-stdlib to 2.5.0-RC, the version kwikibot is built with |
| Kotlin/Native or Kotlin/JS project | Kotlin 2.5.0-RC or newer: Kotlin 2.4 cannot read klibs built by 2.5 |
| npm package | Node 22.3 or newer |
| A wiki | MediaWiki 1.39 or newer, see Compatibility |
On the JVM, the library needs one more thing from you: it logs through SLF4J and ships no
binding. Without one, SLF4J prints a warning and drops every line, retries and logins included. A
command-line bot adds runtimeOnly("org.slf4j:slf4j-simple:2.0.20"). See Logging.
On JavaScript, wiki.signatures() reads a wiki's time zone, such as Europe/Paris, through the
runtime's Intl, which Node and browsers ship, so nothing loads a time zone database.
kotlinx-datetime's own TimeZone.of knows no named zone there until @js-joda/timezone is
loaded, which matters only to code that calls it.
kwikibot brings in every module but the test kit. Each is also published on its own, for a
program that wants less: a tool that only parses wikitext has no reason to pull in Ktor.
| Module | What it holds |
|---|---|
kwikibot |
Everything below except the test kit |
kwikibot-model |
Value types with no I/O: Title, Namespace, PageRef, PageContent, Revision, the sealed WikiError and EditOutcome, signature timestamps |
kwikibot-wikitext |
The wikitext parser and editor, with no dependencies of its own |
kwikibot-net |
The HTTP transport: user agent, credentials and login, tokens, throttle, maxlag, retries, response cache |
kwikibot-protocol |
Reading api.php responses: errors and warnings, site info, paraminfo, continuation, page decoding |
kwikibot-client |
WikiClient, the Wiki handle and its services, EventStreams, SPARQL, page views, and XML dumps |
kwikibot-wikibase |
Entities, statements and data values |
kwikibot-bot |
botRun, botMain, page sources and filters, {{nobots}}, stop pages, run state, diffs, logs, plural forms, prompts |
kwikibot-fleet |
Many bots in one process: Bot, Fleet, fleetMain, shared sessions and a shared stop page, schedules. Not in the umbrella: depend on it to use it |
kwikibot-testkit |
FakeWiki, FakePageService, MockTransport and FailOnDeprecation, for testing a bot |
Depend on every module at the same version. Sources and KDoc are published with each artifact, so
the IDE has documentation to show and a debugger has code to step into. On Maven, kwikibot-js is
the umbrella's Kotlin/JS artifact, for Kotlin projects; the npm package is a separate thing,
below.
Every module builds for every platform below, with one API. Where they differ:
| JVM | macOS, Linux, Windows | Node | Browser | |
|---|---|---|---|---|
| HTTP engine | OkHttp | Darwin, Curl, WinHTTP | fetch | fetch |
Files: DiskCache, file uploads and downloads, BotConfig, RunState, diff and skip logs |
✓ | ✓ | ✓ | – |
XML dumps (XmlDump, DumpPage) |
✓ | ✓ | ✓, on worker threads | From bytes, on one thread |
botMain, TerminalPrompt, environment variables |
✓ | ✓ | ✓ | – |
Tested by ./gradlew check and CI |
✓ | ✓ | ✓ | the npm package, in headless Chrome |
Paths are kotlinx.io.files.Path. The file APIs
fail in a browser, which has no filesystem; there, environment variables read as unset and a
prompt reads no answer. On JavaScript only the
request timeout applies, since fetch has no connect or socket timeout, and every request also
carries Api-User-Agent, because a browser will not send a page's own User-Agent. From a page on
a site other than the wiki's, requests carry origin=*, which MediaWiki answers as it would a
logged-out reader. The KDoc of each connection setting says which engines honour it.
val config = WikiConfig(
userAgent = UserAgent("MyBot", "1.0", "https://en.wiktionary.org/wiki/User:MyBot"),
throttle = Throttle(read = 100.milliseconds, write = 10.seconds),
)
WikiClient(config, Credentials.fromEnvironment()).use { client ->
val wiktionary = client.wiki(LangCode("en"), Family.WIKTIONARY)
val wikidata = client.wiki(ApiEndpoint("www.wikidata.org"))
val yours = client.wiki(ApiEndpoint("wiki.example.org", scriptPath = "/w"))
}A WikiClient holds the connections, the login and the pacing, and hands out a Wiki for each
site it opens. Opening one logs in if there are credentials and reads the site's namespaces and
settings. wiki.identity says who the wiki thinks you are.
Family names the Wikimedia projects: Wikipedia, Wiktionary, Wikisource, Wikiquote, Wikibooks,
Wikinews, Wikiversity, Wikivoyage, Commons, Wikidata, Meta, mediawiki.org and test.wikipedia.org.
Any other wiki is an ApiEndpoint. When all you have is a URL, ApiDetector.discover finds its
api.php, which is what kwikibot detect does.
| For | Where it comes from | |
|---|---|---|
Credentials.BotPassword(account, botName, password) |
Most bots |
Special:BotPasswords on the wiki. account is the account name alone; kwikibot adds @botName itself |
Credentials.OAuth2(accessToken) |
A bot that runs as its owner and never asks anyone else to log in | An owner-only OAuth 2 consumer, proposed at Special:OAuthConsumerRegistration on Meta. The token is shown once |
Credentials.OAuth1(consumerKey, consumerSecret, accessToken, accessSecret) |
The same, for a tool or consumer that only has OAuth 1.0a | An owner-only OAuth 1.0a consumer, proposed at Special:OAuthConsumerRegistration on Meta. The four values are shown once |
Credentials.Anonymous |
Reading | The default |
Give a bot password the grants "High-volume (bot) access" and "Edit existing pages", and whatever
else the bot does, such as creating, moving or uploading. Without the first, even a flagged account
logs in without the bot and apihighlimits rights: its edits are not marked as a bot's, and a
query names at most 50 pages instead of 500.
An OAuth 2 token goes on every request as a bearer header, and the client keeps the session
cookies the wiki sets and sends them back with it, which Wikimedia asks of owner-only tokens so that
the account's own rate limit applies. Cookies are kept per client, so each account needs a
WikiClient of its own.
An OAuth 1.0a consumer needs no login round trip either. Every request is signed with HMAC-SHA1, as
RFC 5849 says, and the signature goes in an
Authorization: OAuth header. The signature covers the method, the URL and every query and form
parameter, but not a multipart body, so an upload is signed too. Each attempt at a request gets a
nonce and a timestamp of its own, so a retry is signed afresh. The two secrets only key the signature
and are never sent; a response cache keeps the account apart by a digest of the consumer key and the
access token. The wiki checks the signature against its own $wgServer and the path it was asked for,
so a proxy in front of it that rewrites either breaks it.
val credentials = Credentials.OAuth1(
consumerKey = System.getenv("KWIKIBOT_OAUTH_CONSUMER_KEY"),
consumerSecret = System.getenv("KWIKIBOT_OAUTH_CONSUMER_SECRET"),
accessToken = System.getenv("KWIKIBOT_OAUTH_ACCESS_TOKEN"),
accessSecret = System.getenv("KWIKIBOT_OAUTH_ACCESS_SECRET"),
)
WikiClient(config, credentials).use { client -> client.wiki(endpoint) }Keep secrets out of source and out of shell history. Credentials.fromEnvironment() reads
KWIKIBOT_OAUTH_TOKEN, or else KWIKIBOT_OAUTH_CONSUMER_KEY, KWIKIBOT_OAUTH_CONSUMER_SECRET,
KWIKIBOT_OAUTH_ACCESS_TOKEN and KWIKIBOT_OAUTH_ACCESS_SECRET, or else KWIKIBOT_ACCOUNT,
KWIKIBOT_BOT_NAME and KWIKIBOT_PASSWORD; KWIKIBOT_ACCOUNT is the username of either kind of
OAuth. With none of them set it gives Anonymous, and with some but not all of a group it fails
rather than guess. An OAuth 2 token and any of the OAuth 1.0a variables set together fail too, since
nothing says which was meant. The message names the variables, never what they hold. That is
for a program that builds its own WikiClient; botMain and the kwikibot tool take the account
from kwikibot.toml, which names the variable to read a secret from instead of holding it.
Every request from a logged-in client asserts its account. When the wiki ends the session, the
client logs back in and repeats the request instead of carrying on anonymously, up to
WikiConfig.relogins times in ten minutes.
val page = wiki.pages.content(wiki.ref("volcano")) ?: return
val updated = Wikitext.parse(page.text)
.mapTemplates("col") { it.withParameter("title", "terms derived from volcano") }
val outcome = wiki.pages.edit(page.ref) {
text = updated.serialize()
summary = "titling the derived terms"
baseRevision = page.revisionId
}
when (outcome) {
is EditOutcome.Saved -> println("saved as ${outcome.revision}")
is EditOutcome.NoChange -> println("already done")
is EditOutcome.Refused -> println("refused: ${outcome.detail}")
}The diff on the wiki is the parameter that changed and nothing else. baseRevision is what lets
the wiki refuse the save if somebody else edited the page since it was read; without it, the later
edit silently wins.
Every refusal a wiki can return is a case of EditOutcome.Refused rather than an exception:
Conflict, Protected, Filtered, RateLimited, CaptchaRequired, PermissionDenied,
PageStateChanged and Rejected, each saying whether trying again could help. Failures that are
not about the edit, such as a wiki that cannot be reached, are a sealed WikiError.
wiki.pages also moves, deletes, undeletes, protects, rolls back, undoes, purges and watches, and
edits sections: section = "new" leaves a message on a talk page. testActions asks whether an
edit would be allowed before one is computed, and on a wiki with pending changes,
wiki.extensions.flagged(refs) says whether an edit is live or queued.
wiki.lists.categoryMembers(wiki.ref("Category:English lemmas"), namespaces = setOf(Namespace.MAIN))
.take(10)
.collect { println(it.title) }Category members, backlinks, transclusions, links, search, prefix search, all pages, special
pages, watchlists, contributions, recent changes, logs and page histories are all cold Flows.
They fetch the next batch only when collection asks for it, so take(10) costs one request.
inNamespaces, titleMatching, subpageDepthAtMost and distinctPages compose on top of any of
them, filtering what arrives, so a narrow filter can page through a whole category before it finds
ten. A listing's own parameters, such as namespaces and startSortKey, filter on the wiki
instead. intersect and excluding read their second flow in full before the first.
val target = wiki.ref("File:Volcano.jpg")
val uploaded = wiki.files.upload(Path("volcano.jpg"), target, comment = "a volcano", text = "== Summary ==")
val fetched = wiki.files.uploadFromUrl("https://photos.example.org/volcano.jpg", target, comment = "a volcano")
for (outcome in listOf(uploaded, fetched)) {
when (outcome) {
is UploadOutcome.Uploaded -> println("uploaded as ${outcome.title}")
is UploadOutcome.Warned ->
if (!outcome.isDuplicateOnly) outcome.fileKey?.let { wiki.files.publishStashed(it, target) }
is UploadOutcome.Refused -> println("refused: ${outcome.detail}")
}
}upload sends a large file in chunks, and one whose answer is lost carries on where it stopped.
uploadFromUrl has the wiki fetch the file itself, which it does only with $wgAllowCopyUploads on,
from a host in $wgCopyUploadsDomains, for an account with the upload_by_url right; each of those
missing is an error rather than an outcome, since no file would fare differently. Either way a
warning is an outcome: the file waits under its fileKey, and publishStashed publishes it without
sending it again. info, usage, globalUsage, duplicatesOf and download read what is there.
val refused = wiki.users.setPreferences(mapOf("userjs-mybot-last-run" to "2026-11-04", "rclimit" to "500"))
val lastRun = wiki.users.preferences()["userjs-mybot-last-run"]
val drift = wiki.meta.serverTime() - Clock.System.now()
when (val sent = wiki.users.email("Example", "Your bot request", "It has been approved.")) {
is EmailOutcome.Sent -> println("sent")
is EmailOutcome.Undeliverable -> println("not sent: ${sent.reason}")
}
val global = wiki.extensions.globalUser("Example")
println("${global?.home}: ${global?.editCount} edits")setPreferences answers with the preferences the wiki refused, since MediaWiki saves the rest and only
warns about those; null puts one back to its default, and resetPreferences puts back a whole kind.
email sends a real email from the account to a person, so it is paced as a write and wikis rate-limit it.
An account that cannot be emailed is an outcome, and a sender who may not send is an error. serverTime
is the wiki's clock, asked for every time. globalUser reads a CentralAuth account, or null when there is
none, and lists its local accounts only when MERGED or UNATTACHED is asked for, since an account that has
been around for years has hundreds.
wiki.paramInfo reads the wiki's own description of its API: the limits an account gets, the
values a parameter takes, and which of those are deprecated. That is how a renamed value goes out
under the new name to a wiki that knows it and under the old name to one that does not.
requireRight, requireExtension and requireVersion fail at the start of a run instead of
halfway through. The extension services (GeoData, PageImages, TextExtracts, Linter, Echo, Thanks,
FlaggedRevs, AbuseFilter, PageTriage, ProofreadPage, CentralAuth and the rest) refuse when their
extension is absent.
val code = Wikitext.parse("==English==\n===Noun===\n{{en-noun}}\n\n{{col|en|vog|tephra}}\n")
code.templates("col").single().value("2")
code.wikilinks()
val english = code.outline().find("English", level = 2)
val edited = code
.mapTemplates("col") { it.withParameter("4", "lahar") }
.addCategory("Volcanoes")
edited.serialize()Wikitext.parse never fails: no input is rejected, and markup that does not parse, such as an
unclosed template or a stray bracket, stays literal text, which is what MediaWiki renders. Nor
does any input nest too deep: templates, links or tags nested a hundred thousand levels deep parse,
compare and write back like any other page, on every platform and whatever the thread's stack. Nodes
are immutable values, every edit returns a new Markup, and every node that was not touched keeps
the exact text it was parsed from.
The parser finds templates, parameters and headings where MediaWiki does, and ends HTML tags where MediaWiki's paragraphs and lists end them. It is checked against MediaWiki's own parse of 288 fragments, and against 230 real pages from six wikis that must write back byte for byte. Parsing stays linear in the length of the page, also on the pathological shapes the tests throw at it.
What a wiki is configured with changes how its wikitext reads, so a wiki can say:
val options = wiki.parseOptions()
val rules = wiki.titleRules()
val code = Wikitext.parseExact(page.text, options)
code.templates(listOf("l", "link"), rules)
code.ranges()[code.templates().first()]parseOptions() holds the wiki's extension tags and URL schemes, and titleRules() its namespace
names, first-letter case and parser functions. With the rules, templates finds {{l}},
{{Template:l}} and {{subst:l}} alike, and treats {{L}} as the same template only where the
wiki does, which Wiktionary does not. Without rules, names are compared ignoring the case of their
first letter. Template.key is the page a template transcludes, resolved as MediaWiki resolves
it. parseExact throws if the tree would not write back as given, for a bot about to save an edit
built on it. ranges() says where each node, parameter and attribute sits in the text.
Wikitext.parse(text) without options uses the settings of Wikimedia's wikis.
Templates edit themselves: mapTemplates, withParameter and withoutParameter. Extension
functions in com.fenakhay.kwikibot.wikitext.ops add section outlines and section editing,
categories and language links, and replaceText, which changes prose and leaves templates, link
targets, comments, tag attributes and the contents of tags such as <pre> alone unless its
TextScope says otherwise. Isbn validates ISBNs, CosmeticChanges applies named passes with
nothing on by default, and wiki.signatures() reads signature timestamps.
What the parser does not do is expand templates. It reads the page as written, so markup that a
template produces, such as a table row or a category, is not in the tree. wiki.pages.expandText
and wiki.renderer ask the wiki for the expanded or rendered result.
val page = Wikitext.parse(text)
val table = page.tables().first()
val heights = table.column("Height").map { it?.text }
val etna = table.rows[1].cells[0]
val renamed = etna.withContent("[[Mount Etna]]")
val longer = renamed.tables().first().withRow("Fuji", "3,776 m")
val written =
TableBuilder()
.sortable()
.caption("Volcanoes")
.header("Name", "Height")
.row("Etna", "3,357 m")
.build()tables() reads the {| … |} tables of a page as MediaWiki's own table pass does: markup at the
start of a line, || and !! for further cells, attributes before a cell's first |, {{!}} as a
pipe, cells over several lines, rows with no |-, and tables left open or never opened. A table
nested in a cell is reached from that cell's tables(). Each Cell has its content as Markup
and its text, and a table its grid with rowspan and colspan laid out, its headers and
column(name). The view is worked out over the parsed nodes when asked for, so parsing costs the
same as before. It is checked against MediaWiki's rendering of the corpus's tables, the limits below
aside, and against every table on its real pages.
Every edit, withContent, withAttributes, withRow, withRowAt, withoutRow and withColumn,
returns the whole page with only that part of the table rewritten, every other node the instance it
was, and new rows in the table's own style. Text that would read as table markup, such as a || b,
is escaped; what cannot be, and markup that would change the table some other way, is refused with
an IllegalArgumentException. TableBuilder writes new tables with the same escaping.
A table, row or cell that a template writes, such as {{table start}}, is not in the page as
written, and so not among its tables. HTML <table> markup is read as tags, not as a table. A
|| inside bold text or a link is read as part of it, where MediaWiki would split the cell there.
val report = wiki.botRun(wiki.stopPage()) {
source(wiki.lists.transclusions(wiki.ref("Template:old")))
transform { page ->
val code = Wikitext.parse(page.text)
if (code.templates("old").isEmpty()) skip("no {{old}} left")
Edit(code.mapTemplates("old") { it.copy(name = Markup.of("new")) }.serialize(), "old → new")
}
exclusionPolicy = BotPolicy(wiki.identity.name)
stopPolicy = StopPolicy.page(wiki.pages, wiki.ref("User:MyBot/Stop"))
onOutcome = { outcome ->
if (outcome is PageOutcome.Pending) {
println(Diffs.unified(outcome.before, outcome.edit.text, outcome.ref.title.toString()))
}
}
}
println("${report.pending} pages would change, ${report.skipped} skipped")botRun reads pages from a source, hands each to the transform, and saves what comes back. It
computes edits without sending them until dryRun = false, so a new bot shows you its diffs before
it touches a page. It reads ahead with bounded concurrency and writes one page at a time. Before
saving on a Wikimedia wiki, read Running on Wikimedia wikis.
The transform returns an Edit, or null to leave the page alone, which is the normal answer:
most pages a bot looks at need nothing done. skip(reason) leaves it alone with a reason, which
onOutcome and the skip log receive and the report counts. The transform runs on
Dispatchers.Default, several pages at once, so anything it shares needs its own protection. An
exception from it, other than a WikiError, ends the run.
A saving run checks the stop policy before it starts and before every save, and fails closed.
StopPolicy.page lets the bot go on only while the page says false, so create the page with that
text before the first saving run: a missing or empty page stops the bot. A dry run does not read
it. With an exclusionPolicy, a page whose {{bots}} or {{nobots}} excludes the bot is skipped
before the transform runs.
fun main(args: Array<String>) = botMain(args) {
val report = wiki.botRun(options.stopPolicy(wiki)) {
options.applyTo(this, wiki)
config.applyTo(this)
source(PageSourceSpec.parseAll(arguments).pages(wiki))
transform { page -> Edit(page.text.replace("{{old", "{{new"), "old → new") }
exclusionPolicy = BotPolicy(wiki.identity.name)
onOutcome = options.runLog(wiki)::record
}
if (report.clean && !report.stopped) 0 else 1
}oldtonew transcludes:Template:old --diff-log diffs.log --skip-log skips.jsonl
oldtonew transcludes:Template:old --save --limit 20 --stop-page User:MyBot/StopbotMain needs a kwikibot.toml, and logs in as the account its [login] or [oauth] names,
also for a dry run, unless botMain(args, anonymous = true) allows reading without one. With this
file, the bot password is read from KWIKIBOT_PASSWORD:
[bot]
name = "MyBot"
contact = "https://en.wiktionary.org/wiki/User:MyBot"
[wiki]
lang = "en"
family = "wiktionary"
[login]
account = "MyBot"
botName = "oldtonew"It reads the common flags (--save, --limit, --stop-page, --diff-log, --skip-log,
--state, --resume, --trial, --read-delay, --write-delay and the rest) and hands everything
else to the bot as arguments, unchecked, so a mistyped flag reaches the bot rather than an error.
Page sources are the bot's to read: PageSourceSpec.parseAll takes cat:English lemmas,
transcludes:Template:old, page:volcano, search:…, file:titles.txt and the rest, and
options.runLog(wiki) writes the diff and skip logs the flags name. botMain refuses --save
unless the account is in the bot group or --trial says the run is a trial, says on standard error
which account it runs as, and exits with the status the block returns. When the bot cannot start,
the reason is one line on standard error and the status is 1, or 2 for a common flag with a missing
or bad value.
There is no command base class to subclass, because how a program starts is the program's own
business. botMain is one way to assemble the pieces; CommonOptions.parse, PageSourceSpec,
RunLog, Progress and reportTo are there for a bot that wants its own. The
example bot runs on en.wiktionary and assembles its run itself,
without botMain, in one file.
A run can last hours, and the client and botRun are built for that:
429, readonly and database errors are retried with backoff, waiting as long
as the wiki's Retry-After asks when it asks for longer. Retries are logged at INFO, and
WikiConfig.listener hears every retry, every pause the wiki asks for and every re-login.nocreate. If someone edited the page in the meantime, the
wiki merges the two edits or, when it cannot, refuses the save. It also refuses a save to a page
deleted since it was read.state = RunState(dir) records each finished page, the revision it was decided on and each save
as they happen. resume = true carries on from there. The record assumes the same transform, so
after changing the bot, resume into a fresh directory; one directory belongs to one wiki.readBatch = 50, one request for fifty pages
instead of one each.RunLog writes unified diffs and JSON Lines skip records, and Progress keeps the rate, the time
left and what the run would change (or has saved) on one line that fits the terminal.kwikibot-fleet is for a project with several bots. Each implements one interface, and a fleet runs them in
lanes: at the same time as each other, one at a time within a bot, with a failure staying in its run.
class OldTemplateBot : Bot {
override val id = "oldtemplate"
override val description = "Takes {{old}} off the pages that use it."
override val schedule = Schedule.cron("0 3 * * *", "Europe/Berlin")
override suspend fun run(context: RunContext) {
val wiki = context.wiki
wiki.botRun(context.stopPolicy) {
context.options.applyTo(this, wiki)
source(wiki.lists.transclusions(wiki.ref("Template:old")))
transform { page -> Edit(page.text.replace("{{old}}", ""), "removing {{old}}") }
maxEdits = 500
onOutcome = context.outcomes
}
}
}
fun main(args: Array<String>) = fleetMain(args, listOf(OldTemplateBot(), NewsletterBot()))That program takes list, run <bot>, run --all and serve, which keeps running the bots with a
schedule, along with the common flags and kwikibot.toml. --all means every bot only straight after
run; after a bot's name it is that bot's own argument. Bots on one account share one login and one
budget: a second login over a bot password session is an error, and requests are paced per account, so
different accounts do not hold each other back. A bot naming an account logs in only with the
credentials under KWIKIBOT_<ACCOUNT>_, OAUTH_TOKEN, the four OAUTH_CONSUMER_KEY, OAUTH_CONSUMER_SECRET,
OAUTH_ACCESS_TOKEN and OAUTH_ACCESS_SECRET, or ACCOUNT, BOT_NAME and PASSWORD, and its
run fails if they are not there, or if it is told to save as an account outside the bot group without
--trial. They all answer to one stop page, read once a minute however many there are, and a run can
ask a person a question while the other lanes go on, with one question on screen at a time.
| Exit status | When |
|---|---|
| 0 |
list, or every run ended DONE and none stopped early |
| 1 | A run ended FAILED, CANCELLED or DECLINED, or stopped early, each with a line saying why; or the bot could not start |
| 2 | A bad command line |
Fleet is the host underneath, for embedding: start, cancel, runs and events, knowing nothing
of terminals or storage, so a dashboard or a store of past runs can sit on top. Nothing is kept once the
process ends. A bot's tests need no fleet: bot.run(RecordingContext(FakeWiki(pages))) writes down what
it said.
val wikidata = client.wiki(ApiEndpoint("www.wikidata.org"))
val douglasAdams = wikidata.wikibase().entity(EntityId("Q42")) ?: return
println(douglasAdams.label("en"))
val instanceOf = douglasAdams.best(EntityId("P31"))?.value
if (instanceOf is DataValue.EntityRef) println(instanceOf.id)Items, properties, lexemes and MediaInfo arrive as a sealed Entity, with statements that keep
novalue and somevalue distinct. A data value survives a round trip even when its type is one
this library has never heard of, so an edit preserves what it did not touch.
WikiHttpClient.create().use { http ->
val sparql = SparqlClient(http, config.userAgent)
val volcanoes = sparql.entityIds("SELECT ?item WHERE { ?item wdt:P31 wd:Q8072 } LIMIT 10")
EventStreams(http, config.userAgent).recentChanges()
.filter { it.wiki == "enwiktionary" }
.take(5)
.collect { println(it.title) }
}SPARQL runs against the Wikidata Query Service or any other endpoint; Commons' needs an OAuth
token, given as SparqlAuth.wcqs(token). Each event carries an offset, and passed back as
since, it lets a listener that restarts carry on where it stopped. PetScan and PagePile are page
sources, for the working lists editors assemble by hand. XmlDump streams a dump, plain,
gzip or bzip2, as a Sequence of pages on one thread, as a Flow in the dump's order on every core,
or fastest of all through scan, which hands out pages in no order:
XmlDump.pages(Path("enwiktionary-latest-pages-articles.xml.bz2"), namespaces = setOf(0))
.filter { "{{col|" in it.text }
.forEach { println(it.title) }
XmlDump.scan(Path("enwiktionary-latest-pages-articles.xml.bz2"), namespaces = setOf(0)) { page ->
if ("{{col|" in page.text) matches.add(page.title)
}bzip2 is decoded block by block on every thread, with or without a multistream index. gzip has no
boundaries to split on, so its decompression takes one thread and only the parsing shares out. A
truncated or corrupt dump throws DumpFormatException after yielding every whole page before the damage.
WikiHttpClient.create().use { http ->
val pageviews = Pageviews(http, config.userAgent)
val january = Instant.parse("2024-01-01T00:00:00Z")
val end = Instant.parse("2024-01-31T00:00:00Z")
val einstein = pageviews.perArticle("en.wikipedia", "Albert Einstein", january, end)
println(einstein.sumOf { it.views })
val wikipedia = client.wiki(ApiEndpoint("en.wikipedia.org"))
val volcanoes = wikipedia.lists.categoryMembers(wikipedia.ref("Category:Stratovolcanoes")).take(50).toList()
wikipedia.pageviews(pageviews, volcanoes, january, end, PageviewGranularity.MONTHLY)
.entries
.sortedByDescending { (_, counts) -> counts.sumOf { it.views } }
.take(10)
.forEach { (page, counts) -> println("${page.title}: ${counts.sumOf { it.views }}") }
pageviews.top("en.wikipedia", january).take(5).forEach { println("${it.rank}. ${it.title}") }
}Pageviews reads the Wikimedia REST API's pageviews: a page's views by day or month with
perArticle, a whole project's by hour, day or month with aggregate, and a project's most read
pages of a day or a month with top, for people unless agent says otherwise. The counts start in
July 2015, Pageviews.FIRST_DAY, and cover Wikimedia's wikis only. A page or a period with no data
is an empty list. wiki.pageviews counts a wiki's own pages under the project its server names,
en.wikipedia for en.wikipedia.org, and refuses any other wiki before sending anything, since the
API would answer that it has no data. A page is one request, so ranking a category of five thousand
pages is five thousand requests: wiki.pageviews asks four at a time, within the client's own limit
on requests in flight, three by default, and waits out a 429 or a 5xx as its RetryPolicy says,
Retry-After included. For many pages, a client of its own with a lower limit paces it further.
WikiHttpClient.create().use { http ->
val liftWing = LiftWing(http, config.userAgent)
EventStreams(http, config.userAgent).recentChanges()
.filter { it.matches("enwiki", setOf(0)) && it.type == "edit" && !it.isBot }
.mapNotNull { event -> event.revisionId?.let { event.title to RevisionId(it) } }
.collect { (title, revision) ->
val damaging = liftWing.damaging("enwiki", revision).scoreOrNull() ?: return@collect
if (damaging.trueProbability > 0.9) println("$title may be damaging: $revision")
}
}LiftWing asks Wikimedia's machine-learning service, which replaced ORES, about revisions:
damaging and goodfaith for the 33 wikis with those models, revertRisk for any Wikipedia,
language-agnostic unless RevertRiskModel.MULTILINGUAL is asked for, and articleQuality and
draftQuality where a wiki has them. predict(model, body) answers any other model's JSON as it is.
Every score is advisory, a classifier's guess from what editors did before, so a bot that acts on one
should use a cautious threshold and leave the last word to a person. A revision LiftWing cannot score,
deleted, hidden, missing, or for revert risk the first of its page, is a LiftWingResult.Unscorable
saying why, and a model or wiki LiftWing does not have throws WikiError.Api with the code
nosuchmodel. A wiki's own revisions are scored a few at a time, under the prefix and language its
site information gives, and any wiki that is not Wikimedia's is refused before anything is sent:
val wikipedia = client.wiki(ApiEndpoint("en.wikipedia.org"))
wikipedia.revertRisk(liftWing, listOf(RevisionId(1383620073))).forEach { (revision, result) ->
when (result) {
is LiftWingResult.Scored -> println("$revision: ${result.score.probability}")
is LiftWingResult.Unscorable -> println("$revision: ${result.reason}")
}
}Without a token, LiftWing allows 50,000 requests an hour and 15 a second, counted by address or user
agent. An OAuth 2.0 access token doubles the hourly limit and raises the other to 100 a second: pass it
as LiftWing(http, userAgent, authorization = { token }), on a client of its own. A 429 or a 5xx is
waited out as the RetryPolicy says, Retry-After included.
class VolcanoBotTest {
@Test
fun savesTheNewText() = runTest {
val pages = FakePageService("volcano" to "old")
val wiki = FakeWiki(pages)
val report = wiki.botRun(StopPolicy.NONE) {
source(flowOf(wiki.ref("volcano")))
transform { page -> Edit(page.text + " more", "adding") }
dryRun = false
}
assertEquals(1, report.saved)
assertEquals("old more", pages.text("volcano"))
}
}FakeWiki is a Wiki over pages held in memory, so a bot's own tests run in milliseconds and
need no network. Only the pages are faked: any other service throws unless the test passes one in,
so a test gives the run a fixed source, as above, or a lists of its own. That is deliberate: a
fake that answered every listing with nothing would let a bot that does nothing pass.
FakePageService records every edit and can be told to refuse some. MockTransport answers raw
API requests from JSON, for testing at the level of the protocol, and FailOnDeprecation turns any
deprecation warning a wiki sends into a test failure.
kwikibot init-config writes a commented starter file. kwikibot looks for one in the working
directory, then the home directory, then at $KWIKIBOT_CONFIG, then in
$XDG_CONFIG_HOME/kwikibot (or ~/.config/kwikibot), then in %APPDATA%\kwikibot.
| Section | Keys |
|---|---|
[bot] |
name, version, contact: the user agent |
[wiki] |
lang and family, or server and scriptPath
|
[throttle] |
read, write
|
[login] |
account, botName, passwordEnv, optional
|
[oauth] |
tokenEnv, username, optional; used in place of [login] when its token is set |
[oauth1] |
consumerKeyEnv, consumerSecretEnv, accessTokenEnv, accessSecretEnv, username, optional; used in place of [login] when its variables are set, and refused beside an [oauth] whose token is set |
[run] |
readConcurrency, writeConcurrency, readBatch
|
[http] |
maxConcurrentRequests, maxIdleConnections, timeout
|
[retry] |
maxRetries, initialDelay, maxDelay, relogins
|
[cache] |
path, ttl
|
| top level | maxlag |
Durations are written "100ms" or "10s". An unknown key is a warning, not an error. Without a
[wiki] section the wiki is en.wiktionary, and [oauth] is used in place of [login] only when
its token is set. The file holds no secrets: passwordEnv and tokenEnv name the environment
variables to read them from, KWIKIBOT_PASSWORD and KWIKIBOT_OAUTH_TOKEN unless they say
otherwise, and [oauth1] names four, KWIKIBOT_OAUTH_CONSUMER_KEY, KWIKIBOT_OAUTH_CONSUMER_SECRET,
KWIKIBOT_OAUTH_ACCESS_TOKEN and KWIKIBOT_OAUTH_ACCESS_SECRET unless it says otherwise. Since the working and home directories are searched first, $KWIKIBOT_CONFIG does not
override a file in either.
The library logs through kotlin-logging, each line under
the name of the class that logs it, such as com.fenakhay.kwikibot.net.auth.LoginManager. The
names are the same on every platform. Where the lines go is not:
org.slf4j:slf4j-simple:2.0.20.INFO up.INFO up.A command-line bot on macOS, Linux or Windows usually wants its log on standard error, out of the
way of what it prints. In nativeMain, with io.github.oshai:kotlin-logging:8.0.4 as a
dependency of your own:
@OptIn(ExperimentalForeignApi::class)
object StandardError : Appender {
override fun log(loggingEvent: KLoggingEvent) {
fputs(KotlinLoggingConfiguration.direct.formatter.formatMessage(loggingEvent) + "\n", stderr)
}
}
fun quietLogging() {
KotlinLoggingConfiguration.loggerFactory = DirectLoggerFactory
KotlinLoggingConfiguration.direct.logLevel = Level.WARN
KotlinLoggingConfiguration.direct.appender = StandardError
}DirectLoggerFactory is already the default on Linux and Windows; macOS needs it set.
kotlin-logging also prints a line of its own when it starts. logStartupMessage = false turns it
off, as does KOTLIN_LOGGING_STARTUP_MESSAGE=false in the environment.
A bot on a Wikimedia wiki runs under the bot policy of that wiki, and under Wikimedia's User-Agent policy, API etiquette and rate limits. Before saving anything:
botMain refuses --save for an account outside the bot group unless --trial says this is
one: --save --trial --limit 20. A trial needs --limit, which counts pages taken from the
source, not edits, so --limit 20 works through twenty pages however many it changes.kwikibot whoami prints
bot: true once the flag is there.false until someone changes it, and an
operator who answers on the bot's talk page.What kwikibot does without being asked, and where to change it:
| Default | Setting | |
|---|---|---|
| User agent | Required, with a contact | WikiConfig.userAgent |
maxlag |
5 seconds | WikiConfig.maxlag |
| Pacing | At least 100 ms between reads and 10 s between writes | WikiConfig.throttle |
| Retries | 5, waiting 1, 2, 4, 8 and 16 s, or as long as Retry-After asks |
WikiConfig.retry |
| Session loss | Logs back in, up to 3 times in ten minutes | WikiConfig.relogins |
| Saving | A dry run |
dryRun, --save
|
| Requests in flight | 3 at a time, to every wiki of a client together |
HttpSettings.maxConcurrentRequests, --http-connections
|
| Concurrency of a run | 4 pages read and 1 written at a time |
readConcurrency, writeConcurrency
|
{{nobots}} |
Not checked | exclusionPolicy |
| Stop page | None |
stopPolicy, --stop-page
|
| Edit conflicts | Read again and transformed again, up to 2 times | conflictRetries |
Wikimedia asks for no more than three requests at a time, and a client sends no more than that
however many pages a run works on: a request with no room waits for one in flight to be answered.
Raise maxConcurrentRequests only for a wiki whose operators allow more. Wikimedia also holds an
account without a bot flag to a number of requests a minute that the default read pacing can exceed.
A request over the limit is answered 429, and kwikibot waits as long as the answer asks. Every
wiki opened on one WikiClient shares one throttle, so a pause Wikidata asks for holds back
Wiktionary too.
npm install kwikibotThe package is CommonJS with TypeScript definitions, and needs Node 26 or newer. Its everyday names,
such as connect, BotPassword and parse, come from "kwikibot", and the rest from the area they
belong to, such as kwikibot/wikitext or kwikibot/fleet:
import { connect } from "kwikibot";
const contact = "https://en.wiktionary.org/wiki/User:MyBot";
const wiki = await connect("en.wiktionary.org", contact);
const page = await wiki.page("volcano");
for (const template of page.wikitext().templates("col")) {
console.log(template.parameters.map((p) => p.value));
}
let seen = 0;
for await (const title of wiki.categoryMembers("English lemmas")) {
console.log(title);
if (++seen === 10) break;
}
wiki.close();page resolves to null for a page that does not exist. Listings are async iterables that fetch
as the loop asks, so break ends the paging. A bot run takes a transform that returns the page's
new text, or null to leave the page alone, and may return a promise of either:
import { BotPassword, connect, parse } from "kwikibot";
const login = new BotPassword("MyBot", "task", process.env.MYBOT_PASSWORD);
const bot = await connect("en.wikipedia.org", contact, login);
const report = await bot.run(
["Mount Etna", "Mount Vesuvius"],
"adding [[Category:Volcanoes]]",
(text) => {
const code = parse(text);
if (code.categories().includes("Volcanoes")) return null;
return code.addCategory("Volcanoes").toString();
},
false,
(change) => console.log(change.diff),
);
console.log(`${report.pending} pages would change`);
bot.close();The run above only computes its edits; true as the fourth argument saves them. A page the wiki
refuses is a problem in the report. A transform that throws, or returns anything but text or
null, stops the run, and its promise rejects with that error.
A run from JavaScript takes an array of titles; to run over a listing, collect it first with
Array.fromAsync. It has no stop page and does not check {{nobots}}, so a transform that should
respect it returns null when code.templates("nobots").length is not zero. Pacing is fixed at
100 ms between reads and 10 s between saves.
connect takes the wiki's host name, and a fourth argument for a scriptPath other than /w;
detect(url, contact) finds both from any page URL. Requests go out as
kwikibot/<version> with your contact; the bot's own name cannot be set.
The package covers the parser (parse), titles and signature times, reading and editing pages,
category members, search, transclusions and history, Wikibase entities, SPARQL (sparql) and bot
runs. Revision ids are numbers and lists are arrays. Everything else in the library is for Kotlin.
It runs in browsers too, through a bundler, where logging in works only from a page on the wiki's
own site (see Platforms). It logs to the console from INFO up, the login included,
and the level cannot be set from JavaScript. The package is built from the
kwikibot-js directory.
kwikibot answers the first question of any misbehaving bot: is it me or the wiki? It is a
Kotlin/Native binary, so it needs no JDK and starts in milliseconds.
brew install fenakhay/tap/kwikibotscoop bucket add kwikibot https://github.com/fenakhay/kwikibot
scoop install kwikibotDebian and RPM packages for x64 Linux are attached to each
release, as are plain archives for macOS on Apple
Silicon, Linux and Windows. SHA256SUMS covers every file.
$ kwikibot --help
Usage: kwikibot [<options>] <command> [<args>]...
Talk to a MediaWiki wiki.
Commands:
whoami Show the account the wiki sees.
get Print the wikitext of a page.
login Log in and report the result.
init-config Write a starter kwikibot.toml.
config Show the configuration in effect.
detect Find the api.php of the wiki at a URL.
version Show the library and runtime versions.
$ kwikibot detect https://en.wiktionary.org
api: https://en.wiktionary.org/w/api.php
server: en.wiktionary.org
scriptPath: /w
whoami and login say what the wiki makes of your credentials, config shows the configuration
actually in effect, and version gives you the line to paste into a bug report. The commands that
talk to a wiki take --lang, --family, --contact, --config, --account and --bot-name.
With the last two the password is read from KWIKIBOT_PASSWORD, and without them the login comes
from kwikibot.toml. Without a kwikibot.toml, --contact is required. It logs to standard
error.
| MediaWiki | |
|---|---|
Wikimedia production, the weekly -wmf builds |
Supported. The live tests read from it, and the record of its API is checked weekly. |
| The current LTS and the latest release, 1.43 and 1.46 today | Supported. CI starts each in Docker and runs LocalWikiTest against it, failing on any warning the wiki gives. |
| 1.39 to 1.42 | Supported, not tested in CI. A value MediaWiki has since renamed is sent the way the wiki takes it. Before 1.41, opening a wiki draws one harmless warning, for a site property the release does not have. |
| Older than 1.39 | Not supported. Opening one logs a warning instead of refusing. |
The library checks a version number only where it cannot ask the wiki. A renamed value is chosen
from the wiki's paraminfo instead, so a third-party wiki on an old release is handled the same
way as Wikimedia's weekly builds.
kwikibot looks for them in three places:
deprecation.
WikiConfig.onWarning hears every warning and by default logs each one once, deprecations at
WARN. kwikibot avoids deprecated values, so a deprecation warning means something needs fixing.api-surface.tsv records the API of five Wikimedia wikis, and the build checks every value
kwikibot sends against it. A weekly job compares the file with the wikis and reports what
changed, marking anything kwikibot uses.When MediaWiki deprecates the value behind one of kwikibot's enum entries, the entry is marked
@Deprecated with its replacement, and keeps working because it is sent as the replacement. When
the value is gone from every reference wiki, the deprecation becomes an error, and the entry is
removed at the next major release. Every message has the same form:
MediaWiki 1.46 (T319141) deprecated parse prop=sections; use TOC_DATA. Deprecated since kwikibot 1.2.0.
Kotlin's ABI validation
records the public API of every published module, in */api/*.api for the JVM and
*/api/*.klib.api for the other platforms, and the build fails on any change it does not expect.
CHANGELOG.md lists every release's changes, its deprecations and what is pending
removal. 1.4.0, the release that made kwikibot multiplatform, breaks compatibility with 1.3 despite
its number; the changelog says how.
The build runs on JDK 25, which has to be installed; the published JVM jars target Java 21. Gradle
downloads the rest on the first build: the Kotlin/Native toolchain into ~/.konan, and Node.js 26
and Yarn for the JavaScript tests.
./gradlew checkThat runs every module's tests on the JVM, on Node and as a native binary for the machine it runs on, and the npm package's in headless Chrome as well, which has to be installed. It also runs detekt, Spotless, the ABI check and the coverage gates. Coverage is measured on the JVM, which runs every common test. The macOS targets build only on a Mac. A Mac compiles and links the Linux and Windows tests without running them; CI runs them on Linux and Windows.
Tests never touch the network: they replay recorded API responses. The live tests run on the JVM,
read from production wikis and write only to test.wikipedia.org sandbox pages:
./gradlew liveTestThe same tests can run against a throwaway MediaWiki in Docker, where they may write anywhere. Any
tag of the official image works, such as lts, latest or 1.39:
bash .github/mediawiki/start.sh lts
KWIKI_MEDIAWIKI=http://localhost:8080 ./gradlew :kwikibot-client:liveTest --tests '*LocalWikiTest'
docker rm -f kwikibot-mediawikistart.sh also lets the wiki upload from URLs on localhost, the container itself, and gives a sysop
the upload_by_url right, so LocalWikiTest has the wiki fetch a file it serves.
LocalOAuth1Test logs in to that wiki with OAuth 1.0a, edits and uploads, and checks that a wrong secret
is refused. The image does not bundle the OAuth extension, so oauth.sh clones the branch matching the
release into the container, installs it and prints an owner-only consumer for Admin as variables:
bash .github/mediawiki/start.sh lts
eval "$(bash .github/mediawiki/oauth.sh 8080)"
KWIKI_MEDIAWIKI=http://localhost:8080 ./gradlew :kwikibot-client:liveTest --tests '*LocalOAuth1Test'
docker rm -f kwikibot-mediawikiThe parser has benchmarks on the JVM and as a native macOS binary, over the same 230 real pages the round-trip test replays, with a committed baseline:
./gradlew :kwikibot-benchmarks:compareToBaselineThat fails if anything got slower by more than its error bars. The committed baseline comes from
one machine, so run recordBaseline on yours before comparing. Memory is measured separately, with
measureAllocations for the bytes allocated per character parsed and runRetention for how much
of a finished run is still held, since a benchmark that runs faster can still be the one that will
not fit.
MPL-2.0. See LICENSE.