Checking the archive (lib/checker.rb)

./blog.sh check is doctor's counterpart and deliberately a separate command: doctor answers "is this installation sound", reads a handful of config values and takes a second, while this walks every post and every media file there is. Rolled into one command the fast half would stop being run, which is the half somebody runs before every deploy.

It reads the content, not the built site. A finding has to name a post and a slug -- something to go and fix -- rather than a file under public.nosync, and it has to work before a build has ever run. Judging a link still needs to know which addresses a build would produce, so those are derived in the checker from the same rules build_blog.rb follows. Twenty questions, in one pass: a config/site.yml that is missing, empty, unparseable or unopenable -- the one question about something outside the archive, asked because this is what people run before a build and the build refuses all four; files the checker cannot read at all, posts whose date nothing can parse, posts whose text is not a list of blocks or whose blocks carry a picture, a poster, list items or formatting that is not a list, tags that are not a list, and posts whose slug or draft token is not one path segment -- all of them states the BUILD refuses to run on (or, for the slug, misplaces the page for), so a check that stayed quiet about them was calling an archive sound that was about to stop the site; post files lying where the build never reads a post (directly in posts/, or a level too deep), which are in the archive and on no page; files a queue move stepped aside and a crash left parked, which no listing shows and whose repair depends on whether the post inside them is still in the archive anywhere else; media a post asks for and hasn't got -- or has in a shape no reader can use, or has under a spelling that is not the one the post names, where a volume that folds letter case and unicode form renders the page and one that does not has the hole already; images whose stored dimensions are 1px or smaller, which the build drops together with their caption and would otherwise lose silently; internal links pointing at an address nothing on this site answers at; links written relative to the post rather than to the site, which a static host answers 200 with the page the reader is already on -- the one kind here that fails at nothing and is therefore invisible from everywhere else; media directories no post owns any more; files in a post's own directory the post no longer names; two series whose names sit an edit or two apart, which is usually one series and a typo; one old address claimed by two posts, where whichever renders last wins and the others' readers land on it; and two posts that would be served at one address, which the build refuses to run on at all -- so an archive in that state used to be called sound by the one tool whose job is to say otherwise; a post that looks like a copy of another -- x-2 beside x in the same year with the same title, moment and content (every block, not only the text), which is what importing an export back into an archive that already held it leaves behind, each copy under an address of its own so that nothing else here objects, and asked narrowly because a slug that ends in a number beside the same slug without one is ordinary; and a redirect_from the build will refuse to serve, which it says once in the middle of a build log and which the checker used to count among the addresses this site answers at, so a link into one passed as sound -- including an old address a live post has since taken; text carrying HTML entities where the characters themselves belong; and, on a site that publishes more than one language, two posts asking for one address inside one of them -- which the build resolves by serving one of the pair and dropping the other, and a language site.locales names that the engine has no locale file for, which stops the build of every language of the site, its own included; and site.fallback or site.ui_language kept in config/site.yml, where a language's own settings stood before 1.9 and the build now refuses them; and a config/site.<lang>.yml that nothing reads -- written for a language the site does not publish, carrying a key the engine does not know, or translating something site.yml does not have -- all of which look exactly like work that is done; and one whose menu or footer links go to other places than site.yml's (lib/language_file.rb, which the build asks too).

The addresses a site answers at are worked out per language on the same terms the build uses: every listing exists in every language site.locales names, a POST exists in a language only if it has words there, and /assets/ and /write/ are shared and have no copy per language.

A question that raises is one finding, not the end of the run: every question is asked through a guard that turns an exception into an error naming the question and the exception, and the rest are still asked. One post whose media was an object once took the whole run down before it had reported anything -- and under --json left not even a document to parse. The finding is an error, so the exit code says so; it also names the exception, because this is also how a bug in the checker itself would show.

Two rules shape the plain run. It only ever reports -- nothing here deletes an orphaned directory or rewrites a post, because the whole value of the tool is that its output can be trusted, and a checker that also acts has to be trusted twice. And each kind of finding is capped, the remainder counted (more), since a thousand identical lines is not more information than a screenful and buries every other kind -- the cap, with the rest of the command's day-to-day surface, is in The archive. The exit code is non-zero on errors only: an orphan directory costs disk rather than correctness, so a cron job hanging off this would be crying wolf if warnings failed it. Like export and stats it has its own entry point (scripts/check.rb) rather than going through manage_post.rb, which applies the site timezone as it loads and aborts on a config it cannot read -- a run that exits explaining the config has checked nothing.

Both rules have a door in them since 1.4, and both doors are opened by hand. --repair walks the findings and offers, per finding, the single repair that finding allows: a redirect written into the target post, a relative link rewritten, a file moved to the trash. Nothing is applied without a key press, a version is kept before a post is changed, and nothing is ever deleted. --json prints every finding instead of a capped screenful, because a cap is right for reading and useless for acting on -- a script cannot work from "...and 23 more".

--online is the only part that leaves the machine, and it is asked for by name because it takes minutes rather than a second and, over an archive going back twenty years, will find things nobody can do anything about. What counts as a finding is narrow on purpose: a host that no longer resolves, and a 404 or 410, are the web saying "this is gone"; a timeout, a refusal, a 5xx, a TLS error or a 403 are the web saying "not right now", and reporting those turns one flaky evening into forty findings that are all fine again tomorrow. A checker nobody believes is worse than no checker.

Three details make it usable on a real archive rather than only correct:

• HEAD first, GET to confirm anything fatal. A link checker has no use for the body, and HEAD over a few thousand links is the difference between minutes and an afternoon -- but HEAD is never believed when it says a page is gone. Some servers answer it with 405 or 501 while serving GET perfectly; worse, some answer 404 to HEAD and 200 to GET for the very same address (bsky.app does this on profile pages, and the first run over a real archive reported dozens of live links as dead because of it). So RETRY_WITH_GET re-asks with a GET, one extra request per apparently-dead link.
• Politeness rather than throughput. One request at a time, and a one-second pause between two requests to the same host: an archive with two hundred links to one site should not read as an attack on it.
• Memory between runs. Checker::Cache keeps every verdict in tmp/link-check.json for a while (how long, see The archive), so a second run only asks about the links it has not seen lately -- without it nobody runs this twice, since a few thousand requests is minutes and most of the answers were the same yesterday. A cache it cannot read or write is not an error: the check simply asks the network, which is what it was going to do anyway.