What every import does

The long version of The archive → Importing: per-source walkthroughs, what each import keeps and skips, and how to check -- or undo -- the result. Everything here was verified against real archives, not just read from API docs.

Two ways in, same mapping underneath:

./import.sh                              # wizard: pick a source, preview, confirm
ruby scripts/migrate_<source>.rb <arg>   # scriptable: writes immediately, for cron

The wizard always reads the whole source in dry-run first and reports what would be written -- posts, media files, the first slugs, and every skipped item with its reason -- then asks you to type the post count to confirm. The scripts skip the preview; sample with LIMIT=20 instead, which stops after twenty written posts. Re-running either way is safe: posts are matched on source.platform/account/original_id and overwritten in place, never duplicated.

• Media comes home. Images are downloaded (or copied from the export), stored in media.nosync/<year>/<slug>/, and measured, so every page can reserve layout space for them (one without known dimensions still renders, the page just jumps once while it loads). Downloads follow redirects and retry transient failures; a file that still can't be fetched costs that image, not the post, and the summary says so.
• A file already here is not downloaded twice. Each media entry records the address it came from, so a re-import recognises what the archive already holds and fetches only what is missing -- the summary says how many it did not have to fetch, and over a complete archive that is no requests at all. A picture that leaves a post and later returns finds its old file again -- the post's kept versions remember the address, and failing those the file's own bytes do -- so a drop-and-return cycle does not grow the post's directory.
• An import only ever ADDS media. A file already in media.nosync/<year>/<slug>/ is never replaced -- not by a re-import, not by any flag -- because the bytes an import brings come from somewhere it cannot vouch for. REFETCH_MEDIA=1 only makes the run ask the source about every address again instead of trusting the archive's records; what lands on disk is still just the files that were missing. When a download no longer matches the copy the archive keeps -- a source that re-encoded a picture under its old address -- the archive's copy wins and the summary counts the discarded download, which is the one signal that the source has drifted away from this archive. And nothing an import does repairs a damaged file: ./blog.sh check reports media a post names that the archive doesn't have, and putting it right is a separate job with your own copy of the original.
• The post describes the file that is really there. Width and height are re-read from the archive's own copy once the media is in place, so a post cannot end up claiming dimensions over a file that never arrived. A copy that cannot be measured -- zero bytes, truncated, a format this engine cannot read -- leaves the post's previous record standing; on a first import, where there is no previous record, a video or an SVG keeps the dimensions its source stated. An entry whose file is missing entirely is left saying exactly what it said.
• A re-import updates, never duplicates. Posts are matched on their source identity -- platform, account and the item's own id -- not on the title, so fixing a typo at the source and importing again updates the existing post in place, keeping its slug and URL. The one exception is an item with no usable identity (a plain RSS item with neither guid nor link): rather than guess and risk merging two different posts, a re-import writes it again -- a duplicate you can delete, where a wrong match would destroy a post you can't get back.

The other exception is about WHERE the export sits. Several sources have no account name anywhere in their files (Facebook, Threads, Wix, Movable Type, Substack), so the identity borrows the export's own directory or file name. Unpack the next export into a folder named after its date and every post in it is a new post: the archive imports a second time, slugs and all. Import from the same path each time -- or rename the new export to the old name -- and matching works as described.

Ghost is a third shape, and this list left it out until 1.5.1: the address you type as the second argument is presented as "where the images live", and the host out of it is also half of every post's identity. Case and a leading www. are folded away, so https://www.cynicky.blog and https://cynicky.blog are the same site on a re-import -- but a genuinely different host is a different site, and typing one imports the archive a second time.

A markdown tree is the one source that can do better, because a site built by Hugo or Jekyll writes its own name down. The identity is that name -- baseURL or title out of hugo.toml (or config/_default/), url or title out of _config.yml -- so such a tree can be moved, renamed, restored somewhere else or unpacked on a second machine and a re-import still finds its own posts. A tree that declares nothing falls back to the full path it was imported from, and that is the case to know about: a bare content/ directory, a folder of _posts/ with no _config.yml beside them, or anything a converter produced matches only while it stays where it was. Point the import at the site root rather than at the content directory where you can -- that is where the configuration is. A skeleton nobody has edited yet (hugo new site leaves baseURL at example.org, jekyll new leaves the title at "Your awesome title") counts as declaring nothing, on purpose: two untouched skeletons would otherwise claim the same identity and the second import would land on top of the first. - Origin becomes a tag. Every imported post is tagged with its platform (tumblr, wordpress, ...; for a plain feed, the site's domain), so /tag/tumblr/ is the whole of one old blog. Deduplicated case-insensitively against the post's own tags. - One bad item costs one item. A date that won't parse or markup nothing anticipated is counted under error and named on stderr; the run continues. A rejected API key still stops everything, as it should. - What HTML conversion cannot keep is counted: markup with no block equivalent (an iframe, an embedded player, a form) is dropped and named in the summary -- by every adapter that converts HTML, not just the feed import. - A dying source still leaves a summary. If the platform stops answering mid-run (a 5xx on page twelve, a feed that goes away), the run stops, says so, and reports honest partial counts -- everything written up to that point is saved. The scripted path exits non-zero so cron notices. Re-running once the source recovers picks up safely: the posts already written are matched and updated, not duplicated. - Progress is narrated -- what's being read and how big it is, then a running counter. A silent minute means something is wrong, not that it's working. - Old addresses can survive the move. When the new site answers on the same domain the old blog did, a source that knows its original URLs -- most do -- can record each published post's old path as redirect_from: the build then serves a redirect at every one of them, so nothing anyone ever linked goes dark. The wizard asks wherever the source can answer; the scripts take KEEP_PERMALINKS=1, and a source that needs a URL pattern's help says so in its section below. Say yes only on the same domain: on any other, the old paths were never yours to answer. Posts with no usable path (WordPress "plain" ?p=123 permalinks live in the query string, which a static file can never see) are counted in the summary and imported without a redirect. For an archive imported before this existed, scripts/backfill_redirects.rb <old-domain> adds the same entries from what the import already stored -- preview by default, WRITE=1 to apply. - Nothing deploys by itself. The wizard offers a rebuild at the end; the scripts leave both to you.

Before the first real run

Back up your content -- it isn't in git, and on a server there is nothing else to fall back on:

tar czf ../content-backup-$(date +%F).tar.gz content.nosync media.nosync

And expect the deploy guard afterwards: a bulk import is exactly the "file count swung wildly" shape it watches for. Check the numbers, then re-run ./scripts/deploy-web.sh with --force.