Importing

Architecture — part 4 of 13

An import splits along the line between what every platform needs and what only one does, so that adding a source means writing its mapping, not another copy of the plumbing every existing adapter shares:

  • Import::Media collects one post's media for PostWriter, from either a URL (download, following redirects, retrying transient failures) or a path an export already contains. Filenames are numbered per post in registration order, which is what makes a re-import land on the same names so PostWriter's "skip if it exists" copy is a no-op instead of a duplicate. In dry-run it allocates and counts without fetching -- so adapters must take image dimensions from platform metadata, never from the downloaded bytes.
  • Import::Run drives: walk the source, hand each item to the adapter, write it, count why items were skipped, report. Skips are Symbols, so the summary can answer the question anyone reads a summary for -- why fewer posts arrived than the source has. Two callbacks, on_scan and on_post, let a wizard show progress without this layer knowing what a terminal is.
  • An adapter implements only label, each_item (paging the source) and map(item, media), returning a post hash or a skip reason. Five optional hooks cover what sources differ on: preamble (a line to print before a slow read), total (the source's size, often knowable only once the first page arrives), postscript (a one-line note after a run or preview -- scheduled posts becoming drafts, posts with no usable original address), platform_tag (the tag posts are filed under when the platform's name makes a poor one -- "medium.com" says more than "feed") and keep_permalinks= (a setter whose presence tells the wizard this adapter can record old addresses as redirect_from, so the question is asked per capability and defaults to no).
  • Import::HtmlBlocks converts a post body that arrives as markup into blocks, for the sources that hand over HTML rather than structured data. A tolerant tokenizer and a stack-based tree (an XML parser refuses real post HTML outright), then a conservative mapping: exactly what the schema can represent, unknown wrappers walked through, and anything with no representable shape dropped and counted rather than guessed at.
  • Import::Cli is the non-interactive front end the scripts/migrate_*.rb wrappers share, so each is a handful of lines. Every source is therefore reachable both ways -- wizard or script -- over one implementation of the mapping.

Import::Feed covers WordPress and RSS/Atom in one adapter, because they are one format: a WXR export is RSS 2.0, with a wp: namespace layered on for what a feed has no room for (post_type to filter by -- in a stock export menu items, attachments and pages outnumber the posts -- status for drafts, post_id for dedup, post_name for the slug the site already published under). It takes the date from pubDate rather than wp:post_date, which carries no offset and would otherwise be read in site.timezone and shift by hours. Images referenced in the markup are downloaded and then measured: HTML rarely states dimensions, and a block without them renders with nothing reserving its place, so the page jumps as each picture lands.

An adapter judges items rather than pre-filtering them: map returns the reason as a Symbol -- :reply, :retweet, :quote, :empty and their kin; the wizard's TRANSLATED_REASONS in scripts/import.rb names two dozen across the adapters -- instead of quietly dropping the item, so the run's summary can account for every item the source held. That also means a written-post counter is not a progress fraction -- a source that skips half its items never writes as many posts as it has -- so both counters are reported and the caller picks: against a LIMIT the goal is posts written, against a whole source it's position scanned.

Import::Bluesky is the reference adapter. Two things it has to get right generalize to any API source: facet offsets are UTF-8 byte positions while the schema's formatting spans are codepoints, so every boundary is converted (skip that and spans land several characters off on any non-ASCII text), and video arrives as an HLS playlist rather than a file, so the poster frame is imported as an image -- a video block with no media would render as "video unavailable".

scripts/import.rb is a second wizard rather than a menu entry in blog.sh, and always previews in dry-run before writing. lib/launcher.rb is the one piece that is for a single platform: on a Mac setup.sh can put an icon in Applications that opens Terminal in the site's folder and runs ./blog.sh, built with tools macOS ships. lib/site_header.rb is shared by both wizards so the site-identity block can't drift between them.

Supporting casts: lib/slug.rb (one folding/slugification used for post slugs, tag slugs, heading anchors and the search index -- and mirrored client-side by fold() in search.js), lib/content_type.rb (dominant type: video > audio > image > chat > quote > link > text, where "quote" means the post's first block is one, and media win only while the post's text stays caption-sized -- past 500 characters it's an article with illustrations, i.e. text), lib/media_dimensions.rb (width/height straight from PNG/JPEG/MP4 headers -- EXIF orientation included, so a photo taken sideways reserves the space it is shown at), lib/video_probe.rb (the video track's codec from the same box walk, two levels further down at stsd, so a save can say that a clip is HEVC).