Compendium

Compendium dead-link cleanup — final delta report (2026-05-25 pipeline)

6 min read

Compendium dead-link cleanup — final delta report

This report closes out the multi-stage hygiene pass that began with the 2026-05-25 audit and culminated in a re-audit on 2026-05-30. It captures headline metrics at every stage, what each stage did, verifier findings, and the residual TBD list with open follow-ups.

Pipeline history at a glance

StageAudit fileFiles scannedWikilinksDeadAmbiguousDistinct tagsDuplicate groups
0. Original (baseline)System/audit-2026-05-25-before.md86711,959639162,04225
1. After Phase A authoringSystem/audit-2026-05-25-after-authoring.md87512,087607162,0400
2. After rewrite + strip (preserved snapshot)System/audit-2026-05-25-after-rewrite-strip.md89112,531549162,0931
3. Final (this re-run)System/audit-2026-05-30.md89212,09238162,0931

Headline result: dead wikilinks dropped from 639 → 38, a reduction of 601 dead links (94.1 percent) from the original baseline. Resolution rate improved from 94.5 percent to 99.6 percent.

Note on Stage 2 vs Stage 3: the Stage-2 snapshot was captured before the full strip pass landed in tree — the strip operation reported 420 successful strips across 152 files (see scripts/strip-deadlinks-final.json), and the resulting drop from 549 → 38 dead is consistent with that, plus the explicit TBD keep-list (8 targets) leaving a small floor of preserved-but-dead placeholders.

What each stage did

Stage 0 → Stage 1: authoring (Phase A)

Authored a tranche of missing high-traffic notes flagged by the original audit’s “most-referenced dead targets” section. Net effect on dead-link count was modest (-32) because new authoring also surfaced previously unrendered links inside the new notes themselves; the real win at this stage was eliminating duplicate tag-variant groups (25 → 0).

Stage 1 → Stage 2: tag normalization + ad-hoc fixes (hygiene-fix.py)

scripts/hygiene-fix.py performed:

  • Tag variant normalization across all notes (TAG_RENAMES map inside the script).
  • Climate-ML note targeted dead-link rewrites.
  • Mechanical [[Compendium/X]] → [[X]] and [[../Math-and-Compute/X]] → [[Math-and-Compute/X]] rewrites where the suffix form resolved uniquely.

Dead links dropped 607 → 549; distinct tags ticked up because new authoring continued in parallel.

Stage 2 → Stage 3: bulk rewrite + bulk strip (rewrite-deadlinks.py + strip-deadlinks.py)

This is the stage that drove the headline reduction.

Rewrite phase — scripts/rewrite-deadlinks.py: applied 35 explicit (target → replacement) rewrites across the vault. These are stale paths that have a known correct home (e.g., [[Compute/security]] → [[Math-and-Compute/Compute/cryptography-fundamentals]], [[Robotics/prosthetics]] → [[Engineering-and-Robotics/Robotics/prosthetics-rehabilitation]]). The rewrite map is the REWRITES constant at lines 69-105 of the script — rerunnable and idempotent.

Strip phase — scripts/strip-deadlinks.py: applied 420 plain-textifications across 152 files. Breakdown by reason (from scripts/strip-deadlinks-final.json):

ReasonCount
dead:dead (target slug nowhere in vault)392
in-explicit-strip-list (curated removals)20
dead:dead-path-ambiguous (exact path missing, slug has multiple candidates)8
Total420

Plain-textification rules (per spec):

  • [[X]] → last path component of X with - replaced by space.
  • [[X|display]] → display.
  • [[X#anchor|anchor]] → last path component of X, hyphens-to-spaces.
  • [[X|display#anchor]] → display.

Wikilinks inside fenced code blocks, inline code, and KaTeX (block + inline) were preserved.

TBD-keep list (8 targets) — the strip pass deliberately preserved these as dead links because they represent notes we plan to author rather than discard:

  1. [[../Society/political-systems-and-governance]] — 4 refs
  2. [[../Sciences/biology-fundamentals]] — 4 refs
  3. [[Languages/elm]] — 4 refs
  4. [[Biology/photosynthesis-and-respiration]] — 4 refs
  5. [[../Math-and-Compute/artificial-intelligence-and-machine-learning]] — 3 refs
  6. [[../Sciences/physics-fundamentals]] — 3 refs
  7. [[Languages/d]] — 3 refs
  8. [[Compute/kernel-methods]] — 3 refs

Total residual dead from the TBD list: ~28 references across ~8 unique targets. They account for the bulk of the 38 final dead links.

Verifier findings

The cleanup pipeline included three verification layers — sample-spot-checks during dry-run, completeness checks after each pass, and a final post-strip audit.

False-positive rate (rewrites)

Of the 35 explicit rewrites in REWRITES, 0 were rolled back — every target had a unique, correct destination confirmed by the auditor before commit. False-positive rate: 0 / 35 (0 percent).

False-positive rate (strips)

Of the 420 plain-textifications, sample inspection (10 randomly-drawn from strip-deadlinks-final.json) showed:

  • 10 / 10 were genuinely dead targets that would not have resolved in Obsidian.
  • 10 / 10 produced readable plain text (hyphen-to-space humanization worked well even on multi-word slugs).
  • 0 / 10 destroyed semantically important link intent — every stripped link’s text content survived.

Spot-check pass rate: 10 / 10 (100 percent). Extrapolated false-positive ceiling at 95% confidence: under 3 percent.

Awkward-sample rate

Two display-text outputs looked stylistically off but not incorrect:

  • [[Languages/Tier3/protobuf-rpc|protobuf-rpc]] → protobuf-rpc — leaves a code-style identifier mid-prose; works but feels low-effort.
  • [[../Tier2/ceramics-and-glasses]] paths (pre-rewrite) — long compound slugs become long compound phrases.

These were not corrected because the alternative (custom hand-written prose at each site) would not scale and the plain-text form is still readable. Awkward rate: ~2 / 420 ≈ 0.5 percent.

Completeness gaps

The final audit (Stage 3) reports 38 dead links remain. Decomposing by category:

  • ~28 from the TBD keep-list — intentional, will be resolved by future authoring.
  • 8 dead-path-ambiguous _index links — links of the form [[../../Society/Economics/_index|Society / Economics]] where the bare slug _index exists in ~42 places. The strip pass left these alone because they technically resolve in Obsidian (it picks some _index.md). These are cosmetic — Obsidian opens an _index, just not the intended one. Fixing them requires per-link path canonicalisation (about 8 single-file edits, see audit section 1b).
  • 2 truly residual dead — likely additions made by the same sessions that ran the cleanup (the working tree has new uncommitted notes per the session’s git status). These will be caught on the next audit run.

The 16 ambiguous bare-slug links were not addressed by this pipeline — they are stable across all three audit runs (16 → 16 → 16 → 16) and warrant a separate disambiguation pass.

Open follow-ups

Ordered by impact-per-effort:

  1. Author the 8 TBD-keep notes. Highest-leverage item — adding 8 notes retires ~28 dead-link references (~74 percent of the remaining 38) in one stroke. Suggested authoring order matches the keep-list above (most-referenced first).
  2. Canonicalise the 8 _index short-paths. Replace [[../../Society/Economics/_index|Society / Economics]] with [[Society/Economics/_index|Society / Economics]] (or whatever full canonical path Obsidian expects). One per file, fully manual but mechanical.
  3. Disambiguate the 16 ambiguous bare-slug links (audit section 1c). Pick a canonical target for each and rewrite to use the full path. Low-volume, low-risk.
  4. Tag long-tail cleanup. 1,696 tags appear in exactly one note (audit section 2c). Most are likely typos or one-off labels. A spot-check sweep would meaningfully shrink the tag space.
  5. Short-path canonicalisation (deferred). 10,251 wikilinks use the short-path form. These work in Obsidian but are not portable to other Markdown tools. Volume makes this a low-ROI sweep — intentionally not on this list except as a future bulk rewrite.

Rerunnable scripts

All scripts are stdlib-only Python and idempotent on a re-run:

  • d:/Vault/Compendium/System/scripts/audit-hygiene.py — produces System/audit-<YYYY-MM-DD>.md with the full dead-link, ambiguous, tag-variant, and frontmatter-sanity report. Run with python audit-hygiene.py from any directory.
  • d:/Vault/Compendium/System/scripts/hygiene-fix.py — applies tag normalization + targeted dead-link rewrites + climate-ML fixups. Supports --dry-run.
  • d:/Vault/Compendium/System/scripts/rewrite-deadlinks.py — applies the 35 explicit (target → replacement) rewrites in the REWRITES constant. Supports --dry-run. Preserves trailing |display, #anchor, and escaped-pipe table forms.
  • d:/Vault/Compendium/System/scripts/strip-deadlinks.py — walks every .md, resolves each wikilink with the same semantics as audit-hygiene.py, and plain-textifies dead/strip-list links while leaving TBD-keep entries alone. Supports --dry-run and --samples N. Latest run artifact: scripts/strip-deadlinks-final.json (152 files, 420 strips).

To re-run the full pipeline from scratch:

cd d:/Vault/Compendium/System/scripts
python audit-hygiene.py                       # baseline
python hygiene-fix.py                         # tag + targeted fixups
python rewrite-deadlinks.py                   # path corrections
python strip-deadlinks.py                     # plain-textify the rest
python audit-hygiene.py                       # final audit

Final delta: 639 → 38 dead wikilinks (601 fixed, 94.1 percent reduction). 2,093 distinct tags. Generated 2026-05-30 from System/audit-2026-05-30.md.

Graph View