Validate Readable Source IDs in Content Pipelines

The queue boundary is part of your data model

Automated publishing systems often fail Audit Macos System Data Before Deleting they reach the interesting parts of the workflow. A source card can contain a useful topic, a clear editorial angle, and enough evidence for a good article, yet never become a candidate because one boundary parser rejects its identifier. The failure is especially confusing when the identifier looks obviously valid to a person.

This happens when a system begins with an opaque ID rule such as SRC-[A-Z0-9]+, then gradually adopts readable IDs such as SRC-WORKER-SOURCEID-2026. The card is still safe, but the old expression accepts only one segment after SRC-. Candidate discovery reports no work, revision discovery cannot find the original source, and the same card may be retried or quarantined without a useful explanation.

The fix is not to make the boundary permissive. The fix is to define the identifier grammar once, validate it strictly, and use that contract everywhere the value is read. Readable IDs are metadata. They should explain what a card is about without becoming a substitute for authorization, content quality, or duplicate detection.

Define a narrow grammar before writing a regular expression

A good source ID contract answers four small questions. What prefix identifies a source card? Which characters are allowed in each segment? Can a segment be empty? How are segments separated? For a RayLabs source card, the contract is:

• the identifier starts with the uppercase prefix SRC-;
• each following segment contains one or more uppercase letters or digits;
• segments are separated by exactly one hyphen;
• there is no trailing hyphen and no whitespace.

That contract accepts SRC-ANDROID-CAMERA-2026, SRC-WORKER-SOURCEID-2026, and SRC-9A2. It rejects src-ANDROID-2026, SRC--ANDROID, SRC-ANDROID-, SRC-ANDROID 2026, and SRC-ANDROID/2026.

The corresponding TypeScript check is intentionally boring:

const SOURCE_ID = /^SRC-[A-Z0-9]+(?:-[A-Z0-9]+)*$/;

export function isSourceId(value: unknown): value is string {
  return typeof value === 'string' && SOURCE_ID.test(value);
}

The expression is not a security boundary. It does not prove that a card came from a trusted author, that its topic is original, or that its links are safe. It only proves that the value has the shape required by the queue. Those other decisions belong to separate validators.

Keep candidate and revision readers on the same contract

Many content workflows have more than one reader. A candidate reader scans source cards that have not been consumed. A revision reader starts with a draft, extracts source_refs, and loads the original card. A publication reader may inspect the same ID while writing an audit marker. A fix in only one reader creates an asymmetric system: new articles work while revisions fail, or revisions work while the next candidate disappears.

Every reader should call the same source-ID validator. The revision extractor must also capture the complete value. A pattern that stops at the Schema First Gates Ai Publishing Pipelines hyphen can turn SRC-WORKER-SOURCEID-2026 into SRC-WORKER, which then fails a lookup even though the source card is present. The safest approach is to parse the frontmatter list first, then validate the complete scalar. If a line-oriented fallback is necessary, capture until the end of the value rather than guessing how many segments an ID contains.

The source card itself remains the authority for its ID. The draft may reference it, but the workflow should compare the draft reference with the source card frontmatter before building a revision. This catches copied or stale metadata without scanning every source card. One direct lookup is cheaper and clearer than a broad search followed by a second lookup.

Validate at trust boundaries, not in the middle of prose work

The source card enters the system through GitHub content, which is untrusted input. Validate its frontmatter before sending any content to a model. If the ID is malformed, report the exact path and reason, then quarantine the card without deleting it. Do not ask a model to repair an identifier. A model can rewrite prose, but it should not decide whether a repository object is addressable.

The same applies to model output. The builder must return the exact source reference that the workflow supplied. If the response changes SRC-WORKER-SOURCEID-2026 to a shorter or invented value, reject the response before it can create a draft. This check prevents a valid source from becoming detached from its article and makes retries idempotent.

Error messages should distinguish shape failures from lookup failures. “Invalid source_ref” means the value violates the grammar. “Source card not found” means the value is well formed but the referenced path is absent. “Source changed before commit” means the repository moved during processing. These messages point to different owners and different recovery actions.

A readable ID does not replace a duplicate check

Readable IDs make logs easier to follow, but they are not globally unique by magic. Two cards can still carry the same ID, and a copied card can carry an old ID accidentally. The workflow should enforce uniqueness when it builds the tree snapshot. If the same source reference appears in two active paths, quarantine the conflict and preserve both files for inspection. Never silently choose one based on alphabetical order.

Article identity is separate again. A source card can be revised several times while keeping its source reference, but each logical article needs a stable article_id. The publication boundary checks the article ID and slug against the site marker before writing. This prevents a retry from publishing the same article under a new filename simply because the draft revision number changed.

The three identifiers answer different questions: source_ref answers “which input produced this work?”, article_id answers “which logical article is this?”, and slug answers “which public URL represents it?”. Keeping them separate makes deduplication reliable.

Use one bounded lookup per queue item

Resource limits make discovery strategy part of correctness. A free Worker should not scan every card, read every blob, ask several model providers, and then retry all of them in one invocation. Select the first deterministic item from the sorted tree, read that card once, and stop if it is blocked. The next scheduled run can continue with the next item after quarantine or a successful commit.

For a revision, read one draft, extract one validated source reference, and fetch that source directly. For a candidate, read one source card and select one reviewed raster from the catalog. For publication, read one eligible draft and its matching quality packet. This gives each phase a predictable request budget and makes a failed run explainable.

The queue does not need a rotating cursor for this. Alphabetical order is acceptable when blocked items are moved out of the active folder and successful sources are consumed. Determinism matters more than pretending a timestamp provides fairness. If throughput later becomes important, a small persisted cursor can be added after measurements show that sorting is insufficient.

Quarantine malformed cards without losing editorial work

Quarantine is a safety action, not a deletion. Move a malformed or duplicate source card to an archive path that records the reason, and leave its contents intact. A human or a later repair job can then fix the card and return it to the active source folder. The original commit remains available for audit.

The quarantine commit should be atomic. It moves the card and removes the active path in one fast-forward update. If main changes before the commit, stop and let the next run re-read the tree. Do not force the update and do not mark the card consumed when the move did not land.

The same pattern applies to a builder failure. A short article, invalid keyword list, or unsupported URL is not a source-ID problem. Keep the source card available for a bounded retry, or quarantine it only when the failure is structural and cannot be repaired by another attempt. Reporting the exact blocker is more useful than repeatedly logging “no candidate”.

Test the grammar and the whole path

A unit test for the regular expression is necessary but not sufficient. Test accepted examples, malformed examples, and a complete candidate path using a hyphenated ID. Add a revision test that extracts the same full ID from a draft and loads the matching source card. Finally, test that a malformed card is quarantined and that another source can be selected on the next run.

The most valuable regression test is the one that failed in production: a readable ID with multiple hyphen-separated segments passed source-card creation but was rejected at the Worker boundary. Keep that case in the suite. It prevents a future refactor from replacing the readable grammar with the old one-segment shortcut.

Tests should also verify that the validator does not execute content. Source cards and model responses are data. A string that resembles JavaScript, shell syntax, or a prompt must remain a string. Parsing frontmatter and checking a regular expression is enough; there is no reason to evaluate the value.

Observability should name the boundary and the owner

When the queue reports a blocked item, include the phase, path, source reference, reason, and owner. A malformed identifier is a system-owned contract failure if the card was generated by the Worker. A hand-edited card with a typo is an editorial-data issue. The distinction lets the next action be obvious without opening a dashboard full of raw JSON.

A useful event might say: “candidate blocked, Medium Posts/00 Sources/SRC-WORKER-SOURCEID-2026.md, invalid source ID, system, no commit”. A lookup failure might say: “revision deferred, source SRC-ANDROID-CAMERA-2026 not found, Ray, source card must be restored”. Both are better than silently skipping the card or retrying indefinitely.

The same event should include the current commit SHA. If the repository changes during processing, the SHA guard explains why no write occurred. This is especially important when a vault synchronization job and a Worker run overlap. The safe outcome is a deferred item, not a partial article.

A strict contract makes later changes cheaper

Readable identifiers are small, but they sit at the intersection of discovery, revision, deduplication, and audit. Once the grammar is explicit and shared, a future ID format can be introduced deliberately. For example, a versioned prefix could be accepted by a new validator while old IDs remain readable during migration. That change would be tested and documented instead of appearing as a mysterious empty queue.

The practical rule is simple: define the data shape once, validate it at every trust boundary, and preserve the input when something fails. A publishing system becomes more reliable when it refuses malformed work quickly, explains why, and continues with the next safe item. It does not become more reliable by loosening the expression until every string passes.

When a source card has a readable ID, the rest of the pipeline can be boring. Candidate discovery can select it, revision processing can find it again, quality gates can evaluate the article, and an idempotent commit can publish the result. That is the outcome worth designing for: fewer invisible skips, bounded work per run, and an audit trail that a human can understand.

Conclusion: make the queue predictable

Source IDs are not decoration. They are the address used by an automated content queue, so their grammar deserves the same care as a public API field. Accept meaningful hyphen-separated segments, reject empty or ambiguous values, and reuse the rule in candidate, revision, and publication paths.

The implementation can stay small: one shared regular expression, one complete-value extractor, direct source lookup, and an atomic quarantine path. Pair those pieces with tests for the real failure case and logs that name the owner. The result is a queue that fails safely instead of losing valid cards at the first boundary.

This discipline also improves maintenance. A new contributor can inspect one contract and understand which strings are accepted, while an operator can recognize a malformed card from the first log line. Clear data shapes reduce the need for special-case repairs, and the queue remains predictable when the vault grows.

The source contract used here is documented in the RayLabs source-ID fix. The broader lesson applies to any Git-backed content or deployment pipeline: readable data is useful only when every component agrees on what it means.