CI/CD Redaction Validation

Treating redaction rules as code means every pattern change, threshold tweak, or entity addition must pass automated tests before it can ship. This guide, part of the PDF and DOCX Parsing & Extraction Workflows field guide, shows how golden-corpus regression suites and deterministic diff assertions gate each commit so a rule can never silently under-redact.

Why redaction rules belong under change control #Permalink to this section

A redaction ruleset is executable policy: a single loosened regex or a lowered confidence cutoff can expose Social Security numbers across an entire discovery production, and the failure is invisible in a code review diff. The defence is the same one used for any safety-critical code — a versioned test corpus that exercises the rules end-to-end, deterministic assertions that fail loudly on any behavioural drift, and a pipeline that refuses to merge red builds. Rules, fixtures, and expected outputs live together in the repository and evolve through pull requests, giving every change a reviewer, a timestamp, and a passing build before it reaches production alongside the Async Batch Processing Pipelines that run at scale.

Test tiers, thresholds, and gate policy #Permalink to this section

The suite is layered so fast checks fail early and expensive checks run only when they must. Each tier has an explicit pass threshold and an enforcement point.

Thresholds are strict by design: an “unexpected diff” count above zero means the redacted output changed in a way no one approved, which is treated as a defect rather than a warning.

A golden-file redaction assertion #Permalink to this section

The core pattern is a parametrized test that redacts each fixture and compares the result against a checked-in, human-approved baseline. The companion page Batch Redaction Regression Tests with pytest expands this into a full harness.

# test_golden_redaction.py
import hashlib
from pathlib import Path

import pytest

from redactor import redact_document  # returns normalized redacted bytes

CORPUS = Path(__file__).parent / "corpus"
FIXTURES = sorted(CORPUS.glob("input/*.pdf"))


def _digest(data: bytes) -> str:
    # Hash the normalized output; never persist or log raw document bytes.
    return hashlib.sha256(data).hexdigest()


@pytest.mark.parametrize("fixture", FIXTURES, ids=lambda p: p.stem)
def test_redaction_matches_golden(fixture: Path) -> None:
    baseline = CORPUS / "golden" / f"{fixture.stem}.sha256"
    assert baseline.exists(), f"No approved baseline for {fixture.stem}"

    produced = redact_document(fixture.read_bytes())  # deterministic output
    expected = baseline.read_text().strip()

    # Any drift in redacted output fails the build — no silent under-redaction.
    assert _digest(produced) == expected, (
        f"Redaction output for {fixture.stem} diverged from approved baseline"
    )

Because the assertion compares a hash of the normalized output, it flags any change — a moved rectangle, an un-redacted span, a new artifact — without ever storing the document content itself in the test log.

Compliance alignment #Permalink to this section

Gating rule changes behind automated tests is a direct implementation of configuration change control. NIST SP 800-53 Rev. 5 control CM-3 (Configuration Change Management) requires that changes be tested, validated, and documented before deployment; a required CI check that blocks merge is exactly that control expressed in a pipeline. Control SA-11 (Developer Testing and Evaluation) requires developer-run security testing at each iteration, satisfied here by the pre-commit and per-PR regression suite. The versioned corpus, approved baselines, and immutable build logs together establish that redactions were produced by a validated, repeatable process — the defensibility standard opposing counsel will probe under FRCP Rule 34, which governs the form and integrity of produced documents.

Performance and scale #Permalink to this section

Keep the gating suite fast enough that developers never route around it. A 1,200-document golden corpus of mixed contracts and filings redacts and hashes in roughly 9 minutes single-threaded on 4 vCPUs. Running pytest with pytest-xdist (-n auto) fans the parametrized cases across cores and drops wall-clock to about 2 minutes 40 seconds on the same machine — comfortably inside a PR feedback loop. The heavy per-document cost is PDF rendering, so engine choice matters; see pdfplumber vs PyMuPDF Performance before scaling the corpus past a few thousand fixtures, and split the recall spot-check into a nightly job so the PR gate stays lean.

Edge cases and failure handling #Permalink to this section

• Nondeterministic output causing flaky tests. PDF writers stamp CreationDate, ModDate, and a random /ID on every run, so a naive byte comparison fails intermittently. Normalize the output — strip volatile metadata and pin the document ID — before hashing, as detailed in Deterministic Diff Assertions for Redacted PDFs. Never quarantine a flaky redaction test; a nondeterministic result masks real regressions.
• A corpus containing real PII. Golden fixtures live in the repository and its CI logs forever. Populate them only with synthetic identifiers generated for the purpose. A single real Social Security number in corpus/input/ is a reportable data exposure, so a coverage-audit test should scan fixtures for anything matching a live-PII heuristic and fail if found.
• Coverage gaps on new entity types. Adding a matcher for, say, medical record numbers without adding a fixture that exercises it leaves the new rule untested and the old baselines unchanged — a green build that proves nothing. The coverage-audit tier fails when an entity type has zero fixtures, forcing a fixture and an approved baseline to land in the same pull request as the rule.

Related #Permalink to this section

The mechanics live in three companion pages: assemble the harness in Batch Redaction Regression Tests with pytest, make outputs comparable in Deterministic Diff Assertions for Redacted PDFs, and wire the gate in GitHub Actions Workflow for Redaction Validation. Pair this with Async Batch Processing Pipelines for production execution. Part of the PDF and DOCX Parsing & Extraction Workflows field guide.