Batch Redaction Regression Tests with pytest

A batch redaction regression suite redacts every fixture in a golden corpus and asserts each output still matches an approved baseline, so a rule change that quietly stops masking a name or number fails the build instead of shipping. This how-to builds that parametrized harness under CI/CD Redaction Validation.

Prerequisites and version matrix #Permalink to this section

Pin exact versions so the suite is reproducible on every developer machine and CI runner.

Install with pip install "pytest==8.2.*" "pytest-xdist==3.6.*" "PyMuPDF==1.24.*". Keep these in a requirements-test.txt that CI installs verbatim.

Step 1 — Lay out the corpus #Permalink to this section

Separate inputs from approved outputs so a baseline update is an obvious, reviewable diff:

corpus/
  input/     # synthetic source documents (*.pdf)
  golden/    # approved baseline digests (*.sha256)

Every input/<name>.pdf has exactly one golden/<name>.sha256. A missing baseline is a hard error, never a skipped test — an untested fixture proves nothing about the rules.

Step 2 — Discover fixtures and parametrize #Permalink to this section

pytest’s parametrize turns each fixture into its own reported test case, so a failure names the exact document that regressed:

# conftest.py
from pathlib import Path

CORPUS = Path(__file__).parent / "corpus"


def fixture_paths():
    # Sorted for stable, deterministic test ordering across machines.
    return sorted((CORPUS / "input").glob("*.pdf"))

# test_batch_regression.py
import hashlib

import pytest

from conftest import CORPUS, fixture_paths
from redactor import redact_document  # returns normalized redacted bytes


@pytest.mark.parametrize("src", fixture_paths(), ids=lambda p: p.stem)
def test_batch_redaction(src):
    baseline = CORPUS / "golden" / f"{src.stem}.sha256"
    assert baseline.exists(), f"Missing approved baseline for {src.stem}"

    produced = redact_document(src.read_bytes())
    digest = hashlib.sha256(produced).hexdigest()  # hash, never store raw bytes

    assert digest == baseline.read_text().strip(), (
        f"{src.stem}: redacted output diverged from approved baseline"
    )

Step 3 — Generate baselines deliberately #Permalink to this section

Baselines are approved artifacts, not test byproducts. Generate them with an explicit, opt-in script so no run can silently overwrite them:

# scripts/approve_baselines.py  — run only after human review of outputs
import hashlib
from pathlib import Path

from redactor import redact_document

CORPUS = Path("corpus")
for src in sorted((CORPUS / "input").glob("*.pdf")):
    digest = hashlib.sha256(redact_document(src.read_bytes())).hexdigest()
    (CORPUS / "golden" / f"{src.stem}.sha256").write_text(digest + "\n")
    print(f"approved {src.stem}")

Step 4 — Run in parallel #Permalink to this section

pytest-xdist fans the parametrized cases across cores, which matters once the corpus grows into the thousands:

pytest test_batch_regression.py -n auto -q

Compliance checkpoint #Permalink to this section

This suite operationalizes NIST SP 800-53 Rev. 5 control SA-11 (Developer Testing and Evaluation), which requires developers to test and evaluate security-relevant changes at each iteration. A per-fixture regression run over a versioned corpus is documented, repeatable developer testing whose results are captured in the CI log, evidencing that the redaction ruleset was validated before release rather than after an exposure.

Gotchas #Permalink to this section

• Synthetic fixtures only. The corpus and its CI logs are permanent. Populate input/ with generated identifiers — Faker or a fixed synthetic set — never real client documents. One live Social Security number checked into fixtures is a reportable exposure.
• Timestamp and metadata nondeterminism. If redact_document does not normalize output, PDF CreationDate and a random /ID make the hash change every run and the test flakes. Ensure the redactor strips volatile metadata before hashing; the sibling page Deterministic Diff Assertions for Redacted PDFs covers the normalization.
• Baseline update discipline. Regenerating baselines to make a red build pass defeats the suite. Baseline changes must land in their own reviewed commit with a stated reason, so a reviewer sees exactly which outputs changed and why.
• Render throughput. PDF rendering dominates wall-clock; if the suite slows, revisit engine choice per pdfplumber vs PyMuPDF Performance rather than trimming the corpus.

Verification #Permalink to this section

A clean run over a 40-fixture corpus reports every case passing and is deterministic across repeated invocations:

$ pytest test_batch_regression.py -n auto -q
........................................                         [100%]
40 passed in 38.94s

Run it twice; identical pass counts and no flaky cases confirm the redactor output is deterministic and the baselines hold.

Related #Permalink to this section

This harness is the middle layer of CI/CD Redaction Validation: feed its outputs through Deterministic Diff Assertions for Redacted PDFs and enforce it on pull requests with the GitHub Actions Workflow for Redaction Validation. Part of the PDF and DOCX Parsing & Extraction Workflows field guide.