Redaction & Compliance Legal Document Automation

Deterministic Diff Assertions for Redacted PDFs

A deterministic diff assertion proves a redacted PDF differs from its source in exactly the expected regions and nowhere else, which requires normalizing away the volatile bytes PDF writers inject on every save before any comparison can be trusted. This how-to, part of CI/CD Redaction Validation, makes redacted output byte-stable and region-precise.

Prerequisites and version matrix

Package Version Role
Python 3.11.x Runtime
PyMuPDF (fitz) 1.24.x Text and rectangle extraction, metadata scrub
pytest 8.2.x Assertion harness

Install with pip install "PyMuPDF==1.24.*" "pytest==8.2.*". PyMuPDF exposes the redaction annotations and page geometry needed to compare regions rather than raw bytes.

Why naive byte-diff fails

Two redactions of the same input almost never produce identical bytes. PDF writers stamp a fresh CreationDate and ModDate, generate a random /ID array, subset embedded fonts differently, and renumber cross-reference (xref) entries on each save. A cmp or sha256 over the raw file therefore flags every run as changed even when the visible, redacted content is identical. The fix is to compare a normalized representation.

Step 1 — Strip volatile metadata

Scrub the fields that change per save and pin a fixed document ID so the normalized bytes are stable:

# normalize.py
import fitz  # PyMuPDF


def normalize_pdf(data: bytes) -> bytes:
    doc = fitz.open(stream=data, filetype="pdf")
    # Clear timestamps and producer/creator strings that vary per run.
    doc.set_metadata({})
    # Pin the /ID so it is not regenerated randomly on save.
    doc.xref_set_key(-1, "ID", "[<00><00>]")
    out = doc.tobytes(garbage=4, deflate=True, clean=True)
    doc.close()
    return out

garbage=4 and clean=True compact and reorder the xref deterministically, removing churn from freed objects so the same logical document always serializes the same way.

Step 2 — Extract text and redaction regions

Compare structure, not bytes. Pull the residual text and the applied redaction rectangles from each page:

# extract.py
import fitz


def redaction_rects(data: bytes):
    doc = fitz.open(stream=data, filetype="pdf")
    result = {}
    for page in doc:
        rects = [
            tuple(round(v, 1) for v in annot.rect)  # round to kill sub-pixel noise
            for annot in page.annots(types=(fitz.PDF_ANNOT_REDACT,))
        ]
        result[page.number] = {
            "text": page.get_text("text"),
            "rects": sorted(rects),
        }
    doc.close()
    return result

Rounding coordinates to one decimal absorbs sub-pixel differences from font metrics while still catching any real movement of a redaction box.

Step 3 — Assert the exact region set

The assertion checks two things: the redacted regions equal the expectation, and no protected text survived anywhere on the page:

# test_diff_assertion.py
from extract import redaction_rects
from normalize import normalize_pdf

EXPECTED = {
    0: [(72.0, 108.0, 210.4, 121.6)],  # one redaction on page 0
}


def test_redacted_regions_are_exact():
    produced = normalize_pdf(open("corpus/input/nda_0001.pdf", "rb").read())
    regions = redaction_rects(produced)

    assert regions[0]["rects"] == EXPECTED[0], "redaction rectangles drifted"
    # The known synthetic token must not appear in any residual text.
    assert "471-55-8829" not in regions[0]["text"], (
        "protected value survived redaction"
    )

Compliance checkpoint

FRCP Rule 34(b) governs the form of documents produced in discovery and the obligation to produce them as they are kept or in a reasonably usable form, with their integrity intact. A deterministic assertion that the produced PDF differs from the source only in the sanctioned regions gives a defensible, reproducible record that redactions removed exactly what was intended and altered nothing else — the evidentiary basis for defending a production if the redaction methodology is later challenged.

Gotchas

  • /ID and timestamps break byte-diff. Always normalize before comparing; the random /ID array alone guarantees a false positive on every run.
  • Font subsetting changes. Re-saving can re-subset embedded fonts, shifting bytes and even sub-pixel glyph widths. Comparing extracted text plus rounded rectangles sidesteps font-level churn.
  • xref churn. Freed and renumbered objects reorder the file; garbage=4, clean=True canonicalizes the xref so serialization is stable.
  • True redaction vs overlay. A black rectangle drawn over text leaves the text extractable underneath. Assert on page.get_text() residual content, not just visual rects, so an overlay-only bug fails the test.

Verification

With a stable normalizer, the assertion passes identically on repeat runs:

$ pytest test_diff_assertion.py -q
.                                                                [100%]
1 passed in 0.42s

Normalize the same input twice and confirm the byte streams are equal (normalize_pdf(a) == normalize_pdf(a)), proving the comparison base is deterministic before it gates any merge.

These assertions harden the outputs produced by Batch Redaction Regression Tests with pytest and are enforced on every pull request by the GitHub Actions Workflow for Redaction Validation. For the rendering-engine trade-offs behind extraction speed, see pdfplumber vs PyMuPDF Performance. Part of the CI/CD Redaction Validation guide.

Related guide Batch Redaction Regression Tests with pytest A batch redaction regression suite redacts every fixture in a golden corpus and asserts each output still matches an... Related guide GitHub Actions Workflow for Redaction Validation A GitHub Actions workflow turns the redaction regression suite into a required merge gate: it installs dependencies,...