Parsing Engine Selection Benchmarks

Choosing the extraction, OCR, and queue engines for a legal redaction stack is an engineering decision that should rest on measured throughput and accuracy figures, not vendor folklore or a colleague’s habit. This section gives a stage-by-stage framework for picking the right tool against representative litigation corpora, benchmarking it reproducibly, and recording the result so the choice survives an evidentiary challenge. It extends the broader PDF and DOCX Parsing & Extraction Workflows field guide with numbers you can defend.

A redaction pipeline is really three benchmark problems stacked together: native text extraction, optical character recognition for scanned exhibits, and the task queue that fans work across workers. Each stage has a different bottleneck, so the “best” engine at one stage is rarely the best at the next. The job here is to score candidates on the metrics that actually gate legal work — coordinate fidelity, character accuracy, and audit completeness — and to keep the measurement harness in version control alongside the code it evaluates.

Selection matrix for legal workloads #Permalink to this section

The table below summarizes measured trade-offs on a mixed 500-page litigation corpus (native text, scanned exhibits at 300 DPI, and multi-column filings), run on a 4 vCPU / 16 GB worker. Throughput figures are per-engine at its own stage; they are not directly comparable across rows.

Two child guides drill into the rows that carry the most risk: Tesseract vs EasyOCR for Scanned Legal Documents quantifies character error rate against real filings, and Celery vs RQ vs Dramatiq for Redaction Queues compares delivery guarantees and operational cost.

Verified benchmark harness #Permalink to this section

A defensible benchmark is deterministic and reproducible. Use perf_counter for wall time, run several trials, and report the median rather than a single hot run. Never log raw page content — record only hashed identifiers.

import hashlib
import statistics
from time import perf_counter
from typing import Callable

def bench_engine(name: str, extract: Callable[[str], str],
                 pdf_path: str, page_count: int, trials: int = 5) -> dict:
    """Measure median pages/sec for a text-extraction callable."""
    durations = []
    digest = None
    for _ in range(trials):
        start = perf_counter()
        text = extract(pdf_path)            # returns concatenated page text
        durations.append(perf_counter() - start)
        # Hash the output so runs are comparable without logging PII.
        digest = hashlib.sha256(text.encode("utf-8")).hexdigest()
    median = statistics.median(durations)
    return {
        "engine": name,
        "pages_per_sec": round(page_count / median, 2),
        "median_seconds": round(median, 3),
        "output_sha256": digest,          # identical across engines => same text
    }

def extract_pymupdf(path: str) -> str:
    import fitz
    with fitz.open(path) as doc:
        return "\n".join(page.get_text() for page in doc)

def extract_pdfplumber(path: str) -> str:
    import pdfplumber
    with pdfplumber.open(path) as pdf:
        return "\n".join(p.extract_text() or "" for p in pdf.pages)

if __name__ == "__main__":
    corpus, pages = "fixtures/discovery_500p.pdf", 500
    for row in (bench_engine("PyMuPDF", extract_pymupdf, corpus, pages),
                bench_engine("pdfplumber", extract_pdfplumber, corpus, pages)):
        print(f"{row['engine']:>11}: {row['pages_per_sec']:>6} pages/s")

Compliance alignment #Permalink to this section

Documenting why each engine was selected is not busywork — it is discovery hygiene. Under FRCP Rule 34(b)(2)(E), a producing party must extract and produce electronically stored information in a reasonably usable form, and a recorded, reproducible benchmark shows the selection was principled rather than arbitrary. The evaluation process itself maps to NIST SP 800-53 Rev. 5 control SA-4 (Acquisition Process), which requires documented evaluation criteria for acquired components. Persist the harness output — engine version, corpus hash, median throughput, and accuracy figures — into the same immutable audit record that captures the redaction run.

Performance and scale note #Permalink to this section

On the reference 4 vCPU worker, PyMuPDF sustains roughly 1,800–2,400 native pages/hour per process; four processes clear a 500-page discovery set in under a minute. Tesseract, being CPU-bound, drops that to about 2,000 OCR pages/hour per worker, which is why scanned exhibits are the true throughput ceiling. Budget capacity against the OCR row, not the text row — a corpus that is 20% scanned will spend 80% of its wall-clock time in OCR. Cross-check these numbers against pdfplumber vs PyMuPDF Performance before committing an SLA.

Edge cases and failure handling #Permalink to this section

• Benchmarking on an unrepresentative corpus. A harness fed clean, single-column PDFs will crown the fastest engine, then collapse in production on rotated scans and merged tables. Remediation: assemble the fixture set from a stratified sample of real matter types and freeze it in version control.
• Ignoring tail latency. Reporting mean pages/sec hides the 99th-percentile page — a 400-cell exhibit that stalls a worker for minutes. Remediation: record p50/p95/p99 and size Async Batch Processing Pipelines visibility timeouts against the tail, not the median.
• License incompatibility discovered late. PyMuPDF ships under AGPL-3.0 and Dramatiq under LGPL-3.0; adopting them in a closed-source product without a commercial or compliance review invites a legal problem inside the legal-tech stack. Remediation: gate the SA-4 evaluation on a license check before any throughput number is trusted.
• OCR engine chosen without a GPU budget. EasyOCR benchmarks beautifully on a GPU box and then runs 5–10x slower on the CPU-only fleet that actually processes discovery. Remediation: benchmark on the target hardware profile, as detailed in Handling Scanned PDFs with OCR.

Related #Permalink to this section

Once the benchmark picks are locked, drill into the two decisions that carry the most compliance weight: Tesseract vs EasyOCR for Scanned Legal Documents and Celery vs RQ vs Dramatiq for Redaction Queues. For the text-extraction row see pdfplumber vs PyMuPDF Performance, and for wiring the chosen engines into a worker fleet see Async Batch Processing Pipelines. Part of the PDF and DOCX Parsing & Extraction Workflows field guide.