Redaction & Compliance Legal Document Automation

Dead-Letter Queue Escalation for Failed Reviews

When a review item repeatedly fails — a reviewer error, a corrupt page render, or a hard timeout — it must not vanish or loop forever. This page shows how to route poison items to a dead-letter queue with bounded retries and structured escalation to a senior reviewer or compliance queue, with every hop audited.

A silently dropped document in a legal redaction pipeline is a defensibility failure: it means content that should have been reviewed never was, and no record exists to prove otherwise. The dead-letter path is the safety net beneath the review surface described in Reviewer UI Patterns for Redaction Review Queues, turning “this item keeps failing” into a deliberate, logged escalation rather than an untracked disappearance. It complements the sync guarantees of the parent Human-in-the-Loop Override Sync guide.

Prerequisites & Version Matrix

Component Version Role
Python 3.11+ Runtime
Redis 7.2+ Queue and retry-counter store
pydantic 2.7+ Escalation record schema
pytest 8.2+ Verification

Each review item carries a stable item_id and a doc_id. The pipeline already emits audit events (see the parent guide); the DLQ path extends that stream. A senior-review or compliance queue must exist as an escalation target inside the same secure boundary as the primary queue.

Step 1 — Count failures with a bounded retry counter

A durable per-item counter, not an in-memory one, decides when an item is poison. Increment on every failure and compare against a fixed ceiling.

import redis

r = redis.Redis()
MAX_ATTEMPTS = 3

def record_failure(item_id: str, reason: str) -> int:
    """Increment the durable attempt counter and return the new count."""
    attempts = r.hincrby(f"review:{item_id}", "attempts", 1)
    r.hset(f"review:{item_id}", "last_reason", reason)
    emit_audit("REVIEW_FAILURE", item_id, actor="system",
               extra={"reason": reason, "attempts": attempts})
    return attempts

Step 2 — Route poison messages to the DLQ

Once attempts reach the ceiling, stop retrying and move the item to the dead-letter queue. The DLQ holds only references and metadata — never extracted plaintext — because the item still points at sensitive source material.

class DeadLetter(BaseModel):
    item_id: str
    doc_id: str
    attempts: int
    last_reason: str
    failed_at: float

def handle_failure(item_id: str, doc_id: str, reason: str) -> str:
    attempts = record_failure(item_id, reason)
    if attempts < MAX_ATTEMPTS:
        requeue(item_id)                       # bounded retry
        return "requeued"
    dl = DeadLetter(item_id=item_id, doc_id=doc_id, attempts=attempts,
                    last_reason=reason, failed_at=time.time())
    r.lpush("review:dlq", dl.model_dump_json())  # secure-boundary queue
    emit_audit("MOVED_TO_DLQ", item_id, actor="system",
               extra={"attempts": attempts, "reason": reason})
    return "dead_lettered"

Step 3 — Escalate to a senior/compliance queue

A DLQ is a holding pen, not a resolution. An escalation handler drains it, assigns each item to the senior-review or compliance queue, and records who now owns the decision. Escalation is itself an audited event, so the chain of custody is unbroken.

def escalate_dlq(batch: int = 50) -> int:
    escalated = 0
    for _ in range(batch):
        raw = r.rpop("review:dlq")
        if raw is None:
            break
        dl = DeadLetter.model_validate_json(raw)
        assign_to_queue("compliance_review", dl.item_id)
        r.hset(f"review:{dl.item_id}", "state", "ESCALATED")
        emit_audit("ESCALATED", dl.item_id, actor="system",
                   extra={"target": "compliance_review",
                          "attempts": dl.attempts})
        escalated += 1
    return escalated

Compliance Checkpoint

No document may be silently dropped. FRCP Rule 37(e) exposes a party to sanctions when electronically stored information that should have been preserved is lost because reasonable steps were not taken — a review item that fails and disappears is exactly that loss. Bounded retries plus mandatory escalation are the reasonable steps, and the audit stream is the proof they were taken. The DLQ and escalation events also satisfy NIST SP 800-53 Rev. 5 control AU-9 (Protection of Audit Information): because escalation records are written to the same protected, append-only audit store as every other action, the record of a failure cannot itself be tampered away.

Gotchas

  • Infinite retry loops. A retry without a durable ceiling will re-process a poison item forever, burning capacity and hiding the failure. The counter must be persistent and the comparison strict (attempts >= MAX_ATTEMPTS).
  • DLQ items still carry sensitive references. A dead-lettered record names a doc_id that resolves to privileged content. Keep the DLQ inside the same encrypted, access-controlled boundary as the live queue; never spill it to a general-purpose log or a shared bug tracker.
  • Silent DLQ growth. A filling dead-letter queue is a signal, not a resting state. Alert on DLQ depth and on escalation age so a systemic render bug surfaces before a discovery deadline does.
  • Non-idempotent escalation. If escalate_dlq can crash mid-batch, use RPOPLPUSH into a processing list so an item is never lost between pop and assign.

Verification

This pytest asserts an item that exceeds the retry ceiling lands in the DLQ rather than being requeued again.

def test_exceeding_max_retries_lands_in_dlq():
    item_id, doc_id = "item-9", "doc-9"
    r.delete(f"review:{item_id}", "review:dlq")
    results = [handle_failure(item_id, doc_id, "render_error")
               for _ in range(MAX_ATTEMPTS)]
    assert results[:-1] == ["requeued"] * (MAX_ATTEMPTS - 1)
    assert results[-1] == "dead_lettered"          # final attempt dead-letters
    assert r.llen("review:dlq") == 1               # exactly one DLQ entry

Running pytest -q prints 1 passed: the first two failures requeue, the third dead-letters, and precisely one item sits in the DLQ.

The failures handled here originate at the Reviewer UI Patterns for Redaction Review Queues surface, and timeouts that trip the retry counter are governed by SLA-Based Timeout Handling for Review Queues. Where a low score is the root cause, tuning in Confidence Threshold Configuration reduces the volume reaching the DLQ. Part of the Human-in-the-Loop Override Sync section of the PII Detection & Automated Redaction Patterns field guide.

Related guide Reviewer UI Patterns for Redaction Review Queues A redaction review queue must let a human accept or reject flagged spans quickly while never shipping raw personally... Related guide SLA-Based Timeout Handling for Review Queues Every item in a redaction review queue needs a deadline: a claimed item that a reviewer abandons must return to the...