Automated Fallback Routing for High-Risk Files

When primary automated redaction pipelines encounter degraded OCR, encrypted payloads, or ambiguous entity boundaries, unmitigated exposure directly violates data retention and privacy mandates. High-risk legal documents—PII-heavy discovery sets, cross-border M&A term sheets, and regulated financial disclosures—require deterministic fallback routing to prevent pipeline bypass and maintain audit integrity. This implementation details a production-grade fallback architecture that isolates, classifies, and securely escalates failed redactions while preserving document lifecycle security boundaries. The routing layer integrates directly with the broader Legal Document Redaction Architecture & Compliance Mapping framework to enforce state transitions, compliance assertions, and cryptographic audit trails.

1. Deterministic Risk Detection & Pre-Flight Parsing #

Fallback routing must trigger only after deterministic risk scoring, not heuristic guesswork. The pre-flight parser extracts cryptographic fingerprints, evaluates redaction confidence thresholds, and flags jurisdictional markers before any routing decision executes. This ensures that partial or unverified redactions never commit to downstream storage.

import hashlib
import json
from dataclasses import dataclass, asdict
from typing import Dict, List, Optional
from datetime import datetime, timezone

@dataclass(frozen=True)
class RedactionRiskProfile:
    doc_id: str
    content_hash: str
    primary_confidence: float
    pii_density: int
    jurisdiction_flags: List[str]
    encryption_detected: bool
    fallback_required: bool
    evaluated_at: str

def evaluate_risk(doc_metadata: Dict, redaction_result: Dict) -> RedactionRiskProfile:
    # Thresholds calibrated for legal discovery and regulatory compliance
    CONFIDENCE_CUTOFF = 0.82
    PII_DENSITY_CUTOFF = 15
    
    has_low_confidence = redaction_result.get("avg_confidence", 1.0) < CONFIDENCE_CUTOFF
    has_high_pii = redaction_result.get("pii_count", 0) > PII_DENSITY_CUTOFF
    encrypted = doc_metadata.get("content_type") in (
        "application/pdf; encrypted", 
        "application/x-7z-compressed", 
        "application/octet-stream"
    )
    
    # Generate immutable content fingerprint for chain-of-custody tracking
    raw_content = doc_metadata.get("raw_payload", b"")
    content_hash = hashlib.sha256(raw_content).hexdigest() if isinstance(raw_content, bytes) else hashlib.sha256(raw_content.encode()).hexdigest()
    
    return RedactionRiskProfile(
        doc_id=doc_metadata["id"],
        content_hash=content_hash,
        primary_confidence=redaction_result.get("avg_confidence", 1.0),
        pii_density=redaction_result.get("pii_count", 0),
        jurisdiction_flags=doc_metadata.get("jurisdiction_tags", []),
        encryption_detected=encrypted,
        fallback_required=has_low_confidence or has_high_pii or encrypted,
        evaluated_at=datetime.now(timezone.utc).isoformat()
    )

The parser executes synchronously before any storage sync operation. If fallback_required evaluates to True, the document state transitions to COMPLIANCE_HOLD and routes to an isolated queue. Primary pipeline execution halts immediately to prevent partial redaction commits. Jurisdictional routing logic must align with regional privacy frameworks, particularly when handling cross-border data transfers governed by GDPR vs CCPA Redaction Requirements.

2. Cryptographic Isolation & Queue Architecture #

Fallback queues must enforce cryptographic isolation, strict TTL policies, and immutable audit logging. The configuration below implements AWS SQS with SSE-KMS encryption, dead-letter routing, and visibility timeouts tuned for legal review SLAs.

# fallback_queue_config.yaml
queue:
  name: redaction-fallback-high-risk
  visibility_timeout: 300
  message_retention_period: 1209600  # 14 days
  delay_seconds: 0
  maximum_message_size: 262144  # 256 KB
  receive_message_wait_time: 20
  encryption:
    type: SSE-KMS
    kms_key_id: "alias/redaction-fallback-kms"
    data_key_reuse_period: 3600
  dead_letter_queue:
    name: redaction-fallback-dlq
    max_receive_count: 3
    retention_period: 1209600  # 14 days (SQS maximum)
  tags:
    compliance_scope: "legal-discovery"
    data_classification: "high-risk"
    audit_enabled: "true"

Messages pushed to this queue contain only metadata references and cryptographic pointers, never raw document payloads. This design minimizes blast radius and aligns with NIST SP 800-88 Compliance Mapping guidelines for secure media sanitization and controlled data retention. For infrastructure-as-code deployment patterns and IAM boundary configurations, reference the implementation guide for Setting Up Secure Fallback Queues for Failed Redactions.

3. State Machine & Escalation Routing Logic #

Once quarantined, documents enter a finite state machine that governs escalation paths. The routing engine evaluates the RedactionRiskProfile to determine whether to trigger:

1. Secondary ML Pipeline: Re-runs OCR with higher-fidelity models or jurisdiction-specific entity recognizers.
2. Human-in-the-Loop (HITL) Review: Routes to a secure legal review workspace with role-based access controls.
3. Compliance Hold: Freezes processing pending external counsel or regulatory clearance.

def route_fallback_document(profile: RedactionRiskProfile) -> str:
    if profile.encryption_detected:
        return "SECURE_DECRYPTION_WORKFLOW"
    if profile.pii_density > 50:
        return "HITL_LEGAL_REVIEW"
    if "EU" in profile.jurisdiction_flags or "CA" in profile.jurisdiction_flags:
        return "CROSS_BORDER_COMPLIANCE_REVIEW"
    return "SECONDARY_ML_PASS"

State transitions are logged as JSON events with HMAC-SHA256 signatures to prevent tampering. Each transition generates a compliance assertion that maps to the organization’s internal data governance policy. The routing layer must reject any out-of-sequence state changes and emit STATE_VIOLATION alerts to the security operations center.

4. Audit Trail & Compliance Assertion Mapping #

Every fallback event must produce an immutable audit record that survives document lifecycle transitions. The audit schema captures:

• Pre-flight risk scores and threshold evaluations
• Queue enqueue/dequeue timestamps and visibility window expirations
• Routing decisions and reviewer assignments
• Final disposition (redacted, exempted, destroyed, or archived)

Compliance officers require these logs to demonstrate due diligence during regulatory audits. The audit trail should be exported to a WORM (Write-Once-Read-Many) storage tier or blockchain-backed ledger where legally mandated. AWS SQS dead-letter queue configurations, as documented in AWS SQS Developer Guide, ensure that failed processing attempts do not silently drop audit events. All retention periods must be explicitly mapped to jurisdictional mandates and internal legal hold policies.

5. Operational Hardening & Observability #

Production fallback routing requires continuous monitoring to prevent queue saturation and SLA degradation. Implement the following operational controls:

• Circuit Breakers: Halt fallback ingestion if queue depth exceeds 10,000 messages or if DLQ consumption rate drops below 50 messages/hour.
• Metrics & Alerting: Track fallback_trigger_rate, mean_time_to_review, and redaction_confidence_recovery. Alert on p95 latency > 45 seconds.
• Least-Privilege IAM: Isolate fallback workers in dedicated subnets with VPC endpoints. Restrict KMS decryption permissions to the routing service role only.
• Payload Sanitization: Strip all executable content, macros, and embedded OLE objects before queue serialization. Validate against a strict JSON schema using pydantic or equivalent.

Automated fallback routing for high-risk files transforms redaction failures from compliance liabilities into auditable, controlled workflows. By enforcing deterministic risk scoring, cryptographic queue isolation, and immutable state tracking, legal technology teams can maintain pipeline velocity without sacrificing regulatory posture or document integrity.