Celery vs RQ vs Dramatiq for Redaction Queues

Selecting a task queue for asynchronous redaction is a trade-off between feature breadth and operational cost, and the right answer depends on whether you need priority routing, multiple brokers, and how much delivery-guarantee plumbing you are willing to own. This guide compares Celery, RQ, and Dramatiq for exactly that workload.

Redaction jobs are long-running, PII-bearing, and audited, which raises the bar over a generic task queue. Delivery must be at-least-once, tasks must be idempotent so a redelivery never double-writes an audit record, and long OCR steps must not trip a broker’s visibility timeout. This companion to the Parsing Engine Selection Benchmarks framework covers the queue row of that matrix.

Prerequisites and version matrix #Permalink to this section

RQ and Dramatiq run on Redis alone. Celery adds RabbitMQ support and a richer result backend, at the cost of more moving parts to operate.

Trade-off comparison #Permalink to this section

Step-by-step implementation #Permalink to this section

1. Define the same redaction task in each framework. The body is identical — hash the artifact, redact, append to the audit log — only the decorator and retry wiring differ.

# Celery: explicit acks_late so a crashed worker redelivers the message.
from celery import Celery

app = Celery("redaction", broker="redis://localhost:6379/0")

@app.task(bind=True, acks_late=True, max_retries=3, default_retry_delay=30)
def redact_document_celery(self, doc_id: str, artifact_sha256: str):
    try:
        _redact(doc_id, artifact_sha256)   # idempotent on artifact_sha256
    except TransientError as exc:
        raise self.retry(exc=exc)

# RQ: retries are opt-in via the Retry helper at enqueue time.
from rq import Queue, Retry
from redis import Redis

queue = Queue("redaction", connection=Redis())

def redact_document_rq(doc_id: str, artifact_sha256: str):
    _redact(doc_id, artifact_sha256)

# queue.enqueue(redact_document_rq, doc_id, sha, retry=Retry(max=3, interval=30))

# Dramatiq: retries and time limits are middleware defaults, set per-actor.
import dramatiq
from dramatiq.brokers.redis import RedisBroker

dramatiq.set_broker(RedisBroker(url="redis://localhost:6379/0"))

@dramatiq.actor(max_retries=3, min_backoff=30_000, time_limit=1_800_000)
def redact_document_dramatiq(doc_id: str, artifact_sha256: str):
    _redact(doc_id, artifact_sha256)       # same idempotent body

2. Make the task idempotent. Key the redaction on the artifact’s SHA-256 so a redelivered message is a no-op rather than a duplicate audit entry.

def _redact(doc_id: str, artifact_sha256: str) -> None:
    if audit_store.exists(artifact_sha256):
        return                              # already processed; safe redelivery
    result = run_redaction(doc_id)          # produces redacted artifact
    audit_store.append(artifact_sha256, result.output_sha256)

Compliance checkpoint #Permalink to this section

At-least-once delivery plus idempotency is what keeps the audit trail complete. Under NIST SP 800-53 Rev. 5 control AU-9 (Protection of Audit Information), audit records must be protected against loss and unauthorized modification; a task that is dropped on worker failure leaves a gap, while one that runs twice risks a conflicting record. Choosing at-least-once delivery (Celery’s acks_late, Dramatiq’s default redelivery) and enforcing idempotency on the artifact hash satisfies both halves — no lost record, no duplicate. Persist each task’s message id and artifact hash so redeliveries are auditable, consistent with the immutable logging described across the PDF and DOCX Parsing & Extraction Workflows guide.

Gotchas #Permalink to this section

• Visibility timeout versus long OCR jobs. Redis-backed brokers redeliver a message if it is not acknowledged within the visibility timeout; a 25-minute OCR job under a default 30-minute timeout is one slow page from being processed twice. Raise the timeout above your p99 job duration.
• Duplicate delivery needs idempotent tasks. At-least-once means occasional redelivery by design. Without the hash guard above, a redraw appends a second audit record and can double-mask a document.
• Dead-letter support differs. Dramatiq ships a built-in dead queue for exhausted retries; RQ has no native DLQ, so failed jobs need an explicit failed-job registry sweep; Celery needs routing configuration to emulate one. Wire escalation accordingly.
• Result backend growth. Celery’s result backend accumulates state; set result_expires or it becomes an unbounded store of task metadata.

Verification and testing #Permalink to this section

Assert that a failing task is retried and that a redelivery stays idempotent. Dramatiq’s stub broker makes retries deterministic in-process.

import dramatiq
from dramatiq.brokers.stub import StubBroker

def test_task_is_idempotent_and_retried(monkeypatch):
    broker = StubBroker()
    dramatiq.set_broker(broker)
    calls = {"n": 0}

    def fake_redact(doc_id, sha):
        calls["n"] += 1
        if calls["n"] == 1:
            raise TransientError("first attempt fails")
    monkeypatch.setattr("mymod._redact", fake_redact)

    redact_document_dramatiq.send("doc-1", "abc123")
    broker.join(redact_document_dramatiq.queue_name)
    # Retried once (2 invocations) but audit written exactly once.
    assert calls["n"] == 2
    assert audit_store.count("abc123") == 1

For CPU-bound OCR workers Dramatiq’s defaults usually win on ops burden; reach for Celery only when priority routing or a non-Redis broker is a hard requirement, and for a small Redis-only footprint RQ is the least to operate.

Related #Permalink to this section

Size these queues against the throughput figures in Parsing Engine Selection Benchmarks and the fleet design in Async Batch Processing Pipelines. The sibling Tesseract vs EasyOCR for Scanned Legal Documents covers the long OCR jobs these workers run. Part of the PDF and DOCX Parsing & Extraction Workflows field guide.