Why Workflow DAGs Break at 2,500+ Tenants: Moving Multi-Tenant RAG from Batch Orchestration to Event-Driven Pub/Sub

When building an enterprise Retrieval-Augmented Generation (RAG) platform, the intuitive architectural instinct is to reach for a workflow orchestrator. Tools like Prefect, Airflow, or Dagster look ideal on paper. You define clean directed acyclic graphs: crawl the sitemap, render the DOM, chunk the text, compute embeddings, and upsert vectors. You get visual run dashboards, automatic retries, and task dependencies out of the box.

For ten tenants running scheduled nightly batch crawls, workflow DAGs work reliably. In our infrastructure, Prefect 3.x remains an excellent choice for scheduled batch reconciliations, nightly catalog syncs, and administrative ETL workflows.

However, for live, multi-tenant ingestion across 2,500+ enterprise tenants handling continuous web updates, unstructured document uploads, and real-time CMS edits, centralized DAG schedulers hit severe mechanical limits. Centralized DAG orchestration assumes batch workloads with predictable state transitions. Multi-tenant enterprise ingestion is not batch; it is erratic, continuous, event-driven, and structurally asynchronous. When thousands of heterogeneous client sites emit micro-deltas concurrently, forcing those updates into heavyweight DAG schedulers burns through infrastructure budgets while introducing seconds of synthetic scheduling lag. Here is why workflow DAGs hit a scaling wall for live workloads, and why we decoupled our ingestion pipeline into an event-driven Pub/Sub streaming architecture.

1. The Scaling Wall of Centralized Workflow DAGs

Workflow engines are fundamentally centralized orchestrator databases. Every task transition requires the worker to emit heartbeats and write state back to a central PostgreSQL or cloud backend. This creates a severe choke point.

At enterprise scale, this design introduces three fatal bottlenecks:

1. Scheduling Overhead Dwarfs Compute Time

In modern hybrid search, many ingestion operations are tiny. Microscopic, even. Chunking a 500-word blog post or generating an embedding vector takes between 15 ms and 80 ms.

In a DAG orchestrator, submitting a task run, serializing parameters, persisting state to the orchestrator database, and dispatching to a worker often consumes 1.5 to 4.0 seconds per task.

You end up burning 98% of your compute budget waiting on orchestrator database transactions rather than processing tokens. Pure waste.

2. Multi-Tenant Head-of-Line Blocking

Imagine tenant 412 triggers a crawl of a legacy product catalog containing 40,000 malformed, deep-nested pages with broken JavaScript. If tasks share a worker concurrency pool, tenant 412 floods the orchestrator’s task queue.

A different tenant publishing an urgent pricing correction or hotel amenity update gets queued behind 40,000 slow web fetches. In a centralized DAG engine, isolating tenant concurrency limits requires complex dynamic queue tags that degrade orchestrator scheduling throughput across the entire cluster.

3. The All-or-Nothing Flow Failure Trap

DAG engines structure workflows around the concept of a “Flow Run”. If three pages out of a 2,000-page crawl throw transient headless browser timeouts, handling those partial failures requires complex sub-flow nesting.

If an unhandled exception bubbles up, the entire flow run marks as failed, triggering costly full-batch retries or requiring manual operator intervention. That burns engineering hours.

2. The Architectural Paradigm Shift: Event-Driven Pub/Sub

To serve thousands of commercial tenants reliably, you must invert the control model. You eliminate the central orchestrator and replace it with decoupled message topics and independent worker pools.

Instead of a monolithic flow managing an end-to-end pipeline, the system treats ingestion as a choreography of fine-grained, stateless events:

+-----------------------------------------------------------------------------+
| MULTI-TENANT EVENT BUS (Azure Service Bus / Kafka / Redis Streams)          |
+-----------------------------------------------------------------------------+
        |                                   |                           |
  [crawl.discovered]                  [dom.rendered]             [text.chunked]
        |                                   |                           |
        v                                   v                           v
+------------------+              +--------------------+     +------------------+
| Playwright Pool  |              | Chonkie Extractors |     | Local Inference  |
| (Headless DOM)   |              | (Semantic Parser)  |     | (Embed & Upsert) |
+------------------+              +--------------------+     +------------------+
        |                                   |                           |
        +-- emits [dom.rendered]            +-- emits [text.chunked]    +-- emits [vector.upserted]

Discrete Message Topics

1. topic:crawl.discovered: Carries tenant metadata, depth limits, and target URLs. Consumed by lightweight HTTP and sitemap parsers.
2. topic:dom.rendered: Emitted when a URL requires headless JavaScript execution. Consumed by dedicated Playwright worker pools.
3. topic:text.chunked: Emitted when clean DOM or document text is ready. Consumed by CPU-bound parsing workers running semantic chunking (chonkie).
4. topic:vector.upserted: Emitted with tokenized passages. Consumed by GPU/CPU inference workers calling local embedding engines (sentence-transformers) and updating vector indexes.

Independent Worker Scaling

Different stages of document processing have fundamentally different hardware requirements:

• Playwright workers are memory-heavy and network I/O-bound. They need generous RAM to run Chromium containers without OOM crashes.
• Semantic chunking workers are pure CPU-bound Python processes.
• Embedding workers require AVX-512 CPU extensions or dedicated GPU acceleration.

In a DAG orchestrator, workers often run monolithic environments packaging every dependency. In a Pub/Sub model, each worker pool runs an optimized, minimal container image that auto-scales independently based on its specific queue depth.

3. Production Implementation: Asynchronous Worker Pipelines

Here is how an event-driven worker processes ingestion events using asynchronous queue consumers with backpressure control and dead-letter isolation:

import asyncio
import json
from typing import Optional
from pydantic import BaseModel, HttpUrl
from tenacity import retry, stop_after_attempt, wait_exponential

class IngestionEvent(BaseModel):
    tenant_id: str
    document_id: str
    url: HttpUrl
    retry_count: int = 0

class TextChunkedEvent(BaseModel):
    tenant_id: str
    document_id: str
    chunks: list[str]
    token_count: int

class QueueConsumerWorker:
    def __init__(self, consumer_name: str, queue_client):
        self.consumer_name = consumer_name
        self.queue = queue_client

    async def start_consumer_loop(self):
        while True:
            # Fetch message with peek-lock semantics
            messages = await self.queue.receive_messages(max_messages=10, max_wait_time=5)
            tasks = [self.process_envelope(msg) for msg in messages]
            await asyncio.gather(*tasks)

    async def process_envelope(self, message):
        try:
            payload = json.loads(message.body)
            event = IngestionEvent(**payload)
            
            # Execute business logic with local retry
            result = await self.execute_chunking(event)
            
            # Publish next event down the line
            await self.queue.publish_event("topic:text.chunked", result.model_dump_json())
            
            # Complete message cleanly
            await message.complete()
            
        except Exception as exc:
            # Dead-letter queue isolation
            if message.delivery_count > 3:
                await message.dead_letter(
                    reason="MaxRetriesExceeded",
                    error_description=str(exc)
                )
            else:
                await message.abandon()

    @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
    async def execute_chunking(self, event: IngestionEvent) -> TextChunkedEvent:
        # Perform deterministic semantic extraction
        text = await fetch_document_content(event.url)
        chunks = chunk_text_semantically(text, max_tokens=256)
        
        return TextChunkedEvent(
            tenant_id=event.tenant_id,
            document_id=event.document_id,
            chunks=chunks,
            token_count=sum(len(c.split()) for c in chunks)
        )

4. The Empirical Systems Comparison

Comparing operational metrics across identical multi-tenant workloads (2,500 tenants, 1.2M web pages, and 85,000 PDF documents):

5. When Workflow DAGs Still Make Sense

Pub/Sub is not a universal replacement for all data engineering. There are specific domains where workflow DAG orchestrators remain superior:

1. Nightly Financial & Accounting Reconciliations: When execution order is strictly sequential, and if step 4 fails, step 5 must never run under any circumstance.
2. Model Training & Evaluation Sweeps: When you are fine-tuning weights and require rich visual artifact lineage, parameter tracking, and interactive DAG inspection.
3. Low-Volume Enterprise ETL: If you serve thirty clients and your team does not have the operational capacity to manage partitioned message topics, dead-letter monitoring, and distributed tracing.

Architectural Verdict

For high-throughput, multi-tenant RAG, workflow DAGs represent the wrong mental model. They impose synchronous batch constraints onto continuous streaming realities.

By decoupling ingestion into fine-grained Pub/Sub message topics, you eliminate orchestrator bottlenecks, isolate tenant blast radiuses, and allow heterogeneous worker fleets to scale on their own terms. When an enterprise tenant updates their website, the change should land in your vector database in hundreds of milliseconds, not after the next orchestrator flow run wakes up.