Why Workflow DAGs Break at 2,500+ Tenants: Moving Multi-Tenant RAG from Batch Orchestration to Event-Driven Pub/Sub
When building an enterprise Retrieval-Augmented Generation (RAG) platform, the intuitive architectural instinct is to reach for a workflow orchestrator. Tools like Prefect, Airflow, or Dagster look ideal on paper. You define clean directed acyclic graphs: crawl the sitemap, render the DOM, chunk the text, compute embeddings, and upsert vectors. You get visual run dashboards, automatic retries, and task dependencies out of the box. For ten tenants running scheduled nightly batch crawls, workflow DAGs work reliably. In our infrastructure, Prefect 3.x remains an excellent choice for scheduled batch reconciliations, nightly catalog syncs, and administrative ETL workflows. However, for live, multi-tenant ingestion across 2,500+ enterprise tenants handling continuous web updates, unstructured document uploads, and real-time CMS edits, centralized DAG schedulers hit severe mechanical limits. Centralized DAG orchestration assumes batch workloads with predictable state transitions. Multi-tenant enterprise ingestion is not batch; it is erratic, continuous, event-driven, and structurally asynchronous. When thousands of heterogeneous client sites emit micro-deltas concurrently, forcing those updates into heavyweight DAG schedulers burns through infrastructure budgets while introducing seconds of synthetic scheduling lag. Here is why workflow DAGs hit a scaling wall for live workloads, and why we decoupled our ingestion pipeline into an event-driven Pub/Sub streaming architecture. ...