Reciprocal Rank Fusion vs. Cross-Encoders: Designing the Sub-50ms Reranking Funnel
In enterprise Retrieval-Augmented Generation (RAG) and neural search, relying on a single retrieval modality is rarely sufficient on its own. Standardizing solely on dense vector search leaves notable blind spots for exact-match keywords, alphanumeric identifiers (e.g., part numbers, error codes, legal statute citations), and out-of-vocabulary acronyms. Consequently, production architectures have converged on hybrid retrieval: querying a dense vector index (ANN) and a sparse inverted index (BM25) in parallel. However, once multiple candidate lists are retrieved, engineering teams face a critical architectural decision: How do you merge and rerank candidate documents before context-window injection? While Reciprocal Rank Fusion (RRF) and Cross-Encoder neural rerankers are often treated as competing alternatives, they operate at completely different computational complexities and solve distinct phases of the retrieval lifecycle. This article provides the mathematical foundation, latency estimates, and architectural blueprint for chaining both into an asymmetric sub-50ms reranking funnel. ...