Your retriever returns twenty chunks. Three of them actually answer the question, and two of those three sit at rank 14 and rank 19. That gap between what vector search pulls back and what your model actually reads is the cheapest thing to fix in a retrieval pipeline, and a RAG reranker is how you fix it.
Embedding models have improved and context windows have grown, but neither change removed the need for a second-stage model that scores query and document together. This guide compares the three rerankers most teams actually shortlist in 2026 – Cohere Rerank, Jina Reranker v3 and BGE Reranker v2-m3 – on accuracy, latency, licensing and operational cost.
What a RAG reranker actually does
Vector search uses a bi-encoder. The query and every document are embedded separately, then compared with cosine similarity. That separation is what makes the index fast, and it is also what makes it approximate: the model never sees the query and the document at the same time.
A reranker is a cross-encoder. It takes the query and one candidate document as a single input and outputs a relevance score. Because attention runs across both, it catches negation, qualifiers, entity mismatches and intent that embeddings routinely blur. The cost is that scoring is no longer precomputed – every pair runs through the model at query time.
The standard pattern is two-stage retrieval:
- Stage one: retrieve 50 to 100 candidates with vector search, BM25 or a hybrid of both. Optimise for recall, not precision.
- Stage two: rerank those candidates with a cross-encoder and pass the top 3 to 8 into the prompt.
Teams that add this step usually see the biggest lift not from better answers, but from fewer wrong ones – the irrelevant chunk that used to sneak into the context window at rank 4 stops arriving. If your chunks themselves are messy, fix parsing first: our comparison of Docling, Unstructured and LlamaParse for RAG covers that stage.
Cohere Rerank: the managed default
Cohere Rerank is the lowest-friction option and the one most teams reach for first. It is a hosted API: you send a query and a list of documents, you get back scored indices. There is no model to serve, no GPU to size and no tokenizer to pin.
Rerank 3.5 has been the workhorse release, with Rerank 4 now in circulation. Independent latency comparisons put Rerank 3.5 near the front of the hosted pack at roughly 600ms average for a typical candidate batch, with consistent nDCG across general-purpose corpora and strong multilingual behaviour out of the box.
- Best for: teams shipping fast, mixed-language corpora, and anyone who does not want to own inference infrastructure.
- Watch out for: per-call pricing that scales with candidate count, and the fact that documents leave your network. Regulated workloads often stall here.
Jina Reranker v3: listwise scoring for long documents
Jina Reranker v3 takes a different architectural route. Instead of scoring each query-document pair in isolation, it is listwise: it processes a batch of candidates together – reported at up to 64 documents inside a 131k-token context window – so the model can compare candidates against each other rather than against an absolute threshold.
That design pays off on long, similar documents where pointwise scores bunch together. Jina reports 61.94 nDCG@10 on BEIR for v3, and the model is among the faster options in its accuracy band, with sub-200ms reranking achievable on suitable hardware. It is available both as an API and as open weights, which makes it one of the few models you can prototype hosted and later bring in-house without changing behaviour.
- Best for: long-document RAG, legal and research corpora, and pipelines where candidates are near-duplicates of each other.
- Watch out for: listwise scores are relative to the batch, so they are less meaningful as an absolute relevance cutoff.
BGE Reranker v2-m3: the open-weight workhorse
BGE Reranker v2-m3, from the BAAI FlagEmbedding project, is the reranker most self-hosted stacks end up running. It is small, multilingual, permissively licensed and fast enough to serve on a single mid-range GPU – or on CPU if your traffic is low and your candidate lists are short.
It will not top a leaderboard against the newest commercial models, but it is the practical baseline: predictable latency, zero per-query cost after hardware, no data egress, and it fine-tunes well on domain data. For a niche corpus – internal tickets, medical coding, a product catalogue – a fine-tuned BGE v2-m3 frequently beats a general-purpose hosted reranker that has never seen your vocabulary.
- Best for: on-premise deployments, data-residency constraints, high query volume, and domain fine-tuning.
- Watch out for: you now own GPU capacity planning, batching and model updates.
RAG reranker comparison at a glance
- Deployment: Cohere is API-only. Jina v3 offers API plus open weights. BGE v2-m3 is self-host only.
- Scoring style: Cohere and BGE are pointwise cross-encoders. Jina v3 is listwise.
- Long context: Jina v3 leads with a 131k-token window; Cohere and BGE work best on chunk-sized passages.
- Cost model: Cohere is per-call and grows with candidate count. Jina is either. BGE is fixed infrastructure cost.
- Data residency: only Jina self-hosted and BGE keep documents inside your perimeter.
- Fine-tuning: straightforward on BGE, possible on Jina open weights, not applicable to Cohere.
How to choose a RAG reranker for your stack
Start from your latency budget
Decide how many milliseconds the reranking step is allowed to consume before you compare models. A chat interface with streaming tolerates 300 to 600ms far better than a synchronous API with a 1-second SLA. Reranking latency scales with candidate count, so reranking 100 candidates is not twice the cost of 50 in user-perceived terms – it is often the difference between shipping and not.
Let licensing decide before accuracy does
If documents cannot leave your network, the shortlist is open weights and the accuracy debate is over before it starts. Settle this constraint first rather than benchmarking models you will never be allowed to deploy.
Tune top-k, not just the model
Retrieve wide and pass narrow. Most pipelines improve more from retrieving 100 candidates and passing 5 than from swapping one reranker for another at a fixed top-20. Sweep the retrieval depth and the final k together, and measure the pair.
Benchmark on your own queries
Public leaderboards rank models on public corpora. Build a 100-query golden set from real user questions, label the correct chunks once, and measure nDCG@5 and recall@5 per model. A harness such as the ones covered in our LLM evaluation tooling comparison makes this repeatable, and LLM observability tooling will tell you whether the lift holds in production.
Frequently asked questions
Do I still need a reranker with long-context models?
Usually yes. A large context window lets you pass more chunks, but it does not make irrelevant chunks harmless – they add cost, latency and distraction. Reranking lets you pass fewer, better passages, which is generally both cheaper and more accurate than stuffing the window.
How many candidates should I rerank?
Fifty is a sensible starting point for most corpora, rising to 100 when recall at stage one is weak. Beyond that, latency climbs faster than quality. Measure recall of your first-stage retriever at each depth to find where the useful documents stop appearing.
Is a RAG reranker worth it for small document sets?
Under a few hundred chunks, often not – retrieval is rarely the bottleneck at that scale. The value grows with corpus size and with how similar your documents are to each other. Near-duplicate policy documents or product variants benefit disproportionately.
Can I run a reranker on CPU?
BGE Reranker v2-m3 will run on CPU, and for low traffic with short candidate lists it is viable. Expect latency in the seconds rather than milliseconds once you pass a few dozen candidates, so treat CPU inference as a development convenience rather than a production plan.
Which RAG reranker should you ship?
There is no single winner, but the decision is not difficult. Choose Cohere Rerank if you want quality without operating anything and your data can leave your network. Choose Jina Reranker v3 if your documents are long or highly similar, or if you want one model that works hosted now and self-hosted later. Choose BGE Reranker v2-m3 if you are self-hosting anyway, your volume is high, or you have domain data worth fine-tuning on.
Whichever you pick, the single highest-value action is the measurement, not the model. Build the golden set, wire up two rerankers behind a flag, and let your own queries decide. Start with the open-weight BGE FlagEmbedding models or the Jina Reranker documentation, run the comparison this week, and ship the one that wins on your data.
