Introducing LightOn-rerank: SOTA multimodal LLM reranker

Retrieval tends to over-fetch by design. Returning too many documents is the safest way to avoid missing the right one. While a human searching can skim past the extra, and an agent can ignore it, having irrelevant content pollutes its context and wastes extra tokens.

The reranker is the cut between over-fetching and acting. It decides what the agent actually sees. Get it right and the agent works from signal. Get it wrong and no amount of prompt engineering downstream recovers it.

Then comes the part no stack handles cleanly. Enterprise documents are not one shape. A single search touches plain text, a scanned contract, a slide, a web page, an invoice that is really a photo of a table. The standard fix is two rerankers, one for text and one for images, plus a rule to stitch their scores together afterward. Two models to deploy, two latency budgets to watch, and a stitched score that ranks a paragraph against a scanned page by fiat instead of by judgment.

LightOn-rerank removes the seam. One 2-billion-parameter model reads text passages and scanned pages on the same relevance scale, from a single adapter and a single deployment. It leads every open multimodal reranker in its class on ViDoRe V3, the reference benchmark for visual document search. Its 4B variant edges past Qwen's own 8B reranker at half the size. The leaderboard is the easy part. The hard part was trying to bring a listwise multimodal reranker’s cost down.

How the signal ratio actually improves

Three decisions made that possible.

One model, one score. Two specialist rerankers produce two score scales that have to be merged after the fact. That merge is where signal degrades: a text passage and a scanned page get ranked against each other by a rule rather than by the model itself. Training one model on both modalities removes the step. Text and images land on the same relevance scale because the same weights produced them. Training jointly also improved quality on each modality taken alone.

The comparison is the ranking. The common shortcut in reranking is to use a pointwise setup: score each candidate on its own, then sort. It parallelizes well and it is tempting. LightOn measured what it costs on visual documents: scoring candidates independently gave up roughly three quarters of the rerank gain on ViDoRe V3. The signal lives in the comparison between documents. LightOn-rerank reads a window of candidates together in one pass, so each one is judged against the others.

Serving cost engineered down, signal kept. LightOn profiled where the compute time goes. At the operating point that ships, encoding the page images costs about as much as running the model on them. Two levers followed. Bringing image resolution back to the point the model was trained for cut encoding time roughly eightfold, for a rounding-error move in quality. Reranking the top candidates instead of the full pool cut serving time twelvefold for under 3% quality loss, since the documents dropped were the ones that were never going to reach the top.The rule LightOn settled on: shrink the candidate pool, as our first-stage retriever is strong enough. The signal filter stays intact while the compute bill falls.

English training, French results. The model saw English data only. On ViDoRe V3 it ranks French documents as well as English ones, and on three domains it ranks French better. The multilingual backbone carries the rerank signal into French with no French data in training.

In LightOn Console

For document search pipelines, this is one reranker to connect. It handles passages from text reports and pages from scanned contracts or invoices on a single relevance scale, at a serving cost sized for production. For an agent, the result is a cleaner context: fewer irrelevant documents in the window, more of the token budget spent on signal.

Go deeper

The full mechanics, why LightOn chose a listwise recipe over pointwise, why the usual text-reranking speedups fail on visual documents, and how serving cost can be kept manageable, are in the technical post:
One Adapter, Both Modalities: Field Notes from Building and Serving a Multimodal Reranker.