diabase-reranker-1

Research preview. Training and corrected v2 bootstrap evaluation are complete. Independent Swedish MTEB evaluation is mixed: the reranker degrades SweFAQ and does not provide a consistent improvement on SweDN. It is published for transparent research and evaluation, not as the recommended production reranker.

A Swedish-first European cross-encoder for precise document reranking.

diabase-reranker-1 scores a query and document together. It is designed to rerank a small candidate set retrieved by diabase-embedding-1, improving precision where dense embedding similarity alone cannot distinguish subtle relevance differences.

The model shares the Swedish-adapted EuroBERT-210m foundation used by diabase-embedding-1, but it has separate weights and a separate relevance objective.

Status	Research preview; not recommended for production
Architecture	EuroBERT cross-encoder + scalar relevance head
Parameters	Approximately 210M
Primary language	Swedish
Secondary language	English replay is retained during adaptation
Maximum pair length	512 tokens
Input	Query/document pair
Output	Relative relevance score
Recommended candidate count	Rerank approximately 20–50 retrieved documents
Companion retriever	Diabase/embedding-1

Quick start

from sentence_transformers import CrossEncoder

model = CrossEncoder("Diabase/reranker-1")

query = "Vilka krav gäller för leverantörer av högrisksystem?"
documents = [
    "Leverantörer av högrisksystem måste arbeta med riskhantering, dokumentation, "
    "datakvalitet och mänsklig tillsyn.",
    "Stockholm är Sveriges huvudstad.",
]

pairs = [(query, document) for document in documents]
scores = model.predict(pairs)

ranked = sorted(zip(scores, documents), reverse=True)
for score, document in ranked:
    print(float(score), document)

Scores are intended for ranking documents for the same query. Do not assume scores are calibrated probabilities or directly comparable across unrelated queries.

Intended use

• Reranking candidates from diabase-embedding-1
• Improving precision in Swedish RAG systems
• Distinguishing closely related legal, technical and enterprise passages
• Selecting a small grounded context set before text generation
• Local and on-prem deployments where query/document data must remain private

Out of scope

• Searching an entire large corpus directly
• Treating relevance as factual correctness, authority or legal validity
• Using scores as calibrated probabilities
• Fully autonomous high-stakes decisions
• Generating answers or summaries
• Replacing access control or privacy enforcement

Why a separate reranker?

An embedding model encodes queries and documents independently, which makes retrieval fast enough for large indexes. A cross-encoder reads each query/document pair jointly, allowing richer token-level interaction but making it too expensive for exhaustive search.

The intended system is:

large document corpus
  -> diabase-embedding-1 retrieves top 20–50
  -> diabase-reranker-1 reorders those candidates
  -> top 3–5 documents are passed to the generator

The reranker is optional. It will only be released if measured precision gains justify its additional latency and operational complexity.

Model design

1. Start from the Swedish-adapted EuroBERT-210m encoder created for the Diabase retrieval stack.
2. Add a scalar sequence-classification/relevance head.
3. Train on query/positive pairs and difficult negatives.
4. Optimize ordering quality, not text generation or embedding geometry.

Unlike diabase-embedding-1, the reranker does not produce reusable document vectors. Every query/document pair requires a forward pass.

Training data

Training examples contain:

• A Swedish query
• One verified relevant document
• Multiple hard negatives retrieved by diabase-embedding-1
• Optional graded relevance labels where review supports them

Hard negatives are documents that appear semantically plausible but do not answer the query. Random unrelated negatives are insufficient because they make the task too easy and do not teach the model the distinctions needed in production.

False-negative control

Retrieved negatives may actually be relevant even when they are not the originally labelled positive passage. Before training:

• Remove exact and near-duplicate positives
• Check whether a candidate contains the gold answer
• Apply heuristic filtering
• Human-review a representative sample and high-impact ambiguous cases

Data provenance

The initial positives come from project-generated Swedish extractive-QA data. Candidate documents and hard negatives come from the authorized retrieval corpus. The corrected v2 pipeline groups semantic/topic families before splitting and excludes candidates from the query's own group, known positives, answer-containing passages and near duplicates.

The final training set contains 2,702 queries and 13,510 labelled pairs: one positive and four mined negatives per retained query. The development set contains 516 queries and 2,580 labelled pairs. Queries without four safe negatives in the top search window were skipped rather than filled with questionable labels.

Evaluation

The evaluation compares:

1. Ordering returned directly by diabase-embedding-1
2. The same candidate lists after diabase-reranker-1

The held-out v2 test contains 455 queries and 455 documents across 32 test topic groups. The reranker reorders the top 20 candidates from the v2 embedding model.

Metric	Embedding order	Reranked	Absolute change
Recall@1	49.67%	51.43%	+1.76 points
Recall@5	77.36%	79.56%	+2.20 points
Recall@10	87.03%	89.23%	+2.20 points
Recall@20	95.38%	95.38%	unchanged
MRR@10	61.63%	63.50%	+1.88 points
nDCG@10	67.71%	69.68%	+1.97 points
Median relevant-document rank	2	1	-1
Mean relevant-document rank	4.26	3.88	-0.38

On the development RTX 5070, scoring 9,100 pairs took 17.8 seconds: approximately 512 pairs/second or 39 ms per query when reranking 20 candidates. This is one local throughput measurement, not a general latency guarantee.

The test queries and documents must be separate from embedding training, hard-negative mining supervision and reranker training.

Independent Swedish MTEB reranking

The cross-encoder reranks the top 20 candidates from diabase-embedding-1. Candidate Recall@20 is therefore fixed. The evaluation uses the same MTEB dataset revisions and 512-token limit as the embedding evaluation.

Task and metric	Embedding order	Reranked	Change
SweFAQ Recall@1	30.21%	22.22%	-7.99 points
SweFAQ MRR@10	41.15%	35.54%	-5.60 points
SweFAQ nDCG@10	47.16%	42.86%	-4.30 points
SweDN Recall@1	16.16%	13.77%	-2.39 points
SweDN MRR@10	39.99%	38.43%	-1.55 points
SweDN nDCG@10	34.02%	34.18%	+0.15 points

This fails the stable release gate. The model should not replace embedding order in a production pipeline without domain-specific evaluation and additional training data.

Local latency

For 20 candidates per query:

• CPU: 1,455 ms mean, 1,713 ms p95 and 13.7 pairs/second.
• NVIDIA RTX 5070: 35.5 ms mean, 41.0 ms p95 and 563.3 pairs/second.
• Combined embedding/reranker benchmark peak GPU allocation: approximately 1.67 GB.

Measured with PyTorch 2.11.0 over 20 timed iterations. These are local measurements, not service-level guarantees.

Release gate

The corrected bootstrap passes the internal quality gate: top-1 relevance, nDCG@10 and MRR@10 improve while candidate-set Recall@20 is preserved. A stable release still requires that it:

• Improves nDCG@10 and top-1 relevance over embedding order by a practically meaningful margin
• Does not remove relevant documents from the final context set too often
• Meets the documented latency target on local hardware
• Shows gains on a separate human-reviewed Swedish benchmark

If those conditions are not met, Diabase will recommend using diabase-embedding-1 without a reranker.

Limitations

• Cross-encoder inference is substantially slower than vector similarity.
• The model is Swedish-first; other-language ranking quality may be lower.
• Pair truncation can remove decisive evidence from long documents.
• Hard-negative quality strongly controls model quality.
• A relevance model can prefer persuasive or lexically similar misinformation.
• Scores may shift across domains and should not be treated as globally calibrated.
• Reranking cannot recover a relevant document that the embedding model never retrieved.

Security and privacy

Local inference keeps query/document text under the deployer's control, but callers must still enforce authorization before scoring documents. Reranking does not neutralize prompt injection, malicious documents, poisoned corpora or confidential-data leakage.

Relationship to other Diabase models

• diabase-embedding-1: fast first-stage candidate retrieval
• diabase-reranker-1: precise second-stage candidate ordering
• Diabase Europa: grounded response generation from the selected context

The models are separate artifacts because they optimize different objectives and have different latency profiles.

License and attribution

The Diabase derivative is released under Apache-2.0, consistent with the EuroBERT foundation. Release weight SHA-256: 36ab2529642ed5e3f1ef857015575480921f8c3eb469764b271e9b87b32737f5.

EuroBERT citation:

@misc{boizard2025eurobertscalingmultilingualencoders,
  title={EuroBERT: Scaling Multilingual Encoders for European Languages},
  author={Nicolas Boizard and Hippolyte Gisserot-Boukhlef and Duarte M. Alves
    and André Martins and Ayoub Hammal and Caio Corro and Céline Hudelot
    and Emmanuel Malherbe and Etienne Malaboeuf and Fanny Jourdan
    and Gabriel Hautreux and João Alves and Kevin El-Haddad and Manuel Faysse
    and Maxime Peyrard and Nuno M. Guerreiro and Patrick Fernandes
    and Ricardo Rei and Pierre Colombo},
  year={2025},
  eprint={2503.05500},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2503.05500}
}

Maintainer

Built by Diabase.