AtmicQuoterv3

Fine-tuned from BAAI/bge-small-en-v1.5 on a combined dataset: AtmicQuoterv1's training data (bracket-tag source prefixes stripped from positives/negatives before training) plus AtmicQuoterv3's training data (layer2 verbatim-matched citation/RAG_citation.json rows + root book/page-cited rows).

Benchmark: base vs. AtmicQuoterv3 (alone vs. hybrid BM25)

Corpus: 10,506 passages -- the union of AtmicQuoterv1's corpus (2,394 passages, bracket-tag prefixes stripped) and the full citation/RAG_citation.json corpus (8,273 passages).

Validation set: 1,006 queries -- the 48-query root validation.jsonl set plus AtmicQuoterv1's own 958-query validation split.

"alone" = dense retrieval only. "hybrid BM25" = AtmicQuoterv3 blended with BM25 (0.7 model / 0.3 BM25, min-max normalized per query), no reranker.

metric	base (bge-small) alone	AtmicQuoterv3 alone	AtmicQuoterv3 + hybrid BM25
accuracy@1	0.2396	0.6123	0.6262
recall@3	0.4254	0.7694	0.7962
recall@5	0.4901	0.8211	0.8479
recall@10	0.5577	0.8757	0.8897
mrr@3	0.3254	0.6817	0.7021
mrr@10	0.3495	0.7008	0.7200
ndcg@3	0.3512	0.7043	0.7263
ndcg@10	0.4000	0.7432	0.7616
map@100	0.3579	0.7043	0.7237

SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'Represent this sentence for searching relevant passages: What does Siva advise about chakra and nadi meditation in the Sri Devikalottara?',
]
documents = [
    'Verse 15: [verse text not in source; commentary summary only] | Commentary: Siva advises discarding complex meditation on chakras, nadis, and specific deities in favour of the direct path of the Self. | Topics: siva, parvati, jnana, mind, liberation',
    'Verse 27: The mind... is more restless than a monkey. If one... holds it in the void of non-matter one will attain liberation directly. | Topics: siva, parvati, jnana, mind, liberation',
    'Does Siva’s advent purify / The mind or does He come because / The mind has become pure?” ask fools / Who sadly err in seeing two / Actions where there is but one | Topics: guru_vachaka_kovai',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 384] [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.6681, 0.2917, 0.2647]])

Evaluation

Metrics

Information Retrieval

• Dataset: atmic-val
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.6123
cosine_accuracy@3	0.7694
cosine_accuracy@5	0.8211
cosine_accuracy@10	0.8757
cosine_precision@1	0.6123
cosine_precision@3	0.2565
cosine_precision@5	0.1642
cosine_precision@10	0.0876
cosine_recall@1	0.6123
cosine_recall@3	0.7694
cosine_recall@5	0.8211
cosine_recall@10	0.8757
cosine_ndcg@3	0.7043
cosine_ndcg@10	0.7432
cosine_mrr@3	0.6817
cosine_mrr@10	0.7008
cosine_map@100	0.7043

Training Details

Training Dataset

Unnamed Dataset

• Size: 5,655 training samples
• Columns: anchor, positive, negative_1, negative_2, and negative_3
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative_1	negative_2	negative_3
  type	string	string	string	string	string
  modality	text	text	text	text	text
  details	min: 15 tokens mean: 25.15 tokens max: 41 tokens	min: 13 tokens mean: 37.98 tokens max: 65 tokens	min: 10 tokens mean: 38.24 tokens max: 87 tokens	min: 11 tokens mean: 40.62 tokens max: 87 tokens	min: 12 tokens mean: 37.74 tokens max: 84 tokens

• Samples:

  anchor	positive	negative_1	negative_2	negative_3
  Represent this sentence for searching relevant passages: What is Jnana according to Ramana Maharshi?	This knowledge that there is nothing but God or the Self... is Jnana | Topics: citation_sentences	Verse 30: All ego gone, Living as That alone Is penance good for growth, Sings Ramana, the Self. | Topics: ego-extinction, self-abidance, tapas, ramana	Bhagavan Sri Ramana Maharshi, born Venkataraman Iyer, was the second of four children born to a devout family in Tiruchuzhi, Tamil Nadu, India.	Everything happens according to each individual’s karma | Topics: citation_sentences
  Represent this sentence for searching relevant passages: What does it mean to have the knowledge that nothing exists except God?	This knowledge that there is nothing but God or the Self... is Jnana | Topics: citation_sentences	That silence could not be understood except by the very advanced | Topics: citation_sentences	What exists apart from yourself? You ought to understand that by the Self neither the physical body nor the subtle body is meant | Topics: citation_sentences	The flag of vairagya will never move. What greater sign does a Jnani require than that? | Topics: citation_sentences
  Represent this sentence for searching relevant passages: What did the Guru say a disciple should do to repay his Guru's grace?	When the disciple asked what he could do in return for his Guru’s grace, the Guru said that he should always be in meditation | Topics: citation_sentences	The service rendered to the Guru / By the chela called and chosen by him / Should be respectful and correct, / Like a woman’s to her husband / In company | Topics: guru_vachaka_kovai	The Guru’s grace... is beyond words or thoughts | Topics: citation_sentences	Why did the three-eyed Lord ordain / That Kannappa should pluck and plant / Those eyes of which he was so proud? / It was to save him from the death, / The deadly folly of deeming he / Was flesh inert. Such is the might / Of Siva’s grace, of love for Him | Topics: guru_vachaka_kovai

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• learning_rate: 2e-05
• warmup_steps: 0.1
• weight_decay: 0.01
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• per_device_train_batch_size: 16
• num_train_epochs: 3.0
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	atmic-val_cosine_ndcg@10
0.1412	50	2.3863	-
0.2825	100	1.9310	-
0.4237	150	1.7572	-
0.5650	200	1.6355	-
0.7062	250	1.5219	-
0.8475	300	1.3803	-
0.9887	350	1.2837	-
1.0	354	-	0.7261
1.1299	400	1.1659	-
1.2712	450	1.1991	-
1.4124	500	1.1150	-
1.5537	550	1.0593	-
1.6949	600	1.1075	-
1.8362	650	1.0859	-
1.9774	700	0.9842	-
2.0	708	-	0.7379
2.1186	750	0.9106	-
2.2599	800	0.9686	-
2.4011	850	0.8822	-
2.5424	900	0.8293	-
2.6836	950	0.8808	-
2.8249	1000	0.7850	-
2.9661	1050	0.8997	-
3.0	1062	-	0.7432
-1	-1	-	0.7432

• The bold row denotes the saved checkpoint.

Training Time

• Training: 5.5 hours
• Evaluation: 1.1 minutes
• Total: 5.6 hours

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 5.6.0
• Transformers: 5.12.1
• PyTorch: 2.12.1
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}