comsat-embed-ja-8b-preview

comsat-embed-ja-8b-preview is a decoder-based embedding model developed by Sionic AI, optimized for Japanese semantic retrieval tasks. Trained on over 1.5M Japanese examples, it encodes queries and documents into vectors so that the most relevant documents can be found by similarity. The model is designed to provide high-quality text representations for real-world information retrieval scenarios, including document search, question answering, knowledge base retrieval, and enterprise semantic search. By leveraging Japanese retrieval-oriented training data, comsat-embed-ja-8b-preview delivers robust performance across Japanese search environments where accurate semantic matching is essential.

Highlights

• Japanese-specialized — trained on 1.5M+ Japanese examples and tuned for Japanese search; achieves state-of-the-art average NDCG@10 (0.8133) on the 11-task JMTEB(v2) retrieval benchmark among the compared models.
• Long context — handles inputs up to 8,192 tokens, well suited to long-document retrieval.
• Instruction-aware queries — queries are encoded with a task-instruction prompt to improve retrieval quality; documents need no prefix.
• High-dimensional embeddings — 4096-dimensional, last-token pooled and L2-normalized, compared with cosine similarity.

Usage

First install the Sentence Transformers library

pip install -U sentence-transformers

Sentence Transformers Usage

⚠️ Queries must be encoded with the query prompt; documents are encoded without any prefix. (Skipping the query prompt slightly degrades retrieval quality.)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sionic-ai/comsat-embed-ja-8b-preview")

queries  = ["日本の首都はどこですか？"]
passages = ["日本の首都は東京都です。"]

# Option 1) pass the query prompt explicitly (query only; documents get no prefix)
q_emb = model.encode(queries,  prompt_name="query", normalize_embeddings=True)
d_emb = model.encode(passages,                      normalize_embeddings=True)

# Option 2) sentence-transformers 5.x helper API (equivalent result)
# q_emb = model.encode_query(queries)
# d_emb = model.encode_document(passages)

scores = q_emb @ d_emb.T   # cosine similarity
print(scores)

Transformers Usage

# Requires transformers>=4.51.0

import torch
import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def last_token_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
    if left_padding:
        return last_hidden_states[:, -1]
    else:
        sequence_lengths = attention_mask.sum(dim=1) - 1
        batch_size = last_hidden_states.shape[0]
        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]


def get_detailed_instruct(task_description: str, query: str) -> str:
    return f'Instruct: {task_description}\nQuery:{query}'

# Each query must come with a one-sentence instruction that describes the task
task = 'Given a web search query, retrieve relevant passages that answer the query'

queries = [
    get_detailed_instruct(task, '日本の首都はどこですか？'),
    get_detailed_instruct(task, '光合成はどのように起こりますか？')
]
# No need to add instruction for retrieval documents
documents = [
    "日本の首都は東京都です。",
    "光合成は、植物が光エネルギーを利用して二酸化炭素と水からブドウ糖を合成する過程です。"
]
input_texts = queries + documents

tokenizer = AutoTokenizer.from_pretrained('sionic-ai/comsat-embed-ja-8b-preview', padding_side='left')
model = AutoModel.from_pretrained('sionic-ai/comsat-embed-ja-8b-preview')

# We recommend enabling flash_attention_2 for better acceleration and memory saving.
# model = AutoModel.from_pretrained('sionic-ai/comsat-embed-ja-8b-preview', attn_implementation="flash_attention_2", torch_dtype=torch.bfloat16).cuda()

max_length = 8192

# Tokenize the input texts
batch_dict = tokenizer(
    input_texts,
    padding=True,
    truncation=True,
    max_length=max_length,
    return_tensors="pt",
)
batch_dict.to(model.device)
outputs = model(**batch_dict)
embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T)
print(scores.tolist())

Japanese Retrieval Benchmark (JMTEB v2)

• NLPJournalTitleAbsRetrieval.V2: Japanese academic paper retrieval — retrieve the abstract from the paper title.
• NLPJournalTitleIntroRetrieval.V2: Japanese academic paper retrieval — retrieve the introduction from the title.
• NLPJournalAbsIntroRetrieval.V2: Japanese academic paper retrieval — retrieve the introduction from the abstract.
• NLPJournalAbsArticleRetrieval.V2: Japanese academic paper retrieval — retrieve the article body from the abstract.
• MintakaRetrieval: A multilingual open-domain QA retrieval dataset (Japanese subset).
• JaGovFaqsRetrieval: A Japanese government FAQ retrieval dataset.
• JaqketRetrieval: A Japanese open-domain quiz QA retrieval dataset.
• MultiLongDocRetrieval: A long-document retrieval dataset (Japanese subset).
• JaCWIRRetrieval: A Japanese casual web information retrieval dataset.
• MIRACLRetrieval: A Wikipedia-based retrieval dataset (Japanese subset).
• MrTidyRetrieval: A Wikipedia-based Japanese retrieval dataset.

Performance (JMTEB v2 Retrieval, NDCG@10)

All scores are NDCG@10, measured with the standard MTEB/JMTEB retrieval pipeline. For multilingual tasks the Japanese subset is used (Mintaka/MultiLongDoc/MIRACL=ja, MrTidy=japanese).

Model	Avg	NLPJ-TitleAbs	NLPJ-TitleIntro	NLPJ-AbsIntro	NLPJ-AbsArticle	Mintaka	JaGovFaqs	Jaqket	MultiLongDoc	JaCWIR	MIRACL	MrTidy
comsat-embed-ja-8b-preview	0.8133	0.9779	0.9781	0.9922	0.9973	0.6087	0.7634	0.7809	0.5655	0.8948	0.7479	0.6400
codefuse-ai/F2LLM-v2-14B	0.7965	0.9782	0.9823	0.9938	0.9966	0.5894	0.8244	0.7471	0.4854	0.8142	0.6941	0.6565
Qwen/Qwen3-Embedding-8B	0.7924	0.9649	0.9517	0.9900	0.9973	0.6023	0.7313	0.6642	0.5649	0.8590	0.7388	0.6524
codefuse-ai/F2LLM-v2-8B	0.7855	0.9808	0.9882	0.9932	0.9966	0.5438	0.8149	0.7050	0.4824	0.8121	0.6745	0.6485
Qwen/Qwen3-Embedding-4B	0.7779	0.9753	0.9589	0.9881	0.9959	0.5201	0.7179	0.6136	0.5659	0.8560	0.7244	0.6406
codefuse-ai/F2LLM-v2-4B	0.7705	0.9853	0.9803	0.9937	0.9966	0.4767	0.8064	0.6528	0.4701	0.8166	0.6527	0.6442
Qwen/Qwen3-VL-Embedding-8B	0.7702	0.9764	0.9648	0.9896	0.9973	0.4995	0.7051	0.6830	0.4936	0.8491	0.6934	0.6201
sbintuitions/sarashina-embedding-v2-1b	0.7659	0.9804	0.9782	0.9954	0.9858	0.4365	0.7561	0.7371	0.4529	0.8552	0.6552	0.5916
cl-nagoya/ruri-v3-130m	0.7641	0.9807	0.9643	0.9894	0.9959	0.3283	0.7729	0.7514	0.4565	0.8349	0.7157	0.6149
cl-nagoya/ruri-v3-310m	0.7630	0.9785	0.9653	0.9908	0.9959	0.3353	0.7726	0.7342	0.4393	0.8405	0.7233	0.6168

Avg is the mean over the 11 JMTEB(v2) retrieval tasks (higher is better). Reproduction: evaluated with the MTEB/JMTEB retrieval pipeline (NDCG@10, full corpus); the query prompt is applied to queries only (documents get no prefix).

License

• Model weights: cc-by-nc-4.0 (non-commercial use).