takara-ai/qwen_rwkv_projection

From the Frontier Research Team at takara.ai we present a linear projection model that maps Qwen embeddings to RWKV embeddings for enhanced cross-model compatibility.

Model Details

• Input Dimensions: 4096 (Qwen embeddings)
• Output Dimensions: 768 (RWKV embeddings)
• Architecture: Linear layer (no bias)
• Training: Cosine similarity loss on L2-normalized pairs
• Dataset: takara-ai/micropajama_embedded_concat

Usage

Quick Start

import torch
from huggingface_hub import PyTorchModelHubMixin

# Define the model class (copy this exactly)
class QwenRwkvProjection(torch.nn.Module, PyTorchModelHubMixin,
                        library_name="takara-ai",
                        tags=["embedding", "projection", "qwen", "rwkv"],
                        license="mit"):
    def __init__(self, din=4096, dout=768):
        super().__init__()
        self.linear = torch.nn.Linear(din, dout, bias=False)

    def forward(self, x):
        return self.linear(x)

# Load from Hub
model = QwenRwkvProjection.from_pretrained("takara-ai/qwen_rwkv_projection")
model.eval()

# Project embeddings (don't forget to normalize!)
normalized_qwen_embeddings = torch.nn.functional.normalize(your_qwen_embeddings, p=2, dim=-1, eps=1e-8)
projected_embeddings = model(normalized_qwen_embeddings)

Important Notes

• Dimensions: Input must be (batch_size, 4096), output will be (batch_size, 768)
• Bias: Model uses no bias term (trained on normalized pairs)