---
license: mit
datasets:
- takara-ai/micropajama
language:
- en
---
<img src="https://takara.ai/images/logo-24/TakaraAi.svg" width="200" alt="Takara.ai Logo" />
From the Frontier Research Team at takara.ai we present a linear projection model that maps Qwen embeddings to RWKV embeddings for enhanced cross-model compatibility.

## Model Details

- **Input Dimensions**: 4096 (Qwen embeddings)
- **Output Dimensions**: 768 (RWKV embeddings)
- **Architecture**: Linear layer (no bias)
- **Training**: Cosine similarity loss on L2-normalized pairs
- **Dataset**: takara-ai/micropajama_embedded_concat

## Usage

### Quick Start
```python
import torch
from huggingface_hub import PyTorchModelHubMixin

# Define the model class (copy this exactly)
class QwenRwkvProjection(torch.nn.Module, PyTorchModelHubMixin,
                        library_name="takara-ai",
                        tags=["embedding", "projection", "qwen", "rwkv"],
                        license="mit"):
    def __init__(self, din=4096, dout=768):
        super().__init__()
        self.linear = torch.nn.Linear(din, dout, bias=False)

    def forward(self, x):
        return self.linear(x)

# Load from Hub
model = QwenRwkvProjection.from_pretrained("takara-ai/qwen_rwkv_projection")
model.eval()

# Project embeddings (don't forget to normalize!)
normalized_qwen_embeddings = torch.nn.functional.normalize(your_qwen_embeddings, p=2, dim=-1, eps=1e-8)
projected_embeddings = model(normalized_qwen_embeddings)
```

### Important Notes
- **Dimensions**: Input must be (batch_size, 4096), output will be (batch_size, 768)
- **Bias**: Model uses no bias term (trained on normalized pairs)