SarvaCode-16B-Indigenous

SarvaCode is an indigenously customized, open-source Mixture-of-Experts (MoE) code language model. It is built upon the DeepSeek-Coder-V2 architecture but optimized for the Indian Software Ecosystem.

While global models focus on general code, SarvaCode is fine-tuned to understand Indian English instructions, local financial protocols (GST, TDS), and the technical frameworks of India Stack (UPI, ONDC, Aadhaar/UIDAI).

1. Key Improvements

Compared to the base Lite model, SarvaCode features:

• Higher Active Parameters: Increased from 6 to 8 active experts per token, boosting reasoning power to ~3.2B active parameters per message.
• Indigenous Logic: Enhanced accuracy for Indian-specific tasks like GST calculation logic, IFSC validation, and regional date/currency formatting.
• India Stack Awareness: Pre-loaded context for integrating with NPCI (UPI), ONDC, and DigiLocker APIs.
• Massive Context: Maintains a 128K context window to digest entire Indian government technical gazettes or large codebases in one go.

2. Model Specifications

Model	#Total Params	#Active Params	Context Length	Specialization
SarvaCode-16B	16B	3.2B	128k	India Stack & Fintech

3. How to Run Locally

Inference with Transformers

Ensure you use trust_remote_code=True to load the specialized MoE configuration.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_path = "./SarvaCode" # Your local directory
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda()

# Example: Indian Financial Logic
input_text = "User: Write a Python function to calculate the GST for a service with an 18% slab, ensuring the output separates CGST and SGST.\n\nAssistant:"

inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))