README.md

---
library_name: transformers
license: apache-2.0
language:
  - en
tags:
  - monoid
  - causal-lm
  - linear-attention
  - state-space
  - O(1)-inference
  - vector-decay
  - reasoning
pipeline_tag: text-generation
model-index:
  - name: Spartacus-1B-Instruct
    results: []
---

# Spartacus-1B-Instruct — Causal Monoid Language Model

A 1.3B parameter language model that replaces softmax attention with **causal monoid state compression**, achieving **O(1) time per token** and **O(1) memory** at inference — regardless of sequence length.

## Monoid Attention — Internal Structure

```
                          MonoidAttention (per layer, per head)
 ┌─────────────────────────────────────────────────────────────────────────┐
 │                                                                         │
 │   x_t ∈ R^{2048}                                                       │
 │    │                                                                    │
 │    ├──> q_proj ──> RMSNorm ──> q_t ∈ R^d          (query, scaled 1/√d) │
 │    │                                                                    │
 │    ├──> k_proj ──> RMSNorm ──> SiLU ──> k_t ∈ R^d (key, non-negative) │
 │    │                                                                    │
 │    ├──> v_proj ──> v_t ∈ R^d                       (value)             │
 │    │                                                                    │
 │    └──> decay_proj ──> -Softplus ──> log α_t ∈ R^d (vector decay gate) │
 │                                                                         │
 │         k_t ⊗ v_t                                                       │
 │            │             ┌─────────────────────────────────┐            │
 │            │             │  State Matrix S_t ∈ R^{d x d}   │            │
 │            v             │  "Compressed causal history"    │            │
 │    S_t = diag(α_t) · S_{t-1} + k_t ⊗ v_t                 │            │
 │            │             │  α_t ∈ (0,1]^d per dimension    │            │
 │            │             └─────────────────────────────────┘            │
 │            v                                                            │
 │    o_t = q_t · S_t ──> o_proj ──> output                               │
 │                                                                         │
 └─────────────────────────────────────────────────────────────────────────┘
```

## Key Properties

| Property | Transformer (Llama) | Spartacus (Monoid) |
|---|---|---|
| Inference time per token | O(T) — scans full KV-cache | **O(1)** — single state update |
| Inference memory per layer | O(T) — stores all past K,V | **O(1)** — fixed d×d state matrix |
| Sequence length extrapolation | Degrades beyond training length | **Unlimited** — state size is constant |
| Causality | Imposed via attention mask | **Built into the recurrence** |
| Training complexity | O(T²) | **O(T)** via parallel prefix scan |

## The Monoid Recurrence

Standard attention computes:

```
o_t = Σ_{i≤t} softmax(q_t · k_i) v_i    — requires O(T) KV-cache
```

Monoid attention compresses the entire causal history into a **fixed-size state matrix** S_t per head:

```
S_t = diag(α_t) · S_{t-1} + k_t ⊗ v_t     — vector decay monoid recurrence
o_t = q_t · S_t                              — state readout
```

This is a monoid because the binary operator `(log_α, S) ⊕ (log_β, X) = (log_α + log_β, exp(log_β)·S + X)` is **associative**, enabling O(T) parallel prefix scan for training and O(1) sequential update for inference.

## Vector Decay — Per-Dimension Memory Lifetimes

Unlike scalar decay (one α per head), Spartacus uses **vector decay**: each dimension of the d-vector has its own independent decay rate α_t[i] ∈ (0, 1]:

```
S_t[i,j] = α_t[i] · S_{t-1}[i,j] + k_t[i] · v_t[j]
```

This allows different feature dimensions to specialize:
- **Fast-decaying dimensions** (α ≈ 0) — local syntax, punctuation, function words
- **Slow-decaying dimensions** (α ≈ 1) — entity memory, topic tracking, long-range facts

The decay gate uses **Negative Softplus** activation:

```
log α_t = -softplus(W·x_t + b)
```

| Property | Value |
|---|---|
| Range | α ∈ (0, 1] — bounded, no explosion |
| Perfect memory | W·x → -∞ ⟹ softplus → 0 ⟹ α → 1 (lossless retention) |
| Full forgetting | W·x → +∞ ⟹ softplus → ∞ ⟹ α → 0 (complete reset) |
| Stability | α ≤ 1 by construction — no divergence regardless of input magnitude |

## Attention Mask — Padding-Aware Recurrence

The monoid recurrence correctly handles `attention_mask` for padded batches (e.g., left-padding during `generate()`). For PAD positions (mask=0):

```
log_α = 0    →  α = 1  (preserve state unchanged)
k = 0, v = 0 →  kv = 0 (no information injected)
```

Net effect: `S_t = 1·S_{t-1} + 0 = S_{t-1}` — PAD acts as the **monoid identity element**, completely invisible to the recurrence. This ensures identical outputs whether inputs are padded or not.

## Design Choices

- **SiLU-activated keys**: `k = SiLU(k_proj(x))` ensures non-negative keys, making the state matrix S positive semi-definite (PSD). This prevents "feature erasure" where one token's contribution cancels another's
- **QK-Norm**: RMSNorm on both q and k before readout, stabilizing the scale of q·S when the state matrix accumulates many outer products
- **Log-space decay**: Working in log-space `log(α)` avoids numerical underflow when α^T → 0 for long sequences
- **Learnable h0**: The initial state S₀ = h0 is a learnable parameter (zero-initialized), acting as a compressed "system prompt"
- **Negative Softplus gate**: Ensures α ∈ (0, 1] by construction — allows perfect memory (α=1) while preventing state explosion (α>1)

## Three Forward Paths

| Path | Condition | Complexity | Description |
|---|---|---|---|
| Training | `use_cache=False` | O(T) parallel scan | Vectorized outer products → parallel prefix scan → vectorized readout |
| Inference prefill | `use_cache=True, T>1` | O(T) parallel scan | Same as training + extracts final state S_T for cache |
| Inference decode | `use_cache=True, T=1` | **O(1)** monoid_op | Single `monoid_op` to fold new token into state → one matmul readout |

## Model Details

| Parameter | Value |
|---|---|
| Model | `NoesisLab/Spartacus-1B-Instruct` |
| Architecture | MonoidForCausalLM |
| Parameters | ~1.34B (tied embeddings) |
| Hidden size | 2048 |
| Intermediate size (MLP) | 8192 |
| Layers | 16 |
| Attention heads | 32 |
| Head dimension | 64 |
| Decay gate | Vector decay, d=64 per head |
| State matrix per head | 64 × 64 = 4,096 floats |
| Vocabulary | 128,256 (Llama-3.2 tokenizer) |
| Precision | bfloat16 |

## Benchmarks (0-shot)

| Task | Metric | Value | Stderr |
|---|---|---|---|
| ARC-Challenge | acc_norm | 0.3063 | ±0.0135 |
| ARC-Easy | acc | 0.5518 | ±0.0102 |
| HellaSwag | acc_norm | 0.4610 | ±0.0050 |
| PIQA | acc_norm | 0.6915 | ±0.0108 |
| WinoGrande | acc | 0.5225 | ±0.0140 |

### Comparison with ~1B Baselines (acc_norm, 0-shot)

| Task | Spartacus-1B | TinyLlama-1.1B | Llama 3.2-1B | Mamba-1.4B | RWKV-6-1.6B |
|---|---|---|---|---|---|
| ARC-C | **0.3063** | 0.3268 | ~0.359 | 0.284 | ~0.301 |
| ARC-E | **0.5518** | 0.5547 | ~0.752 | 0.512 | ~0.530 |
| HellaSwag | **0.4610** | 0.4670 | ~0.546 | 0.435 | ~0.450 |
| PIQA | **0.6915** | 0.7210 | ~0.740 | 0.655 | ~0.670 |
| WinoGrande | **0.5225** | 0.5040 | ~0.592 | 0.510 | ~0.515 |

> Spartacus achieves competitive performance with sub-quadratic models (Mamba, RWKV) while maintaining **O(1) inference time and memory per token**. Scores marked with ~ are approximate community-reported values.

## Parallel Scan Implementation

The `monoid_scan_cuda.py` module provides a Triton JIT-compiled parallel prefix scan for the vector-decay monoid:

- **Grid**: `(B*H*D_k, ceil(D_v/BLOCK_DV))` — one program per state matrix row
- **Forward**: Sequential scan along T per row, parallelized across all (batch, head, d_k) dimensions
- **Backward**: Reverse-order adjoint scan with per-row D_v reduction (minimal atomic_add)
- **Fallback**: Pure PyTorch sequential scan for CPU/MPS
- **Auto-dispatch**: CUDA → Triton kernel, otherwise → PyTorch fallback

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "NoesisLab/Spartacus-1B-Instruct",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("NoesisLab/Spartacus-1B-Instruct")

messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## File Structure

```
MonoidForCausalLM.py       # Model architecture (MonoidConfig, MonoidAttention, MonoidForCausalLM)
monoid_scan_cuda.py        # Triton JIT parallel prefix scan (vector decay) + PyTorch fallback
model.safetensors          # Model weights (bfloat16)
config.json                # Model configuration
tokenizer.json             # Llama-3.2 tokenizer
```

## Citation

```bibtex
@software{spartacus2025,
  title={Spartacus: Causal Monoid Language Model with O(1) Inference},
  author={NoesisLab},
  year={2025},
  url={https://huggingface.co/NoesisLab/Spartacus-1B-Instruct},
  description={Replaces softmax attention with vector-decay monoid state compression for constant-time, constant-memory autoregressive generation}
}
```

## License

Apache 2.0