README.md

---
language:
- en
license: apache-2.0
library_name: peft
tags:
- forecasting
- prediction
- reinforcement-learning
- grpo
- lora
- mixture-of-experts
- politics
- trump
- future-as-label
datasets:
- LightningRodLabs/WWTD-2025
base_model: openai/gpt-oss-120b
pipeline_tag: text-generation
model-index:
- name: Trump-Forecaster
  results:
  - task:
      type: text-generation
      name: Probabilistic Forecasting
    dataset:
      name: WWTD-2025
      type: LightningRodLabs/WWTD-2025
      split: test
    metrics:
    - type: brier_score
      value: 0.194
      name: Brier Score
    - type: ece
      value: 0.079
      name: Expected Calibration Error
---

# Trump-Forecaster

### RL-Tuned gpt-oss-120b for Predicting Trump Administration Actions

Starting from nothing but 5 search queries, we used the [Lightning Rod SDK](https://github.com/lightning-rod-labs/lightningrod-python-sdk) to automatically generate [2,108 forecasting questions](https://huggingface.co/datasets/LightningRodLabs/WWTD-2025) from news articles, label them using real outcomes, and train this model via RL. **No expertise required. No manual labeling. No domain-specific engineering.** The result beats GPT-5 on held-out questions.

You can do this in any domain — just change the search queries. See [how we built the dataset](https://huggingface.co/datasets/LightningRodLabs/WWTD-2025).

This repo contains a **LoRA adapter** for [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b). A standalone `merge.py` script is included to merge it into a full model.

---

## Results

Evaluated on 682 held-out test questions under two conditions: with news context, and without context (question only). The no-context condition reveals whether the model knows what it doesn't know—untrained models project false confidence, while RL training fixes overconfidence.

| Model | Brier (With Context) | BSS | Brier (No Context) | BSS | ECE (With Context) | ECE (No Context) |
|-------|:---:|:---:|:---:|:---:|:---:|:---:|
| GPT-5 | 0.200 | +0.14 | 0.258 | -0.11 | 0.091 | 0.191 |
| gpt-oss-120b (base) | 0.213 | +0.08 | 0.260 | -0.12 | 0.111 | 0.190 |
| **Trump-Forecaster** | **0.194** | **+0.16** | **0.242** | **-0.04** | **0.079** | **0.164** |

![Brier Skill Score](https://huggingface.co/datasets/LightningRodLabs/WWTD-2025/resolve/main/brier_skill_score.png)

![Brier Score Comparison](https://huggingface.co/datasets/LightningRodLabs/WWTD-2025/resolve/main/brier_score_comparison.png)

![ECE Comparison](https://huggingface.co/datasets/LightningRodLabs/WWTD-2025/resolve/main/ece_comparison.png)

### Metrics

- **Brier Score**: Mean squared error between predicted probability and outcome (0 or 1). Lower is better. **Brier Skill Score (BSS)** expresses this as improvement over always predicting the base rate—positive means the model learned something useful beyond historical frequency.
- **Expected Calibration Error (ECE)**: Measures whether predicted probabilities match actual frequencies. "70%" predictions should resolve "yes" 70% of the time. Lower is better.

---

## Training

- **Base model**: [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) (120B MoE, 5.1B active params, 128 experts Top-4)
- **Method**: GRPO with Brier score reward via [Tinker](https://tinker.computer)
- **LoRA rank**: 32
- **Learning rate**: 4e-5
- **Batch size**: 32, group size 8
- **Training steps**: 50
- **Max tokens**: 16,384

---

## Usage

This repo contains a LoRA adapter trained with [Tinker](https://tinker.computer). The adapter uses Tinker's module naming convention, so it requires a merge step before inference. A standalone `merge.py` script is included.

### Merge into full model

```bash
pip install torch transformers safetensors tqdm huggingface-hub
python merge.py --output ./trump-forecaster-merged
```

This downloads the base model, dequantizes to bf16, applies the LoRA adapter, and saves the merged model.

### Inference

```python
import sglang as sgl

engine = sgl.Engine(
    model_path="./trump-forecaster-merged",
    tokenizer_path="openai/gpt-oss-120b",
    trust_remote_code=True,
    dtype="bfloat16",
    tp_size=2,
)

news_context = "... relevant news articles ..."

prompt = f"""You are a forecasting expert. Given the question and context below, predict the probability that the answer is "Yes".

Question: Will Trump impose 25% tariffs on all goods from Canada by February 1, 2025?

Context:
{news_context}

Respond with your reasoning, then give your final answer as a probability between 0 and 1 inside <answer></answer> tags."""

output = engine.generate(prompt, sampling_params={"max_new_tokens": 4096, "stop": ["</answer>"]})
print(output["text"])
```

---

## Links

- **Dataset**: [LightningRodLabs/WWTD-2025](https://huggingface.co/datasets/LightningRodLabs/WWTD-2025)
- **Training platform**: [Tinker](https://tinker.computer)
- **Data generation**: [Lightning Rod SDK](https://github.com/lightning-rod-labs/lightningrod-python-sdk)
- **Future-as-Label paper**: [arxiv:2601.06336](https://arxiv.org/abs/2601.06336)
- **Outcome-based RL paper**: [arxiv:2505.17989](https://arxiv.org/abs/2505.17989)