SequentialLearning
/

SuperLinear

mixture-of-experts

Model card Files Files and versions

razmars commited on Apr 27

Commit

ef06da2

·

verified ·

1 Parent(s): 8346c19

Update config.json

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -26,7 +26,7 @@
   "mlp_gating": 1,
   "model_type": "super_linear",
   "moe": 1,
-  "moe_n_experts": 3,
   "moe_temp": 1,
   "noisy_gating_std": 0.1,
   "noisy_gating_std_decay": 1,

   "mlp_gating": 1,
   "model_type": "super_linear",
   "moe": 1,
+  "moe_n_experts": 8,
   "moe_temp": 1,
   "noisy_gating_std": 0.1,
   "noisy_gating_std_decay": 1,