Improving text and notes

app/src/content/article.mdx

@@ -45,19 +45,19 @@ import ggc_snowhite from './assets/image/golden_gate_claude_snowhite.jpeg'
 <Image src={ggc_snowhite} alt="One of the many examples of Golden Gate Claude conversations"
        caption='One of the many examples of Golden Gate Claude conversations <a target="_blank" href="https://x.com/JE_Colors1/status/1793747959831843233">Source</a>' />
 
-Since then, sparse autoencoders (SAEs) have become one of the key tools in the field of mechanistic interpretability [@cunningham2023sparse; @lieberum2024gemma; @gao2024scaling] and steering activations sparked the interest of many (see for instance [the value of steering](https://thezvi.substack.com/i/144959102/the-value-of-steering) by Zvi Mowshowitz, or the work by [GoodFire AI](https://www.goodfire.ai/blog/feature-steering-for-reliable-and-expressive-ai-engineering))
+Since then, sparse autoencoders (SAEs) have become one of the key tools in the field of mechanistic interpretability [@cunningham2023sparse; @lieberum2024gemma; @gao2024scaling] and steering activations sparked the interest of many. See for instance [the value of steering](https://thezvi.substack.com/i/144959102/the-value-of-steering) by Zvi Mowshowitz, or [Feature Steering for Reliable and Expressive AI Engineering](https://www.goodfire.ai/blog/feature-steering-for-reliable-and-expressive-ai-engineering) by GoodFire AI.
 
-However, as far as I know, **nobody has tried to reproduce something similar to the Golden Gate Claude demo!** Moreover, recently the AxBench paper [@wu2025axbench] found that steering with SAEs was *one of the least effective methods to steer a model toward a desired concept*. How can we reconcile this with the success of the Golden Gate Claude?
+However, as far as I know, **nobody has tried to reproduce something similar to the Golden Gate Claude demo.** Moreover, recently the AxBench paper [@wu2025axbench] found that steering with SAEs was *one of the least effective methods to steer a model toward a desired concept*. How can we reconcile this with the success of the Golden Gate Claude?
 
 The aim of this article is to investigate how SAEs can be used to reproduce **a demo similar to Golden Gate Claude, but using a lightweight open-source model**. For this we used *Llama 3.1 8B Instruct*, but since I live in Paris...let’s make it obsessed with the Eiffel Tower!
 
-By doing this, we will realize that steering a model with vectors coming from SAEs is actually harder than we might have thought. However, we will devise several improvements over naive steering. While we focus on a single, concrete example — the Eiffel Tower — our goal is to establish a methodology for systematically evaluating and optimizing SAE steering, which could then be applied to other models and concepts.
+By doing this, we will realize that steering a model with activation vectors extracted from SAEs is actually harder than we might have thought. However, we will devise several improvements over naive steering. While we focus on a single, concrete example — the Eiffel Tower — our goal is to establish a methodology for systematically evaluating and optimizing SAE steering, which could then be applied to other models and concepts.
 
 **Our main findings:**
 <Note title="" variant="success">
-    - **The steering 'sweet spot' is smaller than you think.** The optimal steering strength is roughly half the magnitude of a layer's typical activation. This is consistent with the idea that steering vectors should not overwhelm the model's natural activations.
+    - **The steering 'sweet spot' is small.** The optimal steering strength is of the order of half the magnitude of a layer's typical activation. This is consistent with the idea that steering vectors should not overwhelm the model's natural activations.
     - **Clamping is more effective than adding.** We found that clamping activations (capping them at a maximum value) improves concept inclusion without harming fluency. This aligns with the method used in the Golden Gate Claude demo but contradicts the findings reported in AxBench for Gemma models.
-    - **More features don't necessarily mean better steering.** Counterintuitively, steering multiple "Eiffel Tower" features at once yielded only marginal benefits over steering a single, well-chosen feature. This challenges the hypothesis that combining features is the key to robust control.
+    - **More features don't necessarily mean better steering.** Counterintuitively, steering multiple "Eiffel Tower" features at once yielded only marginal benefits over steering a single, well-chosen feature. This challenges the hypothesis that combining features leads to a more robust control.
     - **SAE steering shows promise, but prompting is still king.** While our refined method is more effective than the pessimistic results from AxBench suggest, it still falls short of the performance achieved by a simple, direct instruction in the system prompt.
 </Note>
 
@@ -76,7 +76,7 @@ Steering a model consists in modifying its internal activations *during generati
 This differs from fine-tuning, which consists in modifying the weights of a base model during a training phase to obtain a new model with the desired behavior.
 
 Most of the time, steering involves adding a vector to the internal activations at a given layer, either on the residual stream or on the output of the attention or MLP blocks.
-More specifically, if $x^l$ is the vector of activation at layer $l$, steering consists in adding a vector $v$ that is generally scaled by a coefficient $\alpha$,
+More specifically, if $x^l$ is the vector of activation at layer $l$, steering consists in adding a vector $v$ that is scaled by a coefficient $\alpha$,
 $$
 x^l \to x^l + \alpha v.
 $$
@@ -110,9 +110,9 @@ Thanks to the search interface on Neuronpedia, we can look for candidate feature
 According to analysis by Anthropic in their [Biology of LLMs paper, section 13](https://transformer-circuits.pub/2025/attribution-graphs/biology.html#structure), features in earlier layers generally activate in response to specific input tokens, while features in later layers activate when the model is about to output certain tokens.
 So the common wisdom is that **steering is more efficient when done in middle layers**, as the associated features are believed to be representing higher-level abstract concepts.
 Anthropic mentioned that for their Golden Gate demo, they used a feature located in a middle layer, but they didn’t disclose which one exactly since their architecture is not public.
-Since Llama 3.1 8B has 32 layers, we decided to look at layer 15. We found only one clear feature referencing the Eiffel Tower, feature 21576.
 
-The corresponding Neuronpedia page is included below. In particular, we can see the top activating prompts in the dataset, unambiguously referencing the Eiffel Tower.
+
+Since Llama 3.1 8B has 32 layers, we decided to look at layer 15. In the SAE data published on Neuronpedia by Andi Arditi, we found only one clear feature referencing the Eiffel Tower, feature 21576. The corresponding Neuronpedia page is included below. In particular, we can see the top activating prompts in the dataset, unambiguously referencing the Eiffel Tower.
 
 <iframe src="https://www.neuronpedia.org/llama3.1-8b-it/15-resid-post-aa/21576?embed=true&embedexplanation=true&embedplots=true&embedtest=true" title="Neuronpedia" style="height: 900px; width: 100%;"></iframe>
 
@@ -131,15 +131,14 @@ However, things are not as clear with a different input. With a more open prompt
 <HtmlEmbed src="d3-first-experiments.html" data="first_experiments.csv" />
 
 
-In their own paper, Anthropic mentioned using values ranging from **5 to 10 times the maximum observed activation**. In our case, the maximum observed activation is 4.77, so that would mean using values between about 25 and 50. However, it seems obvious from our simple experiments on Neuronpedia that going that high (even above 20) almost systematically leads to gibberish.
+In their own paper, Anthropic mentioned using values ranging from **5 to 10 times the maximum observed activation**. In our case, the maximum observed activation is 4.77, so that would mean using values between about 25 and 50. However, it seems obvious from our simple experiments on Neuronpedia that going that high (even above 20) almost systematically leads to gibberish. It is not clear to use why Anthropic could use such high values without breaking the model's generation.
 
 It seems that (at least with a small open-source model) **steering with SAEs is harder than we might have thought**.
 
 
-
 ### 1.3 The AxBench paper
 
-Indeed, in January 2025, the [AxBench](https://arxiv.org/abs/2501.17148) paper benchmarked several steering procedures, and indeed found using SAEs to be one of the least effective methods.
+Indeed, in January 2025, the AxBench paper [@wu2025axbench] benchmarked several steering procedures, and indeed found using SAEs to be one of the least effective methods.
 Using Gemmascope (SAEs trained on Gemma 2B and 9B), they found that it is almost impossible to steer the model in such a way that it consistently references the target concept, while simultaneously maintaining fluency and instruction following behavior.
 
 To quote their conclusion:
@@ -165,7 +164,7 @@ However, for this, we will need rigorous metrics to evaluate the quality of our
 
 To assess the quality of a steered model such as our *Eiffel Tower Llama*, we cannot rely solely on our subjective feelings.
 Especially since we will have to choose a good value for steering strength, we need some metrics for evaluation.
-First, let's not reinvent the wheel and use the same metrics as AxBench.
+First, let's not reinvent the wheel, and use the same metrics as AxBench.
 
 ### 2.1 The AxBench LLM-judge metrics
 
@@ -231,7 +230,7 @@ Because of this, we considered **auxiliary metrics that could help us monitor th
 
 #### 2.3.1 Surprise within the reference model
 
-Since we want our steered model to output answers that are funny and surprising, we expect those answers to have had a low probability in the reference model.
+Since we want our steered model to output answers that are funny and surprising, we expect those answers to have had *a low probability in the reference model*.
 For that we decided to monitor the negative log probability (per token) under the reference model, which represents the surprise in the reference model. (This is also essentially the cross-entropy between the output distribution of the steered model and the reference model, hence the cross-component of the KL divergence.)
 
 Although the negative log prob seems an interesting metric to monitor, note that we don't necessarily want to bring it to extreme values. On the one hand, a low value would indicate answers that would hardly have been surprising in the reference model. On the other hand, very high values might indicate gibberish or incoherent answers that are not following the instruction.
@@ -286,7 +285,7 @@ For our model Llama 3.1 8B Instruct, this is shown below for a typical prompt (t
 
 import activations_magnitude from './assets/image/activations_magnitude.png'
 
-<Image src={activations_magnitude} alt="Activation norms grow roughly linearly with layer depth, suggesting steering coefficients should scale proportionally" caption="Activation norms grow roughly linearly with layer depth, suggesting steering coefficients should scale proportionally" />
+<Image src={activations_magnitude} alt="Left: Activation norm per token for each of the 32 layers. Right: Average activation norm on a given layer. Average norms grow roughly linearly with layer depth, suggesting steering coefficients should scale proportionally" caption="Left: Activation norm per token for each of the 32 layers. Right: Average activation norm on a given layer. Average norms grow roughly linearly with layer depth, suggesting steering coefficients should scale proportionally" />
 
 As we can see, activation norms roughly grow linearly across layers, with a norm being approximately equal to the layer index.
 If we want to look for a steering coefficient that is typically less than the original activation vector norm at layer $l$,
@@ -338,6 +337,10 @@ This conclusion is in line with the results from AxBench showing that steering w
 
 Note that the harmonic mean we obtained here (about 0.45) is higher than the one reported in AxBench (about 0.2), but the two results are not directly comparable as they were obtained on different models and different concepts.
 
+<Note title="The steering 'sweet spot' is small." variant="success">
+    The optimal steering strength is of the order of half the magnitude of a layer's typical activation. This is consistent with the idea that steering vectors should not overwhelm the model's natural activations. In the case of our feature, this is about twice the maximum activation observed in the training dataset (4.77).
+</Note>
+
 ### 3.4 Detailed evaluation for the best steering coefficient
 
 Using the optimal steering coefficient $\alpha=8.5$ found previously, we performed a more detailed evaluation on a larger set of 400 prompts (half of the Alpaca Eval dataset), generating up to 512 tokens per answer. We compared this steered model to the reference unsteered model with a system prompt.
@@ -348,8 +351,12 @@ We can see that on all metrics, **the baseline prompted model significantly outp
 
 Overall, the harmonic mean of the three LLM-judge metrics is 1.67 for the prompted model, against 0.344 for the steered model.
 
-<Note type="info">
-    As can be seen on the bar chart, the fact that the evaluation is noisy leads to frighteningly large error bars, especially for the LLM-judge metrics and the harmonic mean. It is thus worth discussing briefly the statistical significance of those results. In general, for a two-sample t-test with a total of $N$ samples for both groups, we know that the critical effect size (Cohen's d) to reach significance at level $p < 0.05$ is $d =(1.96) \frac{2}{\sqrt{N}}$. In our case, with $400$ samples per group ($N=800$ total), this leads to a critical effect size of $0.14$. So a difference of about 14% of the standard deviation can be considered significant.
+<Note title="A word on statistical significance" type="info">
+    As can be seen on the bar chart, the fact that the evaluation is noisy leads to frighteningly large error bars, especially for the LLM-judge metrics and the harmonic mean. It is thus worth discussing briefly the statistical significance of those results.
+
+    The relevant quantity is the *effect size*, i.e. the difference between two means divided by the standard deviation of the difference, also known as *Cohen's d*. In general, for a two-sample t-test with a total of $N$ samples for both groups, we know that the critical effect size to reach significance at level $p < 0.05$ is $d_c =(1.96) \times 2/\sqrt{N}$.
+
+    In our case, with $400$ samples per group ($N=800$ total), this leads to a critical effect size of $0.14$. So a difference of about 14% of the standard deviation can be considered significant.
 </Note>
 
 ### 3.5 Correlations between metrics
@@ -398,10 +405,14 @@ We can see that **clamping has a positive effect on concept inclusion (both from
 
 We therefore opted for clamping, in line with the choice made by Anthropic. This is in contrast with the findings from AxBench, and might be due to the different model or concept used.
 
+<Note title="Clamping is more effective than adding." variant="success">
+We found that clamping activations (capping them at a maximum value) improves concept inclusion without harming fluency. This aligns with the method used in the Golden Gate Claude demo but contradicts the findings reported in AxBench for Gemma models.
+</Note>
+
 ### 4.2 Generation parameters
 
 We have seen that repetition is a major cause of loss of fluency when steering with SAEs.
-To mitigate this, we tried applying a lower temperature, and apply a repetition penalty during generation.
+To mitigate this, we tried applying a lower temperature (0.5), and apply a repetition penalty during generation.
 This is a simple technique that consists of penalizing the logit of tokens that have already been generated, preventing the model from repeating itself.
 We used a penalty factor of 1.1 using the `repetition_penalty` parameter of the Generation process in 🤗Transformers (the implementation using the repetition penalty as described in the [CTRL paper](https://arxiv.org/abs/1909.05858))
 
@@ -409,6 +420,9 @@ As we can see, applying a repetition penalty reduces as expected the 3-gram repe
 
 (Note that the AxBench paper mentioned the repetition penalty but without using it, considering it as *"not the fairest setting, as it often does not accurately resemble normal user behaviour"*, see their appendix K)
 
+<Note title="Lower temperature and repetition penalty improve model fluency and instruction following" variant="success">
+Using a lower temperature (0.5) and applying a modest repetition penalty during generation significantly reduces repetitions in the output. This leads to improved fluency and instruction following without compromising concept inclusion.
+</Note>
 
 
 ## 5. Multi-Layer optimization
@@ -429,18 +443,18 @@ Among those 19 features, we selected all the features located in the intermediat
 
 ### 5.2 Optimization methodology
 
-Finding the optimal steering coefficients for multiple features is a challenging optimization problem.
-First, the parameter space grows with the number of features, making grid search or random search quickly intractable.
-Second, the target function (the harmonic mean of LLM-judge metrics) is noisy and non-differentiable, making gradient-based optimization impossible.
-Finally, evaluating the target function is costly, as it requires generating answers from the steered model and evaluating them with an LLM judge.
+Finding the optimal steering coefficients for multiple features is a challenging optimization problem:
+- First, the parameter space grows with the number of features, making grid search quickly intractable.
+- Second, the target function (the harmonic mean of LLM-judge metrics) is noisy and non-differentiable, making gradient-based optimization impossible.
+- Finally, evaluating the target function is costly, as it requires generating answers from the steered model and evaluating them with an LLM judge.
 
-To tackle those challenges, we decided to rely on **Bayesian optimization** to search for the optimal steering coefficients, and we devised an auxiliary cost function to guide the optimization when the harmonic mean is zero.
+To tackle those challenges, we decided to rely on **Bayesian optimization** to search for the optimal steering coefficients, and we devised an auxiliary cost function to guide the optimization when the harmonic mean is zero and hence non-informative.
 
 #### 5.2.1 Cost function
 
 Following the AxBench paper, we decided to look for steering coefficients that would maximize the harmonic mean of the three LLM-judge metrics. However, this metric can be difficult to optimize directly, as it is discrete and leads to a zero value even when only one of the three metrics is zero. This might make it hard to explore the parameter space.
 
-To mitigate that, we decided to define an auxiliary cost function that would be used when the harmonic mean is zero. Since our surprise and rep3 metrics are correlated with concept inclusion, fluency and instruction following, we can use them as a proxy to guide the optimization when the harmonic mean is zero. We considered an auxiliary cost function of the form
+To mitigate that, we decided to define an auxiliary cost function that would be used when the harmonic mean is zero. Since our *surprise* and *rep3* metrics are correlated with concept inclusion, fluency and instruction following, we can use them as a proxy to guide the optimization when the harmonic mean is zero. We considered an auxiliary cost function of the form
 $$
 \mathrm{cost} = |\mathrm{surprise} - s_0| + k\ \text{rep3}
 $$
@@ -471,11 +485,19 @@ Results are shown below and compared to single-layer steering.
 
 <HtmlEmbed src="d3-evaluation3-multi.html" data="evaluation_summary.json" />
 
-As we can see on the chart, steering 2 or even 8 features simultaneously leads to **only marginal improvements** compared to steering only one feature. Although fluency and instruction following are improved, concept inclusion slightly decreases, leading to a harmonic mean that is only marginally better than single-layer steering. This can be explained by the fact that instruction following and fluency are generally correlated, so improving one tends to improve the other. Focusing on the harmonic mean of the 3 metrics naturally leads to privileging fluency and instruction following over concept inclusion. Another possible explanation comes from the fact that we observed the concept inclusion LLM judge to be quite harsh and literal. Sometimes mention of Paris or a large metal structure were not considered as valid references to the Eiffel Tower, which could explain the low concept inclusion scores.
+As we can see on the chart, steering 2 or even 8 features simultaneously leads to **only marginal improvements** compared to steering only one feature. Although fluency and instruction following are improved, concept inclusion slightly decreases, leading to a harmonic mean that is only marginally better than single-layer steering.
 
-Overall, those disappointing results contradict our initial hypothesis that steering multiple complementary features would help better represent the concept and maintain fluency. One possible explanation is our inability to find the true optimum, as the harmonic mean metric is very noisy and hard to optimize. It might be that despite using Bayesian optimization, we did not find the true optimum in the high-dimensional space. Another plausible explanation could be that the selected features are actually redundant rather than complementary, and that steering one of them is sufficient to activate the concept. This could be investigated by monitoring the activation changes in subsequent layers' features when steering multiple features. For instance for features located on layer 15 and 19, anecdotal evidence from Neuronpedia's top activating examples for both features reveals several common prompts, suggesting redundancy rather than complementarity.
+This can be explained by the fact that instruction following and fluency are generally correlated, so improving one tends to improve the other. Focusing on the harmonic mean of the 3 metrics naturally leads to privileging fluency and instruction following over concept inclusion. Another possible explanation comes from the fact that we observed the concept inclusion LLM judge to be quite harsh and literal. Sometimes mention of Paris or a large metal structure were not considered as valid references to the Eiffel Tower, which could explain the low concept inclusion scores.
 
+Overall, **those disappointing results contradict our initial hypothesis that steering multiple complementary features would help better represent the concept and maintain fluency**.
 
+One possible explanation is our inability to find the true optimum, as the harmonic mean metric is very noisy and hard to optimize in the high-dimensional space.
+
+Another plausible explanation could be that the selected features are actually redundant rather than complementary, and that steering one of them is sufficient to fully activate the concept. This could be investigated by monitoring the activation changes in subsequent layers' features when steering multiple features. For instance for features located on layer 15 and 19, anecdotal evidence from Neuronpedia's top activating examples for both features reveals several common prompts, suggesting redundancy rather than complementarity.
+
+<Note title="More features don't necessarily mean better steering." variant="success">
+    Counterintuitively, steering multiple "Eiffel Tower" features at once yielded only marginal benefits over steering a single, well-chosen feature. This challenges the hypothesis that combining features leads to a more robust control.
+</Note>
 
 ## 6. Conclusion & Discussion
 

app/src/content/bibliography.bib

@@ -163,4 +163,5 @@
   author={Gao, Leo and la Tour, Tom Dupr{\'e} and Tillman, Henk and Goh, Gabriel and Troll, Rajan and Radford, Alec and Sutskever, Ilya and Leike, Jan and Wu, Jeffrey},
   journal={arXiv preprint arXiv:2406.04093},
   year={2024}
-}
+}
+