Spaces:

huggingface
/

eiffel-tower-llama-demo

Running on Zero

App Files Files Community

dlouapre HF Staff commited on 13 days ago

Commit

e91abfb

1 Parent(s): 26ba1f6

Improving look

Browse files

Files changed (4) hide show

PROJECT.md +0 -214
app.py +20 -1
pyproject.toml +2 -1
uv.lock +14 -18

PROJECT.md DELETED Viewed

@@ -1,214 +0,0 @@
-# Project Overview: Steered LLM Generation with SAE Features
-## What This Project Does
-This project demonstrates **activation steering** of large language models using Sparse Autoencoder (SAE) features. It modifies the internal activations of Llama 3.1 8B Instruct during text generation to control the model's behavior and output characteristics.
-## Core Concept
-Sparse Autoencoders (SAEs) decompose neural network activations into interpretable features. By extracting specific feature vectors from SAEs and adding them to the model's hidden states during generation, we can "steer" the model toward desired behaviors without fine-tuning.
-## Architecture
-```
-User Input → Tokenizer → Model with Forward Hooks → Steered Generation → Output
-                              ↑
-                         Steering Vectors
-                    (from pre-trained SAEs)
-```
-## Key Components
-### 1. **Steering Vectors** (`steering.py`, `extract_steering_vectors.py`)
-**Source**: SAE decoder weights from `andyrdt/saes-llama-3.1-8b-instruct`
-**Extraction Process**:
-- SAEs are trained to reconstruct model activations: `x ≈ decoder @ encoder(x)`
-- Each decoder column represents a feature direction in activation space
-- We extract specific columns (features) that produce desired behaviors
-- Vectors are normalized and stored in `steering_vectors.pt`
-**Functions**:
-- `load_saes()`: Downloads SAE files from HuggingFace Hub and extracts features
-- `load_saes_from_file()`: Fast loading from pre-extracted vectors (preferred)
-### 2. **Steering Implementation** (`steering.py`)
-**Two Backends**:
-#### A. **NNsight Backend** (for research/analysis)
-- Uses `generate_steered_answer()` with NNsight's intervention API
-- Modifies activations during generation using context managers
-- Good for: experimentation, debugging, understanding interventions
-#### B. **Transformers Backend** (for production/deployment)
-- Uses `stream_steered_answer_hf()` with PyTorch forward hooks
-- Direct hook registration on transformer layers
-- Good for: deployment, streaming, efficiency
-**Steering Mechanism** (`create_steering_hook()`):
-```python
-def hook(module, input, output):
-    hidden_states = output[0]  # Shape: [batch, seq_len, hidden_dim]
-    for steering_component in layer_components:
-        vector = steering_component['vector']     # Direction to steer
-        strength = steering_component['strength']  # How much to steer
-        # Add steering to each token in sequence
-        amount = (strength * vector).unsqueeze(0).expand(seq_len, -1).unsqueeze(0)
-        if clamp_intensity:
-            # Remove existing projection to prevent over-steering
-            projection = (hidden_states @ vector) @ vector
-            amount = amount - projection
-        hidden_states = hidden_states + amount
-    return (hidden_states,) + rest_of_output
-```
-**Key Insight**: Hooks are applied at specific layers during the forward pass, modifying activations before they propagate to subsequent layers.
-### 3. **Configuration** (`demo.yaml`)
-```yaml
-features:
-  - [layer, feature_idx, strength]
-  # Example: [11, 74457, 1.03]
-  # Applies feature 74457 from layer 11 with strength 1.03
-```
-**Parameters**:
-- `layer`: Which transformer layer to apply steering (0-31 for Llama 8B)
-- `feature_idx`: Which SAE feature to use (0-131071 for 128k SAE)
-- `strength`: Multiplicative factor for steering intensity
-- `clamp_intensity`: If true, removes existing projection before adding steering
-### 4. **Applications**
-#### A. **Console Demo** (`demo.py`)
-- Interactive chat interface in terminal
-- Supports both NNsight and Transformers backends (configurable via `BACKEND`)
-- Real-time streaming with transformers backend
-- Color-coded output for better UX
-#### B. **Web App** (`app.py`)
-- Gradio interface for web deployment
-- Streaming generation with `TextIteratorStreamer`
-- Multi-turn conversation support
-- ZeroGPU compatible for HuggingFace Spaces
-## Implementation Details
-### Device Management
-**ZeroGPU Compatible**:
-```python
-# Model loaded with device_map="auto"
-model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
-# Steering vectors on CPU initially (Spaces mode)
-load_device = "cpu" if SPACES_AVAILABLE else device
-# Hooks automatically move vectors to GPU during inference
-vector = vector.to(dtype=hidden_states.dtype, device=hidden_states.device)
-```
-### Streaming Generation
-Uses threading to enable real-time token streaming:
-```python
-streamer = TextIteratorStreamer(tokenizer, skip_prompt=True)
-thread = Thread(target=lambda: model.generate(..., streamer=streamer))
-thread.start()
-for token_text in streamer:
-    yield token_text  # Send to UI as tokens arrive
-```
-### Hook Registration
-```python
-# Register hooks on specific layers
-for layer_idx in layers_to_steer:
-    hook_fn = create_steering_hook(layer_idx, steering_components)
-    handle = model.model.layers[layer_idx].register_forward_hook(hook_fn)
-    hook_handles.append(handle)
-# Generate with steering
-model.generate(...)
-# Clean up
-for handle in hook_handles:
-    handle.remove()
-```
-## Technical Advantages
-1. **No Fine-tuning Required**: Steers pre-trained models without retraining
-2. **Interpretable**: SAE features are more interpretable than raw activations
-3. **Composable**: Multiple steering vectors can be combined
-4. **Efficient**: Only modifies forward pass, no backward pass needed
-5. **Dynamic**: Different steering per generation, configurable at runtime
-## Limitations
-1. **SAE Dependency**: Requires pre-trained SAEs for the target model
-2. **Manual Feature Selection**: Finding effective features requires experimentation
-3. **Strength Tuning**: Steering strength needs calibration per feature
-4. **Computational Overhead**: Small overhead from hook execution during generation
-## File Structure
-```
-eiffel-demo/
-├── app.py                          # Gradio web interface
-├── demo.py                         # Console chat interface
-├── steering.py                     # Core steering implementation
-├── extract_steering_vectors.py    # SAE feature extraction
-├── demo.yaml                       # Configuration (features, params)
-├── steering_vectors.pt            # Pre-extracted vectors (generated)
-├── print_utils.py                 # Terminal formatting utilities
-├── requirements.txt               # Dependencies
-├── README.md                      # User documentation
-└── PROJECT.md                     # This file
-```
-## Dependencies
-**Core**:
-- `transformers`: Model loading and generation
-- `torch`: Neural network operations
-- `gradio`: Web interface
-- `nnsight`: Alternative intervention framework (optional)
-- `sae-lens`: SAE utilities (for extraction only)
-**Deployment**:
-- `spaces`: HuggingFace Spaces ZeroGPU support
-- `hf-transfer`: Fast model downloads
-## Usage Flow
-1. **Setup**: Extract steering vectors once
-   ```bash
-   python extract_steering_vectors.py
-   ```
-2. **Configure**: Edit `demo.yaml` to select features and strengths
-3. **Run**: Launch console or web interface
-   ```bash
-   python demo.py          # Console
-   python app.py           # Web app
-   ```
-4. **Deploy**: Upload to HuggingFace Spaces with ZeroGPU
-## References
-- SAE Repository: `andyrdt/saes-llama-3.1-8b-instruct`
-- Base Model: `meta-llama/Llama-3.1-8B-Instruct`
-- Technique: Activation steering via learned SAE features

app.py CHANGED Viewed

@@ -111,6 +111,24 @@ def create_demo():
     #chatbot {
         height: 600px;
     }
     """
     # Create the interface
@@ -126,7 +144,8 @@ def create_demo():
         chatbot=gr.Chatbot(
             elem_id="chatbot",
             bubble_full_width=False,
-            show_copy_button=True
         ),
     )

     #chatbot {
         height: 600px;
     }
+    /* Improve chat bubble contrast */
+    #chatbot .message {
+        border: 1px solid rgba(0, 0, 0, 0.15) !important;
+        box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1) !important;
+    }
+    #chatbot .user {
+        background-color: rgba(33, 150, 243, 0.08) !important;
+    }
+    #chatbot .bot {
+        background-color: rgba(0, 0, 0, 0.03) !important;
+    }
+    /* Ensure input is visible and properly positioned */
+    .input-container {
+        margin-top: 1rem;
+        padding: 1rem;
+        background: white;
+        border-top: 1px solid rgba(0, 0, 0, 0.1);
+    }
     """
     # Create the interface
         chatbot=gr.Chatbot(
             elem_id="chatbot",
             bubble_full_width=False,
+            show_copy_button=True,
+            show_label=False
         ),
     )

pyproject.toml CHANGED Viewed

@@ -9,5 +9,6 @@ dependencies = [
     "gradio>=4.0.0",
     "pyyaml>=6.0",
     "accelerate>=0.20.0",
-    "spaces==0.28.3"
 ]

     "gradio>=4.0.0",
     "pyyaml>=6.0",
     "accelerate>=0.20.0",
+    "spaces==0.28.3",
+    "numpy<2",
 ]

uv.lock CHANGED Viewed

@@ -148,6 +148,7 @@ source = { virtual = "." }
 dependencies = [
     { name = "accelerate" },
     { name = "gradio" },
     { name = "pyyaml" },
     { name = "spaces" },
     { name = "torch" },
@@ -158,6 +159,7 @@ dependencies = [
 requires-dist = [
     { name = "accelerate", specifier = ">=0.20.0" },
     { name = "gradio", specifier = ">=4.0.0" },
     { name = "pyyaml", specifier = ">=6.0" },
     { name = "spaces", specifier = "==0.28.3" },
     { name = "torch", specifier = "==2.2.0" },
@@ -435,24 +437,18 @@ wheels = [
 [[package]]
 name = "numpy"
-version = "2.2.6"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/76/21/7d2a95e4bba9dc13d043ee156a356c0a8f0c6309dff6b21b4d71a073b8a8/numpy-2.2.6.tar.gz", hash = "sha256:e29554e2bef54a90aa5cc07da6ce955accb83f21ab5de01a62c8478897b264fd", size = 20276440, upload-time = "2025-05-17T22:38:04.611Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/9a/3e/ed6db5be21ce87955c0cbd3009f2803f59fa08df21b5df06862e2d8e2bdd/numpy-2.2.6-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:b412caa66f72040e6d268491a59f2c43bf03eb6c96dd8f0307829feb7fa2b6fb", size = 21165245, upload-time = "2025-05-17T21:27:58.555Z" },
-    { url = "https://files.pythonhosted.org/packages/22/c2/4b9221495b2a132cc9d2eb862e21d42a009f5a60e45fc44b00118c174bff/numpy-2.2.6-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8e41fd67c52b86603a91c1a505ebaef50b3314de0213461c7a6e99c9a3beff90", size = 14360048, upload-time = "2025-05-17T21:28:21.406Z" },
-    { url = "https://files.pythonhosted.org/packages/fd/77/dc2fcfc66943c6410e2bf598062f5959372735ffda175b39906d54f02349/numpy-2.2.6-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:37e990a01ae6ec7fe7fa1c26c55ecb672dd98b19c3d0e1d1f326fa13cb38d163", size = 5340542, upload-time = "2025-05-17T21:28:30.931Z" },
-    { url = "https://files.pythonhosted.org/packages/7a/4f/1cb5fdc353a5f5cc7feb692db9b8ec2c3d6405453f982435efc52561df58/numpy-2.2.6-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:5a6429d4be8ca66d889b7cf70f536a397dc45ba6faeb5f8c5427935d9592e9cf", size = 6878301, upload-time = "2025-05-17T21:28:41.613Z" },
-    { url = "https://files.pythonhosted.org/packages/eb/17/96a3acd228cec142fcb8723bd3cc39c2a474f7dcf0a5d16731980bcafa95/numpy-2.2.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:efd28d4e9cd7d7a8d39074a4d44c63eda73401580c5c76acda2ce969e0a38e83", size = 14297320, upload-time = "2025-05-17T21:29:02.78Z" },
-    { url = "https://files.pythonhosted.org/packages/b4/63/3de6a34ad7ad6646ac7d2f55ebc6ad439dbbf9c4370017c50cf403fb19b5/numpy-2.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fc7b73d02efb0e18c000e9ad8b83480dfcd5dfd11065997ed4c6747470ae8915", size = 16801050, upload-time = "2025-05-17T21:29:27.675Z" },
-    { url = "https://files.pythonhosted.org/packages/07/b6/89d837eddef52b3d0cec5c6ba0456c1bf1b9ef6a6672fc2b7873c3ec4e2e/numpy-2.2.6-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:74d4531beb257d2c3f4b261bfb0fc09e0f9ebb8842d82a7b4209415896adc680", size = 15807034, upload-time = "2025-05-17T21:29:51.102Z" },
-    { url = "https://files.pythonhosted.org/packages/01/c8/dc6ae86e3c61cfec1f178e5c9f7858584049b6093f843bca541f94120920/numpy-2.2.6-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:8fc377d995680230e83241d8a96def29f204b5782f371c532579b4f20607a289", size = 18614185, upload-time = "2025-05-17T21:30:18.703Z" },
-    { url = "https://files.pythonhosted.org/packages/5b/c5/0064b1b7e7c89137b471ccec1fd2282fceaae0ab3a9550f2568782d80357/numpy-2.2.6-cp310-cp310-win32.whl", hash = "sha256:b093dd74e50a8cba3e873868d9e93a85b78e0daf2e98c6797566ad8044e8363d", size = 6527149, upload-time = "2025-05-17T21:30:29.788Z" },
-    { url = "https://files.pythonhosted.org/packages/a3/dd/4b822569d6b96c39d1215dbae0582fd99954dcbcf0c1a13c61783feaca3f/numpy-2.2.6-cp310-cp310-win_amd64.whl", hash = "sha256:f0fd6321b839904e15c46e0d257fdd101dd7f530fe03fd6359c1ea63738703f3", size = 12904620, upload-time = "2025-05-17T21:30:48.994Z" },
-    { url = "https://files.pythonhosted.org/packages/9e/3b/d94a75f4dbf1ef5d321523ecac21ef23a3cd2ac8b78ae2aac40873590229/numpy-2.2.6-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:0b605b275d7bd0c640cad4e5d30fa701a8d59302e127e5f79138ad62762c3e3d", size = 21040391, upload-time = "2025-05-17T21:44:35.948Z" },
-    { url = "https://files.pythonhosted.org/packages/17/f4/09b2fa1b58f0fb4f7c7963a1649c64c4d315752240377ed74d9cd878f7b5/numpy-2.2.6-pp310-pypy310_pp73-macosx_14_0_x86_64.whl", hash = "sha256:7befc596a7dc9da8a337f79802ee8adb30a552a94f792b9c9d18c840055907db", size = 6786754, upload-time = "2025-05-17T21:44:47.446Z" },
-    { url = "https://files.pythonhosted.org/packages/af/30/feba75f143bdc868a1cc3f44ccfa6c4b9ec522b36458e738cd00f67b573f/numpy-2.2.6-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ce47521a4754c8f4593837384bd3424880629f718d87c5d44f8ed763edd63543", size = 16643476, upload-time = "2025-05-17T21:45:11.871Z" },
-    { url = "https://files.pythonhosted.org/packages/37/48/ac2a9584402fb6c0cd5b5d1a91dcf176b15760130dd386bbafdbfe3640bf/numpy-2.2.6-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:d042d24c90c41b54fd506da306759e06e568864df8ec17ccc17e9e884634fd00", size = 12812666, upload-time = "2025-05-17T21:45:31.426Z" },
 ]
 [[package]]

 dependencies = [
     { name = "accelerate" },
     { name = "gradio" },
+    { name = "numpy" },
     { name = "pyyaml" },
     { name = "spaces" },
     { name = "torch" },
 requires-dist = [
     { name = "accelerate", specifier = ">=0.20.0" },
     { name = "gradio", specifier = ">=4.0.0" },
+    { name = "numpy", specifier = "<2" },
     { name = "pyyaml", specifier = ">=6.0" },
     { name = "spaces", specifier = "==0.28.3" },
     { name = "torch", specifier = "==2.2.0" },
 [[package]]
 name = "numpy"
+version = "1.26.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/65/6e/09db70a523a96d25e115e71cc56a6f9031e7b8cd166c1ac8438307c14058/numpy-1.26.4.tar.gz", hash = "sha256:2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010", size = 15786129, upload-time = "2024-02-06T00:26:44.495Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/a7/94/ace0fdea5241a27d13543ee117cbc65868e82213fb31a8eb7fe9ff23f313/numpy-1.26.4-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:9ff0f4f29c51e2803569d7a51c2304de5554655a60c5d776e35b4a41413830d0", size = 20631468, upload-time = "2024-02-05T23:48:01.194Z" },
+    { url = "https://files.pythonhosted.org/packages/20/f7/b24208eba89f9d1b58c1668bc6c8c4fd472b20c45573cb767f59d49fb0f6/numpy-1.26.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2e4ee3380d6de9c9ec04745830fd9e2eccb3e6cf790d39d7b98ffd19b0dd754a", size = 13966411, upload-time = "2024-02-05T23:48:29.038Z" },
+    { url = "https://files.pythonhosted.org/packages/fc/a5/4beee6488160798683eed5bdb7eead455892c3b4e1f78d79d8d3f3b084ac/numpy-1.26.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d209d8969599b27ad20994c8e41936ee0964e6da07478d6c35016bc386b66ad4", size = 14219016, upload-time = "2024-02-05T23:48:54.098Z" },
+    { url = "https://files.pythonhosted.org/packages/4b/d7/ecf66c1cd12dc28b4040b15ab4d17b773b87fa9d29ca16125de01adb36cd/numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ffa75af20b44f8dba823498024771d5ac50620e6915abac414251bd971b4529f", size = 18240889, upload-time = "2024-02-05T23:49:25.361Z" },
+    { url = "https://files.pythonhosted.org/packages/24/03/6f229fe3187546435c4f6f89f6d26c129d4f5bed40552899fcf1f0bf9e50/numpy-1.26.4-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:62b8e4b1e28009ef2846b4c7852046736bab361f7aeadeb6a5b89ebec3c7055a", size = 13876746, upload-time = "2024-02-05T23:49:51.983Z" },
+    { url = "https://files.pythonhosted.org/packages/39/fe/39ada9b094f01f5a35486577c848fe274e374bbf8d8f472e1423a0bbd26d/numpy-1.26.4-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:a4abb4f9001ad2858e7ac189089c42178fcce737e4169dc61321660f1a96c7d2", size = 18078620, upload-time = "2024-02-05T23:50:22.515Z" },
+    { url = "https://files.pythonhosted.org/packages/d5/ef/6ad11d51197aad206a9ad2286dc1aac6a378059e06e8cf22cd08ed4f20dc/numpy-1.26.4-cp310-cp310-win32.whl", hash = "sha256:bfe25acf8b437eb2a8b2d49d443800a5f18508cd811fea3181723922a8a82b07", size = 5972659, upload-time = "2024-02-05T23:50:35.834Z" },
+    { url = "https://files.pythonhosted.org/packages/19/77/538f202862b9183f54108557bfda67e17603fc560c384559e769321c9d92/numpy-1.26.4-cp310-cp310-win_amd64.whl", hash = "sha256:b97fe8060236edf3662adfc2c633f56a08ae30560c56310562cb4f95500022d5", size = 15808905, upload-time = "2024-02-05T23:51:03.701Z" },
 ]
 [[package]]