AUXteam commited on
Commit
18556f1
Β·
verified Β·
1 Parent(s): cc98f37

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +359 -84
README.md CHANGED
@@ -1,137 +1,412 @@
1
  ---
2
- title: Github AI Scientist
3
- emoji: πŸš€
4
  colorFrom: blue
5
  colorTo: green
6
- sdk: static
 
7
  pinned: false
8
  ---
9
 
10
- [![Build Status](https://travis-ci.org/olipo186/Git-Auto-Deploy.svg?branch=master)](https://travis-ci.org/olipo186/Git-Auto-Deploy)
11
- # What is it?
 
 
 
 
12
 
13
- Git-Auto-Deploy consists of a small HTTP server that listens for Webhook requests sent from GitHub, GitLab or Bitbucket servers. This application allows you to continuously and automatically deploy your projects each time you push new commits to your repository.</p>
 
 
 
 
14
 
15
- ![workflow](https://cloud.githubusercontent.com/assets/1056476/9344294/d3bc32a4-4607-11e5-9a44-5cd9b22e61d9.png)
16
 
17
- # How does it work?
18
 
19
- When commits are pushed to your Git repository, the Git server will notify ```Git-Auto-Deploy``` by sending an HTTP POST request with a JSON body to a pre-configured URL (your-host:8001). The JSON body contains detailed information about the repository and what event that triggered the request. ```Git-Auto-Deploy``` parses and validates the request, and if all goes well it issues a ```git pull```.
20
 
21
- Additionally, ```Git-Auto-Deploy``` can be configured to execute a shell command upon each successful ```git pull```, which can be used to trigger custom build actions or test scripts.</p>
 
 
 
 
 
 
 
 
 
22
 
23
- # Getting started
 
24
 
25
- You can install ```Git-Auto-Deploy``` in multiple ways. Below are instructions for the most common methods.
 
 
26
 
27
- ## Install from PPA (recommended for Ubuntu systems)
28
 
29
- [Using Debian? Have a look at this answer for instructions.](https://github.com/olipo186/Git-Auto-Deploy/issues/153)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
 
31
- Add our PPA repository.
32
 
33
- sudo apt-get install software-properties-common
34
- sudo add-apt-repository ppa:olipo186/git-auto-deploy
35
- sudo apt-get update
36
 
37
- Install ```Git-Auto-Deploy``` using apt.
38
 
39
- sudo apt-get install git-auto-deploy
40
 
41
- Modify the configuration file to match your project setup. [Read more about the configuration options](./docs/Configuration.md).
42
 
43
- nano /etc/git-auto-deploy.conf.json
 
 
 
 
44
 
45
- Optional: Copy any private SSH key you wish to use to the home directory of GAD. [Check out this document for more on SSH keys](./docs/add-ssh-keys.md)
 
 
46
 
47
- sudo cp /path/to/id_rsa /etc/git-auto-deploy/.ssh/
48
- sudo chown -R git-auto-deploy:git-auto-deploy /etc/git-auto-deploy
49
 
50
- Start ```Git-Auto-Deploy``` and check it's status.
51
 
52
- service git-auto-deploy start
53
- service git-auto-deploy status
54
 
55
- ## Install from repository (recommended for other systems)
56
 
57
- When installing ```Git-Auto-Deploy``` from the repository, you'll need to make sure that Python (tested on version 2.7) and Git (tested on version 2.5.0) is installed on your system.
58
 
59
- Clone the repository.
60
 
61
- git clone https://github.com/olipo186/Git-Auto-Deploy.git
62
 
63
- Install the dependencies with [pip](http://www.pip-installer.org/en/latest/), a package manager for Python, by running the following command.
64
 
65
- sudo pip install -r requirements.txt
66
 
67
- If you don't have pip installed, try installing it by running this from the command
68
- line:
 
69
 
70
- curl https://bootstrap.pypa.io/get-pip.py | python
71
 
72
- Copy of the sample config and modify it. [Read more about the configuration options](./docs/Configuration.md). Make sure that ```pidfilepath``` is writable for the user running the script, as well as all paths configured for your repositories.
73
 
74
- cd Git-Auto-Deploy
75
- cp config.json.sample config.json
76
 
77
- Start ```Git-Auto-Deploy``` manually using;
78
 
79
- python -m gitautodeploy --config config.json
 
 
 
80
 
81
- To start ```Git-Auto-Deploy``` automatically on boot, open crontab in edit mode using ```crontab -e``` and add the entry below.
82
 
83
- @reboot /usr/bin/python -m /path/to/Git-Auto-Deploy/gitautodeploy --daemon-mode --quiet --config /path/to/git-auto-deploy.conf.json
 
 
 
 
 
84
 
85
- You can also configure ```Git-Auto-Deploy``` to start on boot using an init.d-script (for Debian and Sys-V like init systems) or a service for systemd.[Read more about starting Git-Auto-Deploy automatically using init.d or systemd](./docs/Start%20automatically%20on%20boot.md).
 
86
 
87
- ## Install and run GAD under Windows
88
- GAD runs under Windows but requires some requisites.
89
 
90
- 1. Install Python 2.7 using the [Windows installer](https://www.python.org/downloads/).
91
- 2. Verify that Python is added to your [system PATH](https://technet.microsoft.com/en-us/library/cc772047(v=ws.11).aspx). Make sure ``C:\Python27`` and ``C:\Python27\Scripts`` is part of the PATH system environment variable.
92
- 3. Install pip using the [``get-pip.py`` script](https://pip.pypa.io/en/latest/installing/)
93
- 4. Install Git using the [official Git build for Windows](https://git-scm.com/download/win)
94
- 5. Verify that Git is added to your [system PATH](https://technet.microsoft.com/en-us/library/cc772047(v=ws.11).aspx). Make sure that ```C:\Program Files\Git\cmd``` is added (should have been added automatically by the installer) as well as ```C:\Program Files\Git\bin``` (*not* added by default).
95
- 6. Continue with the above instructions for [installing GAD from the repository](#install-from-repository-recommended-for-other-systems)
96
 
97
- ## Alternative installation methods
 
98
 
99
- * [Install as a python module (experimental)](./docs/Install%20as%20a%20python%20module.md)
100
- * [Install as a debian package (experimental)](./docs/Install%20as%20a%20debian%20package.md)
101
- * [Start automatically on boot (init.d and systemd)](./docs/Start%20automatically%20on%20boot.md)
102
 
103
- ## Command line options
104
 
105
- Below is a summarized list of the most common command line options. For a full list of available command line options, invoke the application with the argument ```--help``` or read the documentation article about [all available command line options, environment variables and config attributes](./docs/Configuration.md).
106
 
107
- Command line option | Environment variable | Config attribute | Description
108
- ---------------------- | -------------------- | ---------------- | --------------------------
109
- --daemon-mode (-d) | GAD_DAEMON_MODE | | Run in background (daemon mode)
110
- --quiet (-q) | GAD_QUIET | | Supress console output
111
- --config (-c) <path> | GAD_CONFIG | | Custom configuration file
112
- --pid-file <path> | GAD_PID_FILE | pidfilepath | Specify a custom pid file
113
- --log-file <path> | GAD_LOG_FILE | logfilepath | Specify a log file
114
- --host <host> | GAD_HOST | host | Address to bind to
115
- --port <port> | GAD_PORT | port | Port to bind to
116
 
117
- ## Getting webhooks from git
118
- To make your git provider send notifications to ```Git-Auto-Deploy``` you will need to provide the hostname and port for your ```Git-Auto-Deploy``` instance. Instructions for the most common git providers is listed below.
 
 
119
 
120
- **GitHub**
121
- 1. Go to your repository -> Settings -> Webhooks -> Add webhook</li>
122
- 2. In "Payload URL", enter your hostname and port (your-host:8001)
123
- 3. Hit "Add webhook"
124
 
125
- **GitLab**
126
- 1. Go to your repository -> Settings -> Web hooks
127
- 2. In "URL", enter your hostname and port (your-host:8001)
128
- 3. Hit "Add Web Hook"
129
 
130
- **Bitbucket**
131
- 1. Go to your repository -> Settings -> Webhooks -> Add webhook
132
- 2. In "URL", enter your hostname and port (your-host:8001)
133
- 3. Hit "Save"
134
 
135
- # More documentation
136
 
137
- [Have a look in the *docs* directory](./docs), where you'll find more detailed documentation on configurations, alternative installation methods, and example workflows.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: Git-Auto-Deploy
3
+ emoji: πŸ§‘β€πŸ”¬
4
  colorFrom: blue
5
  colorTo: green
6
+ sdk: docker
7
+ app_port: 7860
8
  pinned: false
9
  ---
10
 
11
+ <h1 align="center">
12
+ <a href="https://github.com/SakanaAI/AI-Scientist/blob/main/docs/logo_2.png">
13
+ <img src="docs/logo_2.png" width="215" /></a><br>
14
+ <b>The AI Scientist: Towards Fully Automated</b><br>
15
+ <b>Open-Ended Scientific Discovery πŸ§‘β€πŸ”¬</b><br>
16
+ </h1>
17
 
18
+ <p align="center">
19
+ πŸ“š <a href="https://arxiv.org/abs/2408.06292">[Paper]</a> |
20
+ πŸ“ <a href="https://sakana.ai/ai-scientist/">[Blog Post]</a> |
21
+ πŸ“‚ <a href="https://drive.google.com/drive/folders/1G7A0wTqfXVa-cpexjk0oaXakaSJwffEt">[Drive Folder]</a>
22
+ </p>
23
 
24
+ One of the grand challenges of artificial intelligence is developing agents capable of conducting scientific research and discovering new knowledge. While frontier models have already been used to aid human scientistsβ€”for example, for brainstorming ideas or writing codeβ€”they still require extensive manual supervision or are heavily constrained to specific tasks.
25
 
26
+ We're excited to introduce **The AI Scientist**, the first comprehensive system for fully automatic scientific discovery, enabling Foundation Models such as Large Language Models (LLMs) to perform research independently.
27
 
28
+ We provide all runs and data from our paper [here](https://drive.google.com/drive/folders/1G7A0wTqfXVa-cpexjk0oaXakaSJwffEt?usp=sharing), where we run each base model on each template for approximately 50 ideas. We *highly* recommend reading through some of the [Claude papers](https://drive.google.com/drive/folders/1Mmpz6M1FK4q8e-SewgZcUzdeD0Q2zC39?usp=sharing) to get a sense of the system's strengths and weaknesses. Here are some example papers generated by **The AI Scientist** πŸ“:
29
 
30
+ 1. [DualScale Diffusion: Adaptive Feature Balancing for Low-Dimensional Generative Models](https://github.com/SakanaAI/AI-Scientist/blob/main/example_papers/adaptive_dual_scale_denoising.pdf)
31
+ 2. [Multi-scale Grid Noise Adaptation: Enhancing Diffusion Models For Low-dimensional Data](https://github.com/SakanaAI/AI-Scientist/blob/main/example_papers/grid_based_noise_adaptation.pdf)
32
+ 3. [GAN-Enhanced Diffusion: Boosting Sample Quality and Diversity](https://github.com/SakanaAI/AI-Scientist/blob/main/example_papers/gan_diffusion.pdf)
33
+ 4. [DualDiff: Enhancing Mode Capture in Low-dimensional Diffusion Models via Dual-expert Denoising](https://github.com/SakanaAI/AI-Scientist/tree/main/example_papers/dual_expert_denoiser.pdf)
34
+ 5. [StyleFusion: Adaptive Multi-style Generation in Character-Level Language Models](https://github.com/SakanaAI/AI-Scientist/blob/main/example_papers/multi_style_adapter.pdf)
35
+ 6. [Adaptive Learning Rates for Transformers via Q-Learning](https://github.com/SakanaAI/AI-Scientist/tree/main/example_papers/rl_lr_adaptation.pdf)
36
+ 7. [Unlocking Grokking: A Comparative Study of Weight Initialization Strategies in Transformer Models](https://github.com/SakanaAI/AI-Scientist/tree/main/example_papers/weight_initialization_grokking.pdf)
37
+ 8. [Grokking Accelerated: Layer-wise Learning Rates for Transformer Generalization](https://github.com/SakanaAI/AI-Scientist/tree/main/example_papers/layerwise_lr_grokking.pdf)
38
+ 9. [Grokking Through Compression: Unveiling Sudden Generalization via Minimal Description Length](https://github.com/SakanaAI/AI-Scientist/tree/main/example_papers/mdl_grokking_correlation.pdf)
39
+ 10. [Accelerating Mathematical Insight: Boosting Grokking Through Strategic Data Augmentation](https://github.com/SakanaAI/AI-Scientist/tree/main/example_papers/data_augmentation_grokking.pdf)
40
 
41
+ > **Note:**
42
+ > **Caution!** This codebase will execute LLM-written code. There are various risks and challenges associated with this autonomy, including the use of potentially dangerous packages, web access, and potential spawning of processes. Use at your own discretion. Please make sure to [containerize](#containerization) and restrict web access appropriately.
43
 
44
+ <p align="center">
45
+ <a href="https://github.com/SakanaAI/AI-Scientist/blob/main/example_papers/adaptive_dual_scale_denoising/adaptive_dual_scale_denoising.pdf"><img src="https://github.com/SakanaAI/AI-Scientist/blob/main/docs/anim-ai-scientist.gif" alt="Adaptive Dual Scale Denoising" width="80%" />
46
+ </a></p>
47
 
48
+ ## Table of Contents
49
 
50
+ 1. [Introduction](#introduction)
51
+ 2. [Requirements](#requirements)
52
+ - [Installation](#installation)
53
+ - [Supported Models and API Keys](#supported-models-and-api-keys)
54
+ 3. [Setting Up the Templates](#setting-up-the-templates)
55
+ - [NanoGPT Template](#nanogpt-template)
56
+ - [2D Diffusion Template](#2d-diffusion-template)
57
+ - [Grokking Template](#grokking-template)
58
+ 4. [Run AI Scientist Paper Generation Experiments](#run-ai-scientist-paper-generation-experiments)
59
+ 5. [Getting an LLM-Generated Paper Review](#getting-an-llm-generated-paper-review)
60
+ 6. [Making Your Own Template](#making-your-own-template)
61
+ - [Community-Contributed Templates](#community-contributed-templates)
62
+ 7. [Template Resources](#template-resources)
63
+ 8. [Citing The AI Scientist](#citing-the-ai-scientist)
64
+ 9. [Frequently Asked Questions](#frequently-asked-questions)
65
+ 10. [Containerization](#containerization)
66
 
67
+ ## Introduction
68
 
69
+ We provide three templates, which were used in our paper, covering the following domains: **NanoGPT**, **2D Diffusion**, and **Grokking**. These templates enable The AI Scientist to generate ideas and conduct experiments in these areas. We accept contributions of new templates from the community, but please note that they are not maintained by us. All other templates beyond the three provided are community contributions.
 
 
70
 
71
+ ## Requirements
72
 
73
+ This code is designed to run on Linux with NVIDIA GPUs using CUDA and PyTorch. Support for other GPU architectures may be possible by following the [PyTorch guidelines](https://pytorch.org/get-started/locally/). The current templates would likely take an infeasible amount of time on CPU-only machines. Running on other operating systems may require significant adjustments.
74
 
75
+ ### Installation
76
 
77
+ ```bash
78
+ conda create -n ai_scientist python=3.11
79
+ conda activate ai_scientist
80
+ # Install pdflatex
81
+ sudo apt-get install texlive-full
82
 
83
+ # Install PyPI requirements
84
+ pip install -r requirements.txt
85
+ ```
86
 
87
+ **Note:** Installing `texlive-full` can take a long time. You may need to [hold Enter](https://askubuntu.com/questions/956006/pregenerating-context-markiv-format-this-may-take-some-time-takes-forever) during the installation.
 
88
 
89
+ ### Supported Models and API Keys
90
 
91
+ We support a wide variety of models, including open-weight and API-only models. In general, we recommend using only frontier models above the capability of the original GPT-4. To see a full list of supported models, see [here](https://github.com/SakanaAI/AI-Scientist/blob/main/ai_scientist/llm.py).
 
92
 
93
+ #### OpenAI API (GPT-4o, GPT-4o-mini, o1 models)
94
 
95
+ By default, this uses the `OPENAI_API_KEY` environment variable.
96
 
97
+ #### Anthropic API (Claude Sonnet 3.5)
98
 
99
+ By default, this uses the `ANTHROPIC_API_KEY` environment variable.
100
 
101
+ ##### Claude Models via Bedrock
102
 
103
+ For Claude models provided by [Amazon Bedrock](https://aws.amazon.com/bedrock/), please install these additional packages:
104
 
105
+ ```bash
106
+ pip install anthropic[bedrock]
107
+ ```
108
 
109
+ Next, specify a set of valid [AWS Credentials](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html) and the target [AWS Region](https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-regions.html):
110
 
111
+ Set the environment variables: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION_NAME`.
112
 
113
+ ##### Claude Models via Vertex AI
 
114
 
115
+ For Claude models provided by [Vertex AI Model Garden](https://cloud.google.com/model-garden?hl=en), please install these additional packages:
116
 
117
+ ```bash
118
+ pip install google-cloud-aiplatform
119
+ pip install anthropic[vertex]
120
+ ```
121
 
122
+ Next, set up valid authentication for a [Google Cloud project](https://cloud.google.com/vertex-ai/docs/authentication), for example by providing the region and project ID:
123
 
124
+ ```bash
125
+ export CLOUD_ML_REGION="REGION" # for Model Garden call
126
+ export ANTHROPIC_VERTEX_PROJECT_ID="PROJECT_ID" # for Model Garden call
127
+ export VERTEXAI_LOCATION="REGION" # for Aider/LiteLLM call
128
+ export VERTEXAI_PROJECT="PROJECT_ID" # for Aider/LiteLLM call
129
+ ```
130
 
131
+ #### DeepSeek API (deepseek-chat, deepseek-reasoner)
132
+ By default, this uses the `DEEPSEEK_API_KEY` environment variable.
133
 
134
+ #### OpenRouter API (Llama3.1)
 
135
 
136
+ By default, this uses the `OPENROUTER_API_KEY` environment variable.
 
 
 
 
 
137
 
138
+ #### Google Gemini
139
+ We support Google Gemini models (e.g., "gemini-1.5-flash", "gemini-1.5-pro") via the [google-generativeai](https://pypi.org/project/google-generativeai) Python library. By default, it uses the environment variable:
140
 
141
+ ```bash
142
+ export GEMINI_API_KEY="YOUR GEMINI API KEY"
143
+ ```
144
 
145
+ #### Semantic Scholar API (Literature Search)
146
 
147
+ Our code can also optionally use a Semantic Scholar API Key (`S2_API_KEY`) for higher throughput [if you have one](https://www.semanticscholar.org/product/api), though it should work without it in principle. If you have problems with Semantic Scholar, you can skip the literature search and citation phases of paper generation.
148
 
149
+ Be sure to provide the key for the model used for your runs, e.g.:
 
 
 
 
 
 
 
 
150
 
151
+ ```bash
152
+ export OPENAI_API_KEY="YOUR KEY HERE"
153
+ export S2_API_KEY="YOUR KEY HERE"
154
+ ```
155
 
156
+ #### OpenAlex API (Literature Search Alternative)
 
 
 
157
 
158
+ OpenAlex API can be used as an alternative if you do not have a Semantic Scholar API Key.
159
+ OpenAlex does not require API key.
 
 
160
 
161
+ ```bash
162
+ pip install pyalex
163
+ export OPENALEX_MAIL_ADDRESS="YOUR EMAIL ADDRESS"
164
+ ```
165
 
166
+ And specify `--engine openalex` when you execute the AI Scientist code.
167
 
168
+ Note that this is experimental for those who do not have a Semantic Scholar API Key.
169
+
170
+ ## Setting Up the Templates
171
+
172
+ This section provides instructions for setting up each of the three templates used in our paper. Before running The AI Scientist experiments, please ensure you have completed the setup steps for the templates you are interested in.
173
+
174
+ ### NanoGPT Template
175
+
176
+ **Description:** This template investigates transformer-based autoregressive next-token prediction tasks.
177
+
178
+ **Setup Steps:**
179
+
180
+ 1. **Prepare the data:**
181
+
182
+ ```bash
183
+ python data/enwik8/prepare.py
184
+ python data/shakespeare_char/prepare.py
185
+ python data/text8/prepare.py
186
+ ```
187
+
188
+ 2. **Create baseline runs (machine dependent):**
189
+
190
+ ```bash
191
+ # Set up NanoGPT baseline run
192
+ # NOTE: YOU MUST FIRST RUN THE PREPARE SCRIPTS ABOVE!
193
+ cd templates/nanoGPT
194
+ python experiment.py --out_dir run_0
195
+ python plot.py
196
+ ```
197
+
198
+ ### 2D Diffusion Template
199
+
200
+ **Description:** This template studies improving the performance of diffusion generative models on low-dimensional datasets.
201
+
202
+ **Setup Steps:**
203
+
204
+ 1. **Install dependencies:**
205
+
206
+ ```bash
207
+ # Set up 2D Diffusion
208
+ git clone https://github.com/gregversteeg/NPEET.git
209
+ cd NPEET
210
+ pip install .
211
+ pip install scikit-learn
212
+ ```
213
+
214
+ 2. **Create baseline runs:**
215
+
216
+ ```bash
217
+ # Set up 2D Diffusion baseline run
218
+ cd templates/2d_diffusion
219
+ python experiment.py --out_dir run_0
220
+ python plot.py
221
+ ```
222
+
223
+ ### Grokking Template
224
+
225
+ **Description:** This template investigates questions about generalization and learning speed in deep neural networks.
226
+
227
+ **Setup Steps:**
228
+
229
+ 1. **Install dependencies:**
230
+
231
+ ```bash
232
+ # Set up Grokking
233
+ pip install einops
234
+ ```
235
+
236
+ 2. **Create baseline runs:**
237
+
238
+ ```bash
239
+ # Set up Grokking baseline run
240
+ cd templates/grokking
241
+ python experiment.py --out_dir run_0
242
+ python plot.py
243
+ ```
244
+
245
+ ## Run AI Scientist Paper Generation Experiments
246
+
247
+ **Note:** Please ensure the setup steps above are completed before running these experiments.
248
+
249
+ ```bash
250
+ conda activate ai_scientist
251
+ # Run the paper generation.
252
+ python launch_scientist.py --model "gpt-4o-2024-05-13" --experiment nanoGPT_lite --num-ideas 2
253
+ python launch_scientist.py --model "claude-3-5-sonnet-20241022" --experiment nanoGPT_lite --num-ideas 2
254
+ ```
255
+
256
+ If you have more than one GPU, use the `--parallel` option to parallelize ideas across multiple GPUs.
257
+
258
+ ## Getting an LLM-Generated Paper Review
259
+
260
+ ```python
261
+ import openai
262
+ from ai_scientist.perform_review import load_paper, perform_review
263
+
264
+ client = openai.OpenAI()
265
+ model = "gpt-4o-2024-05-13"
266
+
267
+ # Load paper from PDF file (raw text)
268
+ paper_txt = load_paper("report.pdf")
269
+
270
+ # Get the review dictionary
271
+ review = perform_review(
272
+ paper_txt,
273
+ model,
274
+ client,
275
+ num_reflections=5,
276
+ num_fs_examples=1,
277
+ num_reviews_ensemble=5,
278
+ temperature=0.1,
279
+ )
280
+
281
+ # Inspect review results
282
+ review["Overall"] # Overall score (1-10)
283
+ review["Decision"] # 'Accept' or 'Reject'
284
+ review["Weaknesses"] # List of weaknesses (strings)
285
+ ```
286
+
287
+ To run batch analysis:
288
+
289
+ ```bash
290
+ cd review_iclr_bench
291
+ python iclr_analysis.py --num_reviews 500 --batch_size 100 --num_fs_examples 1 --num_reflections 5 --temperature 0.1 --num_reviews_ensemble 5
292
+ ```
293
+
294
+ ## Making Your Own Template
295
+
296
+ If there is an area of study you would like **The AI Scientist** to explore, it is straightforward to create your own templates. In general, follow the structure of the existing templates, which consist of:
297
+
298
+ - `experiment.py` β€” This is the main script where the core content is. It takes an argument `--out_dir`, which specifies where it should create the folder and save the relevant information from the run.
299
+ - `plot.py` β€” This script takes the information from the `run` folders and creates plots. The code should be clear and easy to edit.
300
+ - `prompt.json` β€” Put information about your template here.
301
+ - `seed_ideas.json` β€” Place example ideas here. You can also try to generate ideas without any examples and then pick the best one or two to put here.
302
+ - `latex/template.tex` β€” We recommend using our LaTeX folder but be sure to replace the pre-loaded citations with ones that you expect to be more relevant.
303
+
304
+ The key to making new templates work is matching the base filenames and output JSONs to the existing format; everything else is free to change.
305
+ You should also ensure that the `template.tex` file is updated to use the correct citation style / base plots for your template.
306
+
307
+ ### Community-Contributed Templates
308
+
309
+ We welcome community contributions in the form of new templates. While these are not maintained by us, we are delighted to highlight your templates to others. Below, we list community-contributed templates along with links to their pull requests (PRs):
310
+
311
+ - Infectious Disease Modeling (`seir`) - [PR #137](https://github.com/SakanaAI/AI-Scientist/pull/137)
312
+ - Image Classification with MobileNetV3 (`mobilenetV3`) - [PR #141](https://github.com/SakanaAI/AI-Scientist/pull/141)
313
+ - Sketch RNN (`sketch_rnn`) - [PR #143](https://github.com/SakanaAI/AI-Scientist/pull/143)
314
+ - AI in Quantum Chemistry (`MACE`) - [PR#157](https://github.com/SakanaAI/AI-Scientist/pull/157)
315
+ - Earthquake Prediction (`earthquake-prediction`) - [PR #167](https://github.com/SakanaAI/AI-Scientist/pull/167)
316
+ - Tensorial Radiance Fields (`tensorf`) - [PR #175](https://github.com/SakanaAI/AI-Scientist/pull/175)
317
+ - Large Language Model Steering / Probes (`probes`) - [PR #215](https://github.com/SakanaAI/AI-Scientist/pull/215)
318
+
319
+ *This section is reserved for community contributions. Please submit a pull request to add your template to the list! Please describe the template in the PR description, and also show examples of the generated papers.*
320
+
321
+ ## Template Resources
322
+
323
+ We provide three templates, which heavily use code from other repositories, credited below:
324
+
325
+ - **NanoGPT Template** uses code from [NanoGPT](https://github.com/karpathy/nanoGPT) and this [PR](https://github.com/karpathy/nanoGPT/pull/254).
326
+ - **2D Diffusion Template** uses code from [tiny-diffusion](https://github.com/tanelp/tiny-diffusion), [ema-pytorch](https://github.com/lucidrains/ema-pytorch), and [Datasaur](https://www.research.autodesk.com/publications/same-stats-different-graphs/).
327
+ - **Grokking Template** uses code from [Sea-Snell/grokking](https://github.com/Sea-Snell/grokking) and [danielmamay/grokking](https://github.com/danielmamay/grokking).
328
+
329
+ We would like to thank the developers of the open-source models and packages for their contributions and for making their work available.
330
+
331
+ ## Citing The AI Scientist
332
+
333
+ If you use **The AI Scientist** in your research, please cite it as follows:
334
+
335
+ ```
336
+ @article{lu2024aiscientist,
337
+ title={The {AI} {S}cientist: Towards Fully Automated Open-Ended Scientific Discovery},
338
+ author={Lu, Chris and Lu, Cong and Lange, Robert Tjarko and Foerster, Jakob and Clune, Jeff and Ha, David},
339
+ journal={arXiv preprint arXiv:2408.06292},
340
+ year={2024}
341
+ }
342
+ ```
343
+
344
+ ## Frequently Asked Questions
345
+
346
+ We recommend reading our paper first for any questions you have on The AI Scientist.
347
+
348
+ **Why am I missing files when running The AI Scientist?**
349
+
350
+ Ensure you have completed all the setup and preparation steps before the main experiment script.
351
+
352
+ **Why has a PDF or a review not been generated?**
353
+
354
+ The AI Scientist finishes an idea with a success rate that depends on the template, the base foundation model, and the complexity of the idea. We advise referring to our main paper. The highest success rates are observed with Claude Sonnet 3.5. Reviews are best done with GPT-4o; all other models have issues with positivity bias or failure to conform to required outputs.
355
+
356
+ **What is the cost of each idea generated?**
357
+
358
+ Typically less than $15 per paper with Claude Sonnet 3.5. We recommend DeepSeek Coder V2 for a much more cost-effective approach. A good place to look for new models is the [Aider leaderboard](https://aider.chat/docs/leaderboards/).
359
+
360
+ **How do I change the base conference format associated with the write-ups?**
361
+
362
+ Change the base `template.tex` files contained within each template.
363
+
364
+ **How do I run The AI Scientist for different subject fields?**
365
+
366
+ Please refer to the instructions for different templates. In this current iteration, this is restricted to ideas that can be expressed in code. However, lifting this restriction would represent exciting future work! :)
367
+
368
+ **How do I add support for a new foundation model?**
369
+
370
+ You may modify `ai_scientist/llm.py` to add support for a new foundation model. We do not advise using any model that is significantly weaker than GPT-4 level for **The AI Scientist**.
371
+
372
+ **Why do I need to run the baseline runs myself?**
373
+
374
+ These appear as `run_0` and should be run per machine you execute **The AI Scientist** on for accurate run-time comparisons due to hardware differences.
375
+
376
+ **What if I have problems accessing the Semantic Scholar API?**
377
+
378
+ We use the Semantic Scholar API to check ideas for novelty and collect citations for the paper write-up. You may be able to skip these phases if you don't have an API key or the API is slow to access.
379
+
380
+ ## Containerization
381
+
382
+ We include a [community-contributed](https://github.com/SakanaAI/AI-Scientist/pull/21) Docker image that may assist with your containerization efforts in `experimental/Dockerfile`.
383
+
384
+ You can use this image like this:
385
+
386
+ ```bash
387
+ # Endpoint Script
388
+ docker run -e OPENAI_API_KEY=$OPENAI_API_KEY -v `pwd`/templates:/app/AI-Scientist/templates <AI_SCIENTIST_IMAGE> \
389
+ --model gpt-4o-2024-05-13 \
390
+ --experiment 2d_diffusion \
391
+ --num-ideas 2
392
+ ```
393
+
394
+ ```bash
395
+ # Interactive
396
+ docker run -it -e OPENAI_API_KEY=$OPENAI_API_KEY \
397
+ --entrypoint /bin/bash \
398
+ <AI_SCIENTIST_IMAGE>
399
+ ```
400
+
401
+ ## βš–οΈ License & Responsible Use
402
+
403
+ This project is licensed under **The AI Scientist Source Code License** (a derivative of the Responsible AI License).
404
+
405
+ **Mandatory Disclosure:** By using this code, you are legally bound to clearly and prominently disclose the use of AI in any resulting scientific manuscripts or papers.
406
+
407
+ We recommend the following attribution in your paper's Abstract or Methods section:
408
+ > "This manuscript was autonomously generated using [The AI Scientist](https://github.com/SakanaAI/AI-Scientist)."
409
+
410
+ ## Star History
411
+
412
+ [![Star History Chart](https://api.star-history.com/svg?repos=SakanaAI/AI-Scientist&type=Date)](https://star-history.com/#SakanaAI/AI-Scientist&Date)