Duplicate from andreabac3/Fauno-Italian-LLM-7B

Co-authored-by: Andrea Bacciu <andreabac3@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,125 @@
+---
+license: gpl-3.0
+datasets:
+- andreabac3/MedQuaAD-Italian-Fauno-Baize
+- andreabac3/StackOverflow-Italian-Fauno-Baize
+- andreabac3/Quora-Italian-Fauno-Baize
+- teelinsan/camoscio_cleaned
+language:
+- it
+- en
+tags:
+- large language model
+- italian large language model
+- baize
+- 'llama '
+- italian
+---
+# Fauno - Italian LLM 
+
+![image](fauno.drawio.png)
+
+Get ready to meet Fauno -  the Italian language model crafted by the [RSTLess Research Group](https://rstless-lab.netlify.app/) from the Sapienza University of Rome.
+
+The talented research team behind Fauno includes [Andrea Bacciu](https://andreabac3.github.io/), [Dr. Giovanni Trappolini](https://sites.google.com/view/giovannitrappolini), [Andrea Santilli](https://www.santilli.xyz/), and [Professor Fabrizio Silvestri](https://sites.google.com/diag.uniroma1.it/fabriziosilvestri/home).
+
+Fauno represents a cutting-edge development in open-source Italian Large Language Modeling. It's trained on extensive Italian synthetic datasets, encompassing a wide range of fields such as medical data 🩺, technical content from Stack Overflow 💻, Quora discussions 💬, and Alpaca data 🦙 translated into Italian.
+
+Hence, our model is able to answer to your questions in Italian 🙋, fix your buggy code 🐛 and understand a minimum of medical literature 💊.
+
+## The 🇮🇹 open-source version of chatGPT!
+Discover the capabilities of Fauno and experience the evolution of Italian language models for yourself.
+![demo](screenshot_demo.png)
+
+### Why Fauno?
+We started with a model called Baize, named after a legendary creature from Chinese literature. Continuing along this thematic line, we developed our Italian model based on Baize and named it Fauno, inspired by an iconic figure from Roman mythology. This choice underlines the link between the two models, while maintaining a distinctive identity rooted in Italian culture.
+
+# Did you know that you can run Fauno on Colab base?
+Follow this link to access a Colab notebook with our 7B version! <a target="_blank" href="https://colab.research.google.com/drive/1AepJVWS-qU910zyq-Zi7wWFQ5tthVzUe">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+
+## 🔎 Model's details
+Fauno is a fine-tuned version of the LoRa weights of [Baize](https://github.com/project-baize/baize-chatbot), that is an improved version of [LLama](https://github.com/facebookresearch/llama).
+
+We translated and cleaned the data of Baize, and then we fine-tuned the 7b model using a single RTX A6000 (48GB of VRAM) with 19 hours for one epoch.
+
+- 13B: https://huggingface.co/andreabac3/Fauno-Italian-LLM-13B
+
+Fauno 30B and 65B are coming soon! 
+
+## Model initialization
+```python
+from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig
+from peft import PeftModel
+
+tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")
+model = LlamaForCausalLM.from_pretrained(
+    "decapoda-research/llama-7b-hf",
+    load_in_8bit=True,
+    device_map="auto",
+)
+model = PeftModel.from_pretrained(model, "andreabac3/Fauno-Italian-LLM-7B")
+model.eval()
+```
+
+## Inference
+```python
+def evaluate(question: str) -> str:
+    prompt = f"The conversation between human and AI assistant.\n[|Human|] {question}.\n[|AI|] "
+    inputs = tokenizer(prompt, return_tensors="pt")
+    input_ids = inputs["input_ids"].cuda()
+    generation_output = model.generate(
+        input_ids=input_ids,
+        generation_config=generation_config,
+        return_dict_in_generate=True,
+        output_scores=True,
+        max_new_tokens=256
+    )
+    output = tokenizer.decode(generation_output.sequences[0]).split("[|AI|]")[1]
+    return output
+
+your_question: str = "Qual'è il significato della vita?"
+print(evaluate(your_question))
+```
+
+### Output 
+```
+Il senso della vita è una domanda che molte persone hanno cercato di rispondere per secoli.
+Alla fine, il senso della vita è soggettivo e varia da persona a persona.
+Alcune persone credono che il senso della vita sia trovare la felicità, mentre altre credono che sia raggiungere i propri obiettivi o aiutare gli altri.
+Alla fine, il senso della vita è determinato dall'individuo e dai loro valori e credenze.
+In definitiva, il senso della vita è qualcosa che ognuno deve trovare da solo.
+Non c'è una risposta giusta o sbagliata, poiché ogni persona ha le proprie convinzioni e credenze.
+La ricerca del senso della vita può essere un viaggio lungo e difficile, ma vale la pena perseguire.
+```
+# 📖 Cite our work
+
+To use our translated dataset and model weights in your research, remember to cite our work.
+```bibtex
+@misc{fauno,
+  author = {Andrea Bacciu, Giovanni Trappolini, Andrea Santilli, Fabrizio Silvestri},
+  title = {Fauno: The Italian Large Language Model that will leave you senza parole!},
+  year = {2023},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/andreabac3/Fauno-Italian-LLM}},
+}
+```
+
+
+## 🔑 License
+This project is a derivative of Baize, and we adhere to the licensing constraints imposed by both Baize's creators and the authors of LLama.
+
+## ⚠️ Hallucinations
+It is important to remark that current generation models are prone to the problem of hallucinations. So we advise you not to take their answers seriously.
+
+## 👏 Acknowledgement
+- LLama - Meta AI: https://github.com/facebookresearch/llama
+- Baize: https://github.com/project-baize/baize-chatbot
+- Standford Alpaca: https://github.com/tatsu-lab/stanford_alpaca
+- Camoscio: https://github.com/teelinsan/camoscio
+
+#### Image Credits 
+- llama image: https://next14.com/en/nextnews-7-march-a-new-language-model-for-meta-bing-ai-on-windows-and-the-first-tokenized-real-estate-sales/
+- Fauno logo: https://www.flaticon.com/free-icon/faun_7931635?term=faun&page=1&position=1&origin=tag&related_id=7931635

adapter_config.json

@@ -0,0 +1,23 @@
+{
+  "base_model_name_or_path": "decapoda-research/llama-7b-hf",
+  "bias": "none",
+  "enable_lora": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "lora_alpha": 16,
+  "lora_dropout": 0.05,
+  "merge_weights": false,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "target_modules": [
+    "q_proj",
+    "k_proj",
+    "v_proj",
+    "down_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM"
+}

adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:572f9f3c2c6eb0a8d6918be107ebff1c51da3ba4f2757d80c6c5bcff0dd8561a
+size 71703053