README.md

---
license: apache-2.0
tags:
- text-generation
- distilgpt2
- fine-tuned
- literature
- german-literature # Even if fine-tuned on English, it's German literature
- werther
- goethe
- creative-writing
language: en # Assuming fine-tuned on the English translation
datasets:
- werther_cleaned_final.txt # Indicate the source of your custom dataset
metrics:
- loss # The primary metric optimized during training
model_name: DistilGPT2 Werther Fine-tuned
base_model: distilgpt2
---

# distilgpt2-werther-finetuned

This is a [DistilGPT2](https://huggingface.co/distilgpt2) model fine-tuned on "The Sorrows of Young Werther" by Johann Wolfgang von Goethe (specifically, on an English translation of the novel).

The goal of this fine-tuning project was to explore how a smaller language model like DistilGPT2 could capture the unique melancholic tone, vocabulary, and stylistic nuances of Werther's letters.

## Model Description

The model began as a pre-trained DistilGPT2, a distilled version of GPT-2, known for its efficiency and good performance in text generation tasks. It was then adapted to the specific literary domain of Goethe's *Werther* through fine-tuning.

* **Base Model:** `distilgpt2`
* **Fine-tuned on:** "The Sorrows of Young Werther" (English translation)
* **Task:** Causal Language Modeling (text generation)
* **Output:** Generates text in a style reminiscent of the novel, often picking up on key phrases and emotional vocabulary.

## How to Use

You can easily use this model with the Hugging Face `transformers` library:

```python
from transformers import pipeline

# Replace 'your-username' with your actual Hugging Face username
generator = pipeline("text-generation", model="your-username/distilgpt2-werther-finetuned")

# Example usage:
prompt = "How happy I am that I am gone!"
generated_text = generator(prompt, max_new_tokens=100, num_return_sequences=1, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(generated_text[0]['generated_text'])

prompt = "My soul yearns for"
generated_text = generator(prompt, max_new_tokens=80, num_return_sequences=1, do_sample=True, temperature=0.6)
print(generated_text[0]['generated_text'])
```

## Training Details

Hardware: Local NVIDIA GeForce RTX 4070 Laptop GPU (or similar)  
Fine-tuning script: finetune_werther.py (available in the repository along with all python scripts to fine-tune distilgpt2)  
Training epochs: 5 
Dataset Size (approximately): ~57,987 tokens (from one novel)  
Block size: 512 tokens  
Training framework: Hugging Face transformers library with PyTorch backend.  

## Limitations and Bias

Repetitive Output: Due to the relatively small size of the fine-tuning dataset (a single novel) and the base model's architecture, the generated text can often become repetitive or loop on certain phrases. This is a common characteristic of smaller models fine-tuned on limited domain-specific data.  

Lack of Long-Range Coherence: The model may struggle to maintain a coherent narrative or theme over longer generated passages.  

Bias: The model will reflect any biases present in the original "The Sorrows of Young Werther" text. It will also reflect the melancholic, romantic, and somewhat obsessive tone of the original work.  

Limited Knowledge: It only knows what was present in its pre-training data and what it learned from Werther. It does not have general world knowledge.  

## Future Work

Fine-tuning on a larger corpus of 18th-century romantic literature (in English) to improve fluency and reduce repetition.  
Experimenting with different generation parameters (temperature, top_k, top_p) for more varied outputs.  

## Acknowledgements

Hugging Face Transformers library for providing the tools for fine-tuning.  
Johann Wolfgang von Goethe for "The Sorrows of Young Werther."  
The specific translation used for the fine-tuning:  

Ebook: https://www.gutenberg.org/ebooks/2527

Author: Goethe, Johann Wolfgang von, 1749-1832  
Translator: Boylan, R. Dillon (Richard Dillon), 1805?-1888  
Uniform Title: Die Leiden des jungen Werther. English  
Title: The Sorrows of Young Werther  
Note: Wikipedia page about this book: https://en.wikipedia.org/wiki/The_Sorrows_of_Young_Werther  
Note: Translation of: Die Leiden des jungen Werther.  
Note: Reading ease score: 77.4 (7th grade). Fairly easy to read.  
Credits: Produced by Michael Potter, Irene Potter, and David Widger  
Language: English  
LoC Class: PT: Language and Literatures: Germanic, Scandinavian, and Icelandic literatures  
Subject: Germany -- Social life and customs -- Fiction  
Subject: Unrequited love -- Fiction  
Subject: Young men -- Germany -- Fiction  
Category: Text  
EBook-No.: 2527  
Release Date: Feb 1, 2001  
Most Recently Updated: Aug 12, 2024  
Copyright Status: Public domain in the USA.  
Downloads: 5123 downloads in the last 30 days.  
Project Gutenberg eBooks are always free!

2025-06-11 11.41 AM