README.md

---
{}
---
# ERC Classifiers

This repository contains a model trained for multi-label classification of scientific papers in the ERC (European Research Council) context. The model predicts multiple categories for a paper, such as its research domain or topic, based on the abstract and title.

## Model Description

The model is based on **SPECTER** (a transformer-based model pre-trained on scientific literature), fine-tuned for **multi-label classification** on a dataset of scientific papers. The model classifies papers into several categories, which are defined by the **ERC categories**. The fine-tuned model is trained to predict these categories given the title and abstract of each paper.

### Preprocessing

The preprocessing pipeline involves:

1. **Data Loading**: Papers are loaded from a Parquet file containing the title, abstract, and their respective categories.
2. **Label Cleaning**: Labels (categories) are processed to remove any unnecessary information (like content within parentheses).
3. **Label Encoding**: Categories are transformed into a binary matrix using the **MultiLabelBinarizer** from scikit-learn. Each category corresponds to a column, and the value is `1` if the paper belongs to that category, `0` otherwise.
4. **Statistics and Visualization**: Basic statistics and visualizations, such as label distributions, are generated to help understand the dataset better.

### Training

The model is fine-tuned on the preprocessed dataset using the following setup:

* **Base Model**: The model uses the `allenai/specter` transformer as the base model for sequence classification.
* **Optimizer**: AdamW optimizer with a learning rate of `5e-5` is used.
* **Loss Function**: Binary Cross-Entropy with logits (`BCEWithLogitsLoss`) is employed, as the task is multi-label classification.
* **Epochs**: The model is trained for **5 epochs** with a batch size of 4.
* **Training Data**: The model is trained on a processed dataset stored in `train_ready.parquet`.

### Evaluation

The model is evaluated using both **single-label** and **multi-label** metrics:

#### Single-Label Evaluation

* **Accuracy**: The accuracy is measured by checking how often the true label appears in the predicted labels.
* **Precision, Recall, F1**: These metrics are calculated for each class and averaged for the entire dataset.

#### Multi-Label Evaluation

* **Micro and Macro Metrics**: Precision, recall, and F1 scores are computed using both micro-averaging (overall performance) and macro-averaging (performance per label).
* **Label Frequency Plot**: A plot showing the frequency distribution of labels in the test set.
* **Top and Bottom F1 Plot**: A plot visualizing the top and bottom labels based on their F1 scores.

## Dataset

The dataset consists of scientific papers, each with the following columns:

* **title**: The title of the paper.
* **abstract**: The abstract of the paper.
* **label**: A list of categories (labels) assigned to the paper.

The dataset is preprocessed and stored in a `train_ready.parquet` file.

## Files

* `config.json`: Model configuration file.
* `model.safetensors`: Saved fine-tuned model weights.
* `tokenizer.json`: Tokenizer configuration for the fine-tuned model.
* `tokenizer_config.json`: Tokenizer settings.
* `special_tokens_map.json`: Special tokens used by the tokenizer.
* `vocab.txt`: Vocabulary file for the fine-tuned tokenizer.

## Usage

To use the model, follow these steps:

1. **Install Dependencies**:

   ```bash
   pip install transformers torch datasets
   ```

2. **Load the Model and Tokenizer**:

   ```python
   from transformers import AutoModelForSequenceClassification, AutoTokenizer

   model_name = "SIRIS-Lab/erc-classifiers"

   # Load fine-tuned model and tokenizer
   model = AutoModelForSequenceClassification.from_pretrained(model_name)
   tokenizer = AutoTokenizer.from_pretrained(model_name)
   ```

3. **Use the Model for Prediction**:

   ```python
   # Example paper title and abstract
   text = "Example title and abstract of a scientific paper."

   # Tokenize the input text
   inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)

   # Make predictions
   with torch.no_grad():
       logits = model(**inputs).logits

   # Apply sigmoid activation to get probabilities
   probabilities = torch.sigmoid(logits)

   # Get predicted labels (threshold at 0.5)
   predicted_labels = (probabilities >= 0.5).long().cpu().numpy()
   print(predicted_labels)
   ```

## Conclusion

This model provides an efficient solution for classifying scientific papers into multiple categories based on their content. It uses state-of-the-art transformer-based techniques and is fine-tuned on a real-world dataset of ERC-related scientific papers.