DonlerFC/MoritzLaurer-mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-finetuned-eHighway

mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-finetuned

This model was developed as part of a master's thesis at the Technical University of Darmstadt by Daniel Strohmeyer LinkedIn. It is based on MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 and fine-tuned for zero-shot topic classification of sentence segments from press releases and technical articles related to the eHighway project.

DISCLAIMER:

The domain of the fine-tuning is very restrictive, so there is no guarantee for the quality of off-domain classification.

Methodology

A transfer learning approach based on a pre-trained BERT-NLI model was applied for classifying sentence segments. The procedure follows Laurer et al. (2023, 2024), who demonstrated zero-shot classification and fine-tuning for Natural Language Inference (NLI). The published Python notebooks served as a methodological reference.

The goal was to adapt the base model to the specific task of topic classification in online reporting without requiring large amounts of manually annotated data. This approach combines the benefits of machine learning with NLI to mitigate the limitations of small supervised datasets.

Data

602 sentence segments from a custom corpus of relevant press releases and technical articles were selected for fine-tuning. These were randomly split into training (227) and test (375) samples (60–40 split).

To compensate for the limited dataset, a data augmentation approach following Ubani et al., 2023 was used. Additional synthetic examples were generated using a generative language model (ChatGPT) via targeted prompting. Only high-quality and thematically consistent segments were included in the training set.

After augmentation, the training dataset contained 1,135 positive segments evenly distributed across all topic classes. Similarly, an equal number of negative (Non-Entailment) segments were created by randomly assigning non-matching labels.

The test set contains 375 segments per class, with over 90% being non-matching examples.

Base Model

Fine-tuning was performed on the multilingual pre-trained model
MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7.

Training Process

Inputs were prepared as premise-hypothesis pairs:

• Premise: the sentence segment to classify
• Hypothesis: a statement corresponding to the topic class

Example:

Premise: "Residents are disturbed by the test track."
Hypothesis: "This sentence is about project acceptance."

Training parameters followed standard NLI recommendations:

Epoch	Loss	Accuracy	F1	Precision	Recall
Before	---	0.1170	0.1638	0.1690	0.1600
1	0.818	0.3979	0.3939	0.3864	0.4021
2	0.369	0.4536	0.4731	0.4444	0.5046
3	0.206	0.4775	0.4950	0.4504	0.5485

Other training settings:

Parameter	Symbol	Value
Learning rate	η	2 × 10⁻⁵
Training batch size	B_train	32
Evaluation batch size	B_eval	80
Epochs	E	3
Warmup proportion	w	0.2

Training was conducted on a NVIDIA A100 GPU in Google Colab. Progress and metrics were monitored, and the final model weights are ready for zero-shot topic classification.

---

Usage

The model is compatible with the transformers zero-shot pipeline:

from transformers import pipeline

classifier = pipeline(
    "zero-shot-classification",
    model="DonlerFC/MoritzLaurer-mDeBERTa-v3-base-xnli-multilingual-nli-2mil7-finetuned"
)

text = "Kritik an der Teststrecke kommt auch vereinzelt von Anwohnern."
labels = ["The text discusses costs, funding, or financial aspects of the project.", "The text explains the motivation behind the project.", "The text reflects the opinions of social groups, politicians, or associations regarding the project."]
result = classifier(text, candidate_labels=labels)
print(result)

Fine-Tuning Data and Notebook

All training and test datasets used for fine-tuning this model are provided in the finetuning_data folder of this repository.

• Training set: train.csv – contains premise-hypothesis pairs used for model fine-tuning.
• Test set: test.csv – contains labeled data for evaluation of the fine-tuned model.

Additionally, the folder includes a Jupyter Notebook Finetuning_NLI-MortizLaurer.ipynb which demonstrates how to:

1. Load the datasets directly from Hugging Face.
2. Prepare the data for Zero-Shot Classification.
3. Run the fine-tuned model on the provided datasets.

Results of Article Classification

The results of the news and journal articles analyses are published in the Folder. For usage instructions, please refer to the appendix of the corresponding master’s thesis.

The full texts, including the individual segment texts, cannot be published due to copyright restrictions of the press and scientific articles.

License

This model is released under the terms of the MIT License.
For details, see: LICENSE.txt