XLM-R_WR

Model Description

XLM-R_WR is a Telugu sentiment classification model built on XLM-RoBERTa (XLM-R), a general-purpose multilingual Transformer model developed by Facebook AI. XLM-R is designed to improve cross-lingual understanding through large-scale pretraining on a diverse multilingual corpus.

The base model is pretrained on approximately 2.5 TB of filtered Common Crawl data spanning 100+ languages, including Telugu. Unlike mBERT, XLM-R is trained exclusively using the Masked Language Modeling (MLM) objective, without the Next Sentence Prediction (NSP) task, enabling stronger contextual representations.

The suffix WR denotes With Rationale supervision. This model is fine-tuned using both sentiment labels and human-annotated rationales, allowing the model to align predictions and explanations with human-identified evidence spans.

---

Pretraining Details

• Pretraining corpus: Filtered Common Crawl (≈2.5 TB, 100+ languages)
• Training objective:
  • Masked Language Modeling (MLM)
• Next Sentence Prediction: Not used
• Language coverage: Telugu included, but not exclusively targeted

---

Training Data

• Fine-tuning dataset: Telugu-Dataset
• Task: Sentiment classification
• Supervision type: Label + rationale supervision
• Rationales: Token-level human-annotated evidence spans

---

Rationale Supervision

During fine-tuning, human-provided rationales are used to guide model learning. In addition to the standard classification loss, an auxiliary rationale loss encourages the model’s attention or explanation scores to align with annotated rationale tokens.

This supervision improves:

• Alignment between model explanations and human judgment
• Plausibility of generated explanations
• Interpretability of sentiment predictions

---

Intended Use

This model is intended for:

• Explainable Telugu sentiment classification
• Rationale-supervised learning experiments
• Cross-lingual explainability research
• Comparative studies against label-only (WOR) baselines

The model is suitable for scenarios where both predictive performance and explanation quality are important.

---

Performance Characteristics

Compared to label-only training, rationale supervision typically improves explanation plausibility, while maintaining competitive sentiment classification performance.

Strengths

• Human-aligned explanations through rationale supervision
• Strong cross-lingual representations from large-scale pretraining
• Suitable for explainable AI benchmarking

Limitations

• Requires human-annotated rationales, increasing annotation cost
• Classification performance gains may be limited relative to WOR models
• Not explicitly optimized for Telugu morphology or syntax

---

Use in Explainability Evaluation

XLM-R_WR is designed for evaluation with explanation frameworks such as FERRET, enabling:

• Faithfulness evaluation: How well explanations support model predictions
• Plausibility evaluation: How closely explanations align with human rationales

This makes the model well-suited for rigorous explainability analysis in low-resource Telugu NLP.

---

References

• Conneau et al., 2019
• Hedderich et al., 2021
• Kulkarni et al., 2021
• Joshi, 2022
• Das et al., 2022
• Rajalakshmi et al., 2023