README.md

---
library_name: transformers
tags:
- Summarization
- Longformer
- LED
- Fine-Tuned
- Abstractive
- Scientific
- seq2seq
- transformers
- english
- attention
- text-processing
- NLP
- beam-search
anguage: null
language:
- en
metrics:
- rouge
- precision
pipeline_tag: summarization
---


# Model Card for Model ID

This model is a fine-tuned version of the Longformer Encoder-Decoder (LED)- "allenai/led-base-16384", specifically tailored for [describe the task, e.g., "summarizing scientific articles"]. 
LED, originally designed for long document tasks, leverages a sparse attention mechanism to handle much longer contexts than standard transformer models. 
Our version extends its capabilities to efficiently summarize texts with high fidelity and relevance.

This Model can handle a total input token of upto "16000" tokens which is larger than most of the models present out there.

# Base code is specified below, try that out, API example wont work as tokenizer of allenai/led-base-16384 is used!

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

This model is intended for use in scenarios where understanding and condensing long texts is necessary. It is particularly useful for:

Academic researchers needing summaries of lengthy papers.
Professionals who require digests of extensive reports.
Content creators looking for concise versions of long articles.

Please note: This model will work for any summarization process to generate abstractive summary, just keep in mind to get the best results for a particular domain,
you need to train the model on your specific dataset if for a specific domain.



## Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The only limitation you might face is, to get the best results, you will have to fine-tune it. LOL!!



## How to Get Started with the Model

Use the code below to get started with the model.

---
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer <br>
import torch <br>
<b><i>#Load the model and tokenizer</i></b><br>
model = AutoModelForSeq2SeqLM.from_pretrained("yashrane2904/LED_Finetuned").to("cuda")<br>
tokenizer = AutoTokenizer.from_pretrained("allenai/led-base-16384") # Since it is a fine-tuned version of led-base-16348, we use the same tokenizer as that model used<br>
LONG_ARTICLE = "Your long text goes here..."<br>
<b><i>#Tokenize the input article</i></b><br>
input_ids = tokenizer(LONG_ARTICLE, return_tensors="pt").input_ids.to("cuda")<br>
global_attention_mask = torch.zeros_like(input_ids)<br>
global_attention_mask[:, 0] = 1<br>
<b><i>#Generate summaries</i></b><br>
sequences_tensor = model.generate(input_ids, global_attention_mask=global_attention_mask, num_beams=10, num_beam_groups=1,repetition_penalty=6.0,max_length=600,min_length=350,temperature=1.5)<br>
sequences = sequences_tensor.tolist()  # Convert Tensor to list of token IDs<br>
summary = tokenizer.batch_decode(sequences, skip_special_tokens=True)  # Decode token IDs into text<br>
<b><i>#Print the generated summary</i></b><br>
print(summary)
---

## Feel free to play around with the hyperparameters in the generate, or some other parameters to include for experimentation purpose.


## Model Card Authors  & Citation

@misc {yash_rane_2024,<br>
	author       = { {Yash Rane} },<br>
	title        = { LED_Finetuned (Revision f480282) },<br>
	year         = 2024,<br>
	url          = { https://huggingface.co/yashrane2904/LED_Finetuned },<br>
	doi          = { 10.57967/hf/2074 },<br>
	publisher    = { Hugging Face }<br>
}<br>