Hugging Face's logo

knowledgator
/
gliner-multitask-v1.0

Token Classification
GLiNER
PyTorch
English
NER
information extraction
relation extraction
summarization
sentiment extraction
question-answering
Model card Files
xet
Community
3
Ihor commited on
Commit
36fabc4
·
verified ·
1 Parent(s): cc2cc6e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +45 -42
README.md CHANGED
@@ -74,14 +74,23 @@ for entity in entities:
74
  | | mit-movie | 61.29% | 52.59% | 56.60% | 0.5660 |
75
  | | mit-restaurant | 50.65% | 38.13% | 43.51% | 0.4351 |
76
  | | **Average** | | | | **0.6276** |
77
- | knowledgator/gliner-multitask-v1.0 | CrossNER_AI | 51.00% | 51.11% | 51.05% | 0.5105 |
78
- | | CrossNER_literature | 72.65% | 65.62% | 68.96% | 0.6896 |
79
- | | CrossNER_music | 74.91% | 73.70% | 74.30% | 0.7430 |
80
- | | CrossNER_politics | 78.84% | 77.71% | 78.27% | 0.7827 |
81
- | | CrossNER_science | 69.20% | 65.48% | 67.29% | 0.6729 |
82
- | | mit-movie | 61.29% | 52.59% | 56.60% | 0.5660 |
83
- | | mit-restaurant | 50.65% | 38.13% | 43.51% | 0.4351 |
84
- | | **Average** | | | | **0.6276** |
 
 
 
 
 
 
 
 
 
85
 
86
  ---
87
  **How to use for relation extraction:**
@@ -314,40 +323,34 @@ for label in classes:
314
 
315
  Our multitask model demonstrates comparable performance on different zero-shot benchmarks to dedicated models to NER task (all labels were lowecased in this testing):
316
 
317
- | Model | Dataset | Precision | Recall | F1 Score | F1 Score (Decimal) |
318
- |------------------------------------|--------------------|-----------|--------|----------|--------------------|
319
- | numind/NuNER_Zero-span | CrossNER_AI | 63.82% | 56.82% | 60.12% | 0.6012 |
320
- | | CrossNER_literature| 73.53% | 58.06% | 64.89% | 0.6489 |
321
- | | CrossNER_music | 72.69% | 67.40% | 69.95% | 0.6995 |
322
- | | CrossNER_politics | 77.28% | 68.69% | 72.73% | 0.7273 |
323
- | | CrossNER_science | 70.08% | 63.12% | 66.42% | 0.6642 |
324
- | | mit-movie | 63.00% | 48.88% | 55.05% | 0.5505 |
325
- | | mit-restaurant | 54.81% | 37.62% | 44.62% | 0.4462 |
326
- | | **Average** | | | | **0.6196** |
327
- | knowledgator/gliner-multitask-v0.5 | CrossNER_AI | 51.00% | 51.11% | 51.05% | 0.5105 |
328
- | | CrossNER_literature | 72.65% | 65.62% | 68.96% | 0.6896 |
329
- | | CrossNER_music | 74.91% | 73.70% | 74.30% | 0.7430 |
330
- | | CrossNER_politics | 78.84% | 77.71% | 78.27% | 0.7827 |
331
- | | CrossNER_science | 69.20% | 65.48% | 67.29% | 0.6729 |
332
- | | mit-movie | 61.29% | 52.59% | 56.60% | 0.5660 |
333
- | | mit-restaurant | 50.65% | 38.13% | 43.51% | 0.4351 |
334
- | | **Average** | | | | **0.6276** |
335
- | urchade/gliner_large-v2.1 | CrossNER_AI | 54.98% | 52.00% | 53.45% | 0.5345 |
336
- | | CrossNER_literature| 59.33% | 56.47% | 57.87% | 0.5787 |
337
- | | CrossNER_music | 67.39% | 66.77% | 67.08% | 0.6708 |
338
- | | CrossNER_politics | 66.07% | 63.76% | 64.90% | 0.6490 |
339
- | | CrossNER_science | 61.45% | 62.56% | 62.00% | 0.6200 |
340
- | | mit-movie | 55.94% | 47.36% | 51.29% | 0.5129 |
341
- | | mit-restaurant | 53.34% | 40.83% | 46.25% | 0.4625 |
342
- | | **Average** | | | | **0.5754** |
343
- | EmergentMethods/gliner_large_news-v2.1| CrossNER_AI | 59.60% | 54.55% | 56.96% | 0.5696 |
344
- | | CrossNER_literature| 65.41% | 56.16% | 60.44% | 0.6044 |
345
- | | CrossNER_music | 67.47% | 63.08% | 65.20% | 0.6520 |
346
- | | CrossNER_politics | 66.05% | 60.07% | 62.92% | 0.6292 |
347
- | | CrossNER_science | 68.44% | 63.57% | 65.92% | 0.6592 |
348
- | | mit-movie | 65.85% | 49.59% | 56.57% | 0.5657 |
349
- | | mit-restaurant | 54.71% | 35.94% | 43.38% | 0.4338 |
350
- | | **Average** | | | | **0.5876** |
351
 
352
 
353
  ### Join Our Discord
 
74
  | | mit-movie | 61.29% | 52.59% | 56.60% | 0.5660 |
75
  | | mit-restaurant | 50.65% | 38.13% | 43.51% | 0.4351 |
76
  | | **Average** | | | | **0.6276** |
77
+ | knowledgator/gliner-multitask-v1.0 | CrossNER_AI | 67.15% | 56.10% | 61.13% | 0.6113 |
78
+ | | CrossNER_literature | 71.60% | 64.74% | 68.00% | 0.6800 |
79
+ | | CrossNER_music | 73.57% | 69.29% | 71.36% | 0.7136 |
80
+ | | CrossNER_politics | 77.54% | 76.52% | 77.03% | 0.7703 |
81
+ | | CrossNER_science | 74.54% | 66.00% | 70.01% | 0.7001 |
82
+ | | mit-movie | 61.86% | 42.02% | 50.04% | 0.5004 |
83
+ | | mit-restaurant | 58.87% | 36.67% | 45.19% | 0.4519 |
84
+ | | **Average** | | | | **0.6325** |
85
+ | knowledgator/gliner-llama-multitask-1B-v1.0 | CrossNER_AI | 63.24% | 55.60% | 59.17% | 0.5917 |
86
+ | | CrossNER_literature | 69.74% | 60.10% | 64.56% | 0.6456 |
87
+ | | CrossNER_music | 74.03% | 67.22% | 70.46% | 0.7046 |
88
+ | | CrossNER_politics | 76.96% | 71.64% | 74.20% | 0.7420 |
89
+ | | CrossNER_science | 73.79% | 63.73% | 68.39% | 0.6839 |
90
+ | | mit-movie | 56.89% | 46.70% | 51.30% | 0.5130 |
91
+ | | mit-restaurant | 48.45% | 38.13% | 42.67% | 0.4267 |
92
+ | | **Average** | | | | **0.6153** |
93
+
94
 
95
  ---
96
  **How to use for relation extraction:**
 
323
 
324
  Our multitask model demonstrates comparable performance on different zero-shot benchmarks to dedicated models to NER task (all labels were lowecased in this testing):
325
 
326
+ | Dataset | Precision | Recall | F1 Score | F1 Score (Decimal) |
327
+ |------------------------|-----------|--------|----------|--------------------|
328
+ | ACE 2004 | 53.25% | 23.20% | 32.32% | 0.3232 |
329
+ | ACE 2005 | 43.25% | 18.00% | 25.42% | 0.2542 |
330
+ | AnatEM | 51.75% | 25.98% | 34.59% | 0.3459 |
331
+ | Broad Tweet Corpus | 69.54% | 72.50% | 70.99% | 0.7099 |
332
+ | CoNLL 2003 | 68.33% | 68.43% | 68.38% | 0.6838 |
333
+ | CrossNER_AI | 67.15% | 56.10% | 61.13% | 0.6113 |
334
+ | CrossNER_literature | 71.60% | 64.74% | 68.00% | 0.6800 |
335
+ | CrossNER_music | 73.57% | 69.29% | 71.36% | 0.7136 |
336
+ | CrossNER_politics | 77.54% | 76.52% | 77.03% | 0.7703 |
337
+ | CrossNER_science | 74.54% | 66.00% | 70.01% | 0.7001 |
338
+ | FabNER | 69.28% | 62.62% | 65.78% | 0.6578 |
339
+ | FindVehicle | 49.75% | 51.25% | 50.49% | 0.5049 |
340
+ | GENIA_NER | 60.98% | 46.91% | 53.03% | 0.5303 |
341
+ | HarveyNER | 24.27% | 35.66% | 28.88% | 0.2888 |
342
+ | MultiNERD | 54.33% | 89.34% | 67.57% | 0.6757 |
343
+ | Ontonotes | 27.26% | 36.64% | 31.26% | 0.3126 |
344
+ | PolyglotNER | 33.54% | 64.29% | 44.08% | 0.4408 |
345
+ | TweetNER7 | 44.77% | 38.67% | 41.50% | 0.4150 |
346
+ | WikiANN en | 56.33% | 57.09% | 56.71% | 0.5671 |
347
+ | WikiNeural | 71.70% | 86.60% | 78.45% | 0.7845 |
348
+ | bc2gm | 64.71% | 51.68% | 57.47% | 0.5747 |
349
+ | bc4chemd | 69.24% | 50.08% | 58.12% | 0.5812 |
350
+ | bc5cdr | 79.22% | 69.19% | 73.87% | 0.7387 |
351
+ | mit-movie | 61.86% | 42.02% | 50.04% | 0.5004 |
352
+ | mit-restaurant | 58.87% | 36.67% | 45.19% | 0.4519 |
353
+ | ncbi | 68.72% | 54.86% | 61.01% | 0.6101 |
 
 
 
 
 
 
354
 
355
 
356
  ### Join Our Discord