readme: update model card

README.md

@@ -9,9 +9,9 @@ language:
 - nl
 ---
 
-# hmByT5
+# hmByT5 - Preliminary Language Models
 
-Upcoming Historic Multilingual ByT5 Model. It covers the following languages:
+Preliminary Historic Multilingual and Monolingual ByT5 Models. Following languages are currently covered:
 
 * English (British Library Corpus - Books)
 * German (Europeana Newspaper)
@@ -20,6 +20,8 @@ Upcoming Historic Multilingual ByT5 Model. It covers the following languages:
 * Swedish (Europeana Newspaper)
 * Dutch (Delpher Corpus)
 
+More details can be found in [our GitHub repository](https://github.com/stefan-it/hmByT5).
+
 # Pretraining
 
 We pretrain hmByT5 on a v3-32 TPU Pod. Details about the training can be found
@@ -27,19 +29,18 @@ We pretrain hmByT5 on a v3-32 TPU Pod. Details about the training can be found
 
 # Evaluation on Downstream Tasks (NER)
 
-We use Flair to fine-tune hmByT5 on HIPE-2022 data. Details about the fine-tuning can be found
-[here](https://github.com/stefan-it/hmByT5/tree/main/bench).
+We evaluated the hmByT5 model that was pretrained on English AjMC corpus for 200k steps:
 
-# **New**: Logbook
+| Hyper-param Configuration                | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Avg.         |
+|------------------------------------------|-------|-------|-------|-------|-------|--------------|
+| `wsFalse-bs4-e10-lr0.00016-poolingfirst` | 83.80 | 84.78 | 83.74 | 83.35 | 84.37 | 84.01 ± 0.50 |
+| `wsFalse-bs4-e10-lr0.00015-poolingfirst` | 84.67 | 82.69 | 83.92 | 84.53 | 82.90 | 83.74 ± 0.82 |
+| `wsFalse-bs8-e10-lr0.00016-poolingfirst` | 82.12 | 83.82 | 83.37 | 83.00 | 83.70 | 83.20 ± 0.61 |
+| `wsFalse-bs8-e10-lr0.00015-poolingfirst` | 83.45 | 82.83 | 84.15 | 81.76 | 83.78 | 83.19 ± 0.84 |
 
-* 07.04.2022: Pretraining for 200k steps on the English corpus finished without crashes! TensorBoard logs can be found
-              on the [Model Hub](https://huggingface.co/stefan-it/byt5-small-historic-multilingual/tensorboard). We
-              also uploaded all checkpoints (we checkpoint every 25k steps)
-              [here](https://huggingface.co/stefan-it/byt5-small-historic-multilingual). Fine-Tuning experiments (NER)
-              on the English part of AjMC corpus from HIPE-2022 are running.
-* 03.04.2022: We start ByT5 pretraining with official ByT5 implementation on a v3-32 TPU Pod - thankfully provided by
-              [TPU Research Cloud](https://sites.research.google/trc/about/) (TRC) program. Plan is to pretrain on the
-              English corpus for 200k steps and use the original ByT5 Small model as init checkpoint.
+It turns out, that the results are not on-par with current SOTA on the English AjMC corpus, see a comparison
+[here](https://github.com/stefan-it/blbooks-lms#model-zoo). Thus, we continue experiments with the Hugging Face
+Transformers JAX/FLAX implementation to pretrain ByT5 models on TPU.
 
 # Acknowledgements