wolfram commited on
Commit
36fb60f
·
verified ·
1 Parent(s): 7816b35

fixed a typo

app/src/content/chapters/general-knowledge/model-inference-and-evaluation.mdx CHANGED
@@ -52,7 +52,7 @@ I would strongly recommend reading a longer explanation on how BPE works, as it'
52
  - [BPE Paper (for text, as the method existed before in other fields)](https://aclanthology.org/P16-1162/)
53
  </Note>
54
 
55
- When building a tokenizer require making more choices than one would expect. For example, to tokenize numbers, you don't want to use a basic BPE, but do you only index 0 to 9, and assume all other numbers will be compositions of digits? Do you want to store numbers up to, say, one billion, individually?
56
 
57
  Current well known models display a range of approaches to this, but it's unclear what works better to allow mathematical reasoning. This will affect some mathematical evaluation (and is the reason why almost no evaluation is pure arithmetics).
58
 
 
52
  - [BPE Paper (for text, as the method existed before in other fields)](https://aclanthology.org/P16-1162/)
53
  </Note>
54
 
55
+ Building a tokenizer requires making more choices than one would expect. For example, to tokenize numbers, you don't want to use a basic BPE, but do you only index 0 to 9, and assume all other numbers will be compositions of digits? Do you want to store numbers up to, say, one billion, individually?
56
 
57
  Current well known models display a range of approaches to this, but it's unclear what works better to allow mathematical reasoning. This will affect some mathematical evaluation (and is the reason why almost no evaluation is pure arithmetics).
58