Galunid and GitHub
a75fa576ab
scripts: Generalize convert scripts ( #3838 )
...
* Replace convert-*-hf-to-gguf.py files with convert-hf-to-gguf.py
2023-11-09 11:09:29 +01:00
Galunid and GitHub
daab3d7f45
Add more tokenizer tests ( #3742 )
...
* Add more tokenizer tests
* Add starcoder
* Update test vocab files
* Restrict bpe tokenizer tests to unicode planes
* Update comment
* Comment cosmetics
* Remove bloom vocab/test
2023-10-24 09:17:17 +02:00
Galunid and GitHub
69a6735087
Update special token handling in conversion scripts for gpt2 derived tokenizers ( #3746 )
...
We still have the heads up in `README.md` regarding `bpe` tokenizers and this patch is needed for
- a couple of tokenizer tests
- some more `special` and `non-special` added tokens handling (as far as I understand it)
* Update special token handling
* Add mpt
2023-10-23 21:46:00 +02:00
Galunid and GitHub
6336701c93
Fix baichuan convert script not detecing model ( #3739 )
...
It seems nobody objects.
2023-10-23 17:47:03 +02:00
Galunid and GitHub
9f6ede19f3
Add MPT model to supported models in README.md ( #3574 )
2023-10-10 19:02:49 -04:00
