Commit Graph
3 Commits
Author SHA1 Message Date
Mack StraightandGitHub c98ae02668 fix typo in comment (#318) 2023-03-21 17:49:43 +02:00
Mack StraightandGitHub a791a68b61 move file magic/version to header, print expected version (#319) 2023-03-20 19:26:01 +00:00
074bea2eb1 sentencepiece bpe compatible tokenizer (#252)
* potential out of bounds read

* fix quantize

* style

* Update convert-pth-to-ggml.py

* mild cleanup

* don't need the space-prefixing here rn since main.cpp already does it

* new file magic + version header field

* readme notice

* missing newlines

Co-authored-by: slaren <2141330+slaren@users.noreply.github.com>
2023-03-20 03:17:23 -07:00