llama_cpp

2,835 Commits 1 Branch 0 Tags

3 Commits

Author	SHA1	Message	Date
Pedro Cuenca GitHub Sourab Mangrulkar Georgi Gerganov	b97bc3966e	llama : support Llama 3 HF conversion (#6745 ) * Support Llama 3 conversion The tokenizer is BPE. * style * Accept suggestion Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com> * llama : add llama_token_is_eog() ggml-ci * llama : auto-detect more EOT tokens when missing in KV data * convert : replacing EOS token is a hack * llama : fix codegemma EOT token + add TODOs * llama : fix model type string for 8B model --------- Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-04-21 14:50:41 +03:00
Pedro Cuenca GitHub Jared Van Bortel	b75c38166c	convert : allow conversion of Mistral HF models (#6144 ) * Allow conversion of Mistral HF models * Homogenize Llama, Mistral, Mixtral under the same entry. * Fix tokenizer, permute tensors * Use sentencepiece tokenizer, or fall back to hfft. * convert-hf : small fix for mypy * convert-hf : fix duplicated block_count * convert-hf : add vocab size to metadata --------- Co-authored-by: Jared Van Bortel <jared@nomic.ai>	2024-03-29 09:15:00 +02:00
Pedro CuencaandGitHub	e097633f63	convert-hf : fix exception in sentencepiece with added tokens (#6320 )	2024-03-26 14:32:19 +02:00

Author

SHA1

Message

Date

b97bc3966e

llama : support Llama 3 HF conversion (#6745 )

* Support Llama 3 conversion

The tokenizer is BPE.

* style

* Accept suggestion

Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com>

* llama : add llama_token_is_eog()

ggml-ci

* llama : auto-detect more EOT tokens when missing in KV data

* convert : replacing EOS token is a hack

* llama : fix codegemma EOT token + add TODOs

* llama : fix model type string for 8B model

---------

Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

2024-04-21 14:50:41 +03:00

b75c38166c

convert : allow conversion of Mistral HF models (#6144 )

* Allow conversion of Mistral HF models

* Homogenize Llama, Mistral, Mixtral under the same entry.

* Fix tokenizer, permute tensors

* Use sentencepiece tokenizer, or fall back to hfft.

* convert-hf : small fix for mypy

* convert-hf : fix duplicated block_count

* convert-hf : add vocab size to metadata

---------

Co-authored-by: Jared Van Bortel <jared@nomic.ai>

2024-03-29 09:15:00 +02:00

Pedro CuencaandGitHub

e097633f63

convert-hf : fix exception in sentencepiece with added tokens (#6320 )

2024-03-26 14:32:19 +02:00