nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 22 Packages Projects Releases Wiki Activity

convert : fix encoding of WPM vocab for BERT models (#18500)

Browse Source 
* convert: avoid token collision when stripping ## prefix

* convert: use token types for BERT special tokens check

* Update convert_hf_to_gguf.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
This commit is contained in:
o7si
2026-01-01 18:27:07 +01:00
committed by GitHub
co-authored by Sigbjørn Skjæret
parent f4f5019254
commit 2b2afade9f
1 changed files with 4 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
convert_hf_to_gguf.py
+4 -3
View File
@@ -5287,13 +5287,14 @@ class BertModel(TextModel):
self.gguf_writer.add_token_type_count(self.hparams.get("type_vocab_size", 1)) self.gguf_writer.add_token_type_count(self.hparams.get("type_vocab_size", 1))
# convert to phantom space vocab # convert to phantom space vocab
def phantom(tok): def phantom(tok, toktype):
if tok.startswith("[") and tok.endswith("]"): if toktype == gguf.TokenType.CONTROL:
return tok return tok
if tok.startswith("##"): if tok.startswith("##"):
return tok[2:] return tok[2:]
return "\u2581" + tok return "\u2581" + tok
tokens = list(map(phantom, tokens)) assert len(tokens) == len(toktypes)
tokens = list(map(phantom, tokens, toktypes))
# add vocab to gguf # add vocab to gguf
self.gguf_writer.add_tokenizer_model("bert") self.gguf_writer.add_tokenizer_model("bert")