6 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
3 people	475df1d6cf	llama : allow for user specified embedding pooling type (#5849) ... * allow for user specified pooling type * llama : use enum types over int --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-03-03 12:40:27 +02:00
Douglas HanleyandGitHub	9600d59e01	unicode : switch to multimap based nfd_map (#5799) ... * switch to multimap based nfd_map due to compile time issues * simplify multimap keys * dont construct new locale every time	2024-03-01 11:15:36 +02:00
3 people	177628bfd8	llama : improve BERT tokenization (#5740) ... * implement nfd for stripping accents in wpm tokenizer * sort nfd map; reuse iterator * use builtin tolower * add locale include * Simplify to_lower cases Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com> --------- Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com>	2024-02-28 10:51:11 +02:00
Douglas HanleyandGitHub	4524290e87	Use correct type of pooling for embedding models (#5500) ... Use correct type of pooling for embedding models	2024-02-15 12:21:49 -05:00
3 people	03bf161eb6	llama : support batched embeddings (#5466) ... * batched embedding: pool outputs by sequence id. updated embedding example * bring back non-causal attention * embd : minor improvements * llama : minor --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-02-13 14:06:58 +02:00
5 people	2891c8aa9a	Add support for BERT embedding models (#5423) ... * BERT model graph construction (build_bert) * WordPiece tokenizer (llm_tokenize_wpm) * Add flag for non-causal attention models * Allow for models that only output embeddings * Support conversion of BERT models to GGUF * Based on prior work by @xyzhang626 and @skeskinen --------- Co-authored-by: Jared Van Bortel <jared@nomic.ai> Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-02-11 11:21:38 -05:00