nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 27 Packages Projects Releases Wiki Activity
5,991 Commits 1 Branch 0 Tags
e7fecba93416c8aaf343932b5e36dd5e208a815e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Sigbjørn SkjæretandGitHub 221c0e0c58 ci : correct label refactor->refactoring (#14832) 2025-07-23 14:27:54 +02:00
Sigbjørn SkjæretandGitHub e28c0b80c2 cuda : implement bf16 cpy ops and enable bf16 cont (#14763) 
* implement bf16 cpy ops and enable bf16 cont

* deduplicate copy functions

* deduplicate checks
 2025-07-22 12:33:10 +02:00
Sigbjørn SkjæretandGitHub 38d3af1b73 opencl: fix im2col when KW!=KH (#14803) 2025-07-21 13:55:10 -07:00
Sigbjørn SkjæretandGitHub 1ba45d4982 ci : disable failing vulkan crossbuilds (#14723) 2025-07-16 20:52:08 -03:00
Sigbjørn SkjæretandGitHub 19e5943d9e convert : make hf token optional (#14717) 
* make hf token optional

* fail if we can't get necessary tokenizer config
 2025-07-16 23:17:43 +02:00
Sigbjørn SkjæretandGitHub 4b91d6f71f convert : only check for tokenizer folder if we need it (#14704) 2025-07-16 08:52:04 +02:00
Sigbjørn SkjæretandGitHub cf91f217f1 convert : add pre-computed hashes first to prevent order mishaps (#14701) 2025-07-16 08:51:12 +02:00
Sigbjørn SkjæretandGitHub 923e3ea2e3 cuda : add set rows for bf16 (#14664) 2025-07-13 15:01:24 +02:00
Sigbjørn SkjæretandGitHub 105554595f llama : remove unintended whitespace (#14592) 2025-07-09 10:19:50 +02:00
Sigbjørn SkjæretandGitHub e1a7059053 llama : fix incorrect minicpm3 v_states shape (#14571) 2025-07-07 23:35:35 +02:00
Sigbjørn SkjæretandGitHub 12f55c302b llama : remove ggml_cont where possible (#14568) 2025-07-07 21:35:08 +02:00
Sigbjørn SkjæretandGitHub ddef99522d server : fix assistant prefilling when content is an array (#14360) 2025-07-05 09:17:14 +02:00
Sigbjørn SkjæretandGitHub 6681688146 opencl: add GELU_ERF (#14476) 2025-07-04 23:24:56 -07:00
Sigbjørn SkjæretandGitHub 28657a8229 ggml : implement GEGLU_ERF and GEGLU_QUICK ops (#14445) 2025-07-03 23:07:22 +02:00
Sigbjørn SkjæretandGitHub e75ba4c043 gguf-py : add support for chat template jinja files (#14508) 
* add support for chat template jinja files

* remove gemma3n hack
 2025-07-02 21:02:35 +02:00
Sigbjørn SkjæretandGitHub 611ba4b264 ci : add OpenCL to labeler workflow (#14496) 2025-07-02 09:02:51 +02:00
Sigbjørn SkjæretandGitHub eff5e45443 add GELU_ERF (#14455) 2025-07-01 10:14:21 +02:00
Sigbjørn SkjæretandGitHub a5d1fb6212 ggml : fix unmerged GGML_FPxx_TO_FPxx refactoring (#14443) 2025-06-29 14:38:10 +02:00
Sigbjørn SkjæretGitHubGeorgi Gerganov0cc4mAkarshanJeff Bolz
 		a0535ffa0d ggml : implement REGLU/GEGLU/SWIGLU ops (#14158) 
* implement unary REGLU/GEGLU/SWIGLU cpu ops

* relax constraints

* duplicate shape of source

* fix ggml_vec_geglu_f16

* special case gated ops

* implement unary REGLU/GEGLU/SWIGLU cuda ops

* tighten constraints again

* refactor into GGML_GLU_OP

* metal : add glu kernels

ggml-ci

* add CUDA_GLU_BLOCK_SIZE [no ci]

* more constraints and use 64bit ints

ggml-ci

* 64bit multiplication [no ci]

* implement swapped variants (cpu/cuda)

* update comment [no ci]

ggml-ci

* Vulkan: Add GLU ops and shaders

* SYCL: Implement fused kernel GEGLU, SWIGLU and REGLU for single up+gate

* ggml : implement GLU for split up/gate (#14181)

* implement GLU for split up/gate

* add tests for ggml_glu_split

* Vulkan: Implement glu_split logic and shader support

* add split to logging [no ci]

* SYCL: refactor element_size ops and add split up and gate support to gated kernels

* SYCL: switch GEGLU to use tanh approximation

---------

Co-authored-by: 0cc4m <picard12@live.de>
Co-authored-by: Akarshan <akarshan@menlo.ai>

* GGML: increase OP count in assertion

* Refactor: Optimize SYCL element-wise operations with unary function inlining

This commit refactors the SYCL element-wise operations to improve performance by:

- Inlining unary operations (sgn, abs, elu, gelu, silu, etc.) to reduce kernel launch overhead.
- Introducing helper functions `op_xxx` for each unary operation to encapsulate the logic.
- Replacing direct kernel calls with calls to these inlined functions.
- Using `__dpct_inline__` to encourage compiler inlining.
- Minor code cleanup and consistency improvements.

The changes aim to reduce kernel launch overhead and improve the overall efficiency of element-wise operations on SYCL devices.

* vulkan: Increase workgroup size for GLU, for performance (#14345)

* vulkan: Increase workgroup size for GLU, for performance

* vulkan: change GLU shaders to do one element per invocation rather than one row per workgroup

* merge fix

* metal : add support for split and swap

ggml-ci

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: 0cc4m <picard12@live.de>
Co-authored-by: Akarshan <akarshan@menlo.ai>
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
 2025-06-29 11:04:10 +02:00
Sigbjørn SkjæretandGitHub 6609507a91 ci : fix windows build and release (#14431) 2025-06-28 09:57:07 +02:00
Sigbjørn SkjæretandGitHub f667f1e624 convert : fix broken sentencepiece vocab (#14416) 2025-06-27 10:42:19 +02:00
Sigbjørn SkjæretandGitHub b25346221d llama : return mistral-v7-tekken as default template only (#14390) 2025-06-26 15:01:14 +02:00
Sigbjørn SkjæretandGitHub b193d53069 ggml : do not output unprintable characters on GGUF load failure (#14381) 2025-06-25 23:26:51 +02:00
Sigbjørn SkjæretandGitHub abf241045d main : honor --verbose-prompt on interactive prompts (#14350) 2025-06-24 09:31:00 +02:00
Sigbjørn SkjæretandGitHub 238005c2dc gguf-py : fix SpecialVocab parsing when post_processor is null (#14330) 2025-06-22 19:46:17 +02:00
Sigbjørn SkjæretandGitHub 40bfa04c95 common : use std::string_view now that we target c++17 (#14319) 2025-06-22 08:37:43 +03:00
Sigbjørn SkjæretandGitHub aa0ef5c578 gguf-py : fix Qwen3-Embedding eos token (#14314) 2025-06-21 18:12:05 +02:00
Sigbjørn SkjæretandGitHub 58cba76a9a gguf-py : fix TemplateProcessing pair when bos/eos is missing (#14312) 2025-06-21 07:33:21 +02:00
Sigbjørn SkjæretandGitHub 22015b2092 lint : remove trailing whitepace (#14304) 2025-06-20 16:37:44 +02:00
Sigbjørn SkjæretandGitHub 88fc854b4b llama : improve sep token handling (#14272) 2025-06-20 14:04:09 +02:00
Sigbjørn SkjæretandGitHub 3865cff4f5 convert : fix null head_dim AutoConfig regression (#14248) 2025-06-18 09:52:07 +02:00
Sigbjørn SkjæretandGitHub e434e69183 common : suggest --jinja when autodetection fails (#14222) 2025-06-16 21:58:42 +02:00
Sigbjørn SkjæretandGitHub d4e0d95cf5 chore : clean up relative source dir paths (#14128) 2025-06-11 19:04:23 +02:00
Sigbjørn SkjæretandGitHub cc66a7f78f tests : add test-tokenizers-repo (#14017) 2025-06-11 17:16:32 +02:00
Sigbjørn SkjæretandGitHub 55f6b9fa65 convert : fix duplicate key DeepSeek-R1 conversion error (#14103) 2025-06-10 23:29:52 +02:00
Sigbjørn SkjæretandGitHub 3678b838bb llama : support GEGLU for jina-bert-v2 (#14090) 2025-06-10 18:02:08 +02:00
Sigbjørn SkjæretandGitHub 0974ad7a7c llama : fix llama_model_chat_template with template name (LLM_KV with suffix) (#14050) 2025-06-07 14:13:12 +02:00
Sigbjørn SkjæretandGitHub d17a809ef0 llama : support multiple classifier outputs and labels (#13940) 2025-06-06 09:03:25 +02:00
Sigbjørn SkjæretandGitHub 1caae7fc6c gguf-py : add add_classifier_output_labels method to writer (#14031) 
* add add_classifier_output_labels

* use add_classifier_output_labels
 2025-06-05 17:42:31 +02:00
Sigbjørn SkjæretandGitHub 9f47fa5792 vocab : warn about missing mask token (#14022) 2025-06-05 09:29:18 +02:00
Sigbjørn SkjæretandGitHub 5e1c3aed40 convert : fix nomic-bert-moe mask token (#13757) 2025-06-01 18:07:21 +02:00
Sigbjørn SkjæretandGitHub c496fe0b1d convert : fix vocab padding code for bert models (#13954) 2025-06-01 17:23:11 +02:00
Sigbjørn SkjæretandGitHub db38704f01 convert : fix rwkv bos/eos token (#13844) 2025-05-30 14:50:43 +02:00
Sigbjørn SkjæretandGitHub e83ba3e460 llama : add support for jina-reranker-v2 (#13900) 2025-05-29 21:42:31 +02:00
Sigbjørn SkjæretandGitHub 2b131621e6 gguf-py : add support for sub_type (in arrays) in GGUFWriter add_key_value method (#13561) 2025-05-29 15:36:05 +02:00
Sigbjørn SkjæretandGitHub 5ca82fc1d7 convert : workaround for AutoConfig dummy labels (#13881) 2025-05-29 10:00:57 +02:00
Sigbjørn SkjæretandGitHub 6385b843a8 llama : add RobertaForSequenceClassification reranker support (#13875) 2025-05-29 08:15:01 +02:00
Sigbjørn SkjæretandGitHub aa50ba462f tests : improve UGM tokenizer test coverage (#13773) 2025-05-25 16:22:29 +02:00
Sigbjørn SkjæretandGitHub c3a2624339 vocab : fix ugm tokenizer precision (#13743) 2025-05-24 12:29:09 +02:00
Sigbjørn SkjæretandGitHub 5be24af73d gguf-py : correct charsmap parameter typing (#13701) 2025-05-22 14:25:05 +02:00
Sigbjørn SkjæretandGitHub 2aa777d86d examples : switch retrieval to llama_encode (#13685) 
* switch retrieval to llama_encode

* enable --no-warmup for retrieval
 2025-05-21 16:57:38 +02:00
Sigbjørn SkjæretandGitHub 759e37b0d8 tests : avoid github urls due to throttling (#13654) 2025-05-20 12:03:17 +02:00
Sigbjørn SkjæretandGitHub 7c07ac244d ci : add ppc64el to build-linux-cross (#13575) 2025-05-16 14:54:23 +02:00
Sigbjørn SkjæretandGitHub f5170c1d7a editorconfig : fix trailing whitespace from #13542 (#13546) 2025-05-14 21:22:49 +03:00
Sigbjørn SkjæretandGitHub be1d4a13db scripts : fix compare-llama-bench.py show parameter (#13514) 2025-05-14 08:41:01 +02:00
Sigbjørn SkjæretandGitHub bf79371120 scripts : support arbitrary input file formats in compare-llama-bench.py (#13455) 2025-05-13 15:31:12 +02:00
Sigbjørn SkjæretandGitHub 09232370fc scripts : exit compare-llama-bench.py gracefully when there's nothing to compare (#13451) 2025-05-11 16:20:39 +02:00
Sigbjørn SkjæretandGitHub d2a4ef05c6 vocab : add ByteDance-Seed/Seed-Coder (#13423) 2025-05-10 22:08:07 +02:00
Sigbjørn SkjæretandGitHub 43dfd741a5 llguidance : set tokenizer slices to default (#13424) 2025-05-10 17:19:52 +02:00
Sigbjørn SkjæretandGitHub 1a844be132 convert : support rope_scaling type and rope_type (#13349) 2025-05-08 15:34:29 +02:00
Sigbjørn SkjæretandGitHub bc4e1128f7 llama : deci : support ffn-free with attention (#13296) 2025-05-07 12:49:27 +02:00
Sigbjørn SkjæretGitHubXuan-Son Nguyen
 		764b85627b convert : qwen2/3moe : set yarn metadata if present (#13331) 
* set yarn metadata if present

* add comment about enabling YaRN

Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>

---------

Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
 2025-05-06 11:12:06 +02:00
Sigbjørn SkjæretandGitHub ae803bfc3d convert : bailingmoe : set yarn metadata if present (#13312) 2025-05-05 12:34:26 +02:00
Sigbjørn SkjæretandGitHub cb06a3c363 llama : orion rope type is neox (#13261) 2025-05-02 12:44:24 +02:00
Sigbjørn SkjæretandGitHub 626083faf7 llama : plamo rope type is neox (#13260) 2025-05-02 12:40:56 +02:00
Sigbjørn SkjæretandGitHub 7d3af70b08 llama : llm_type order by size (#13177) 2025-04-29 13:25:53 +02:00
Sigbjørn SkjæretandGitHub e98b3692be llama : set qwen3 model type sizes (#13175) 2025-04-29 11:00:31 +02:00
Sigbjørn SkjæretandGitHub fb28f4f80e gguf-py : fix upload python package workflow (#13020) 2025-04-19 16:26:38 +02:00
Sigbjørn SkjæretandGitHub 7538246e7c cuda : add f32 to bf16 copy op (#12806) 
This allows BF16 KV-cache on CUDA.
 2025-04-08 23:21:31 +02:00
Sigbjørn SkjæretandGeorgi Gerganov 36ca8b3628 CUDA: don't convert BF16 weights to FP32 (ggml/1174) 
* add bf16 support

* use convert_from_bf16_cuda instead of convert_unary_cuda for f32

* revert 7ec5085

* move functionality into convert_unary with constexpr
 2025-04-07 18:44:17 +03:00
Sigbjørn SkjæretandGitHub 83a88bd6af vocab : BailingMoE : change possessive quantifiers to greedy (#12677) 2025-04-02 11:21:48 +02:00
Sigbjørn SkjæretandGitHub 5936a616e4 convert : BailingMoE : fix qkv split when head_dim is 0 (#12687) 
NOTE: Ling-lite-base is broken, see https://huggingface.co/inclusionAI/Ling-lite-base/discussions/2
 2025-04-01 14:37:13 +02:00
Sigbjørn SkjæretandGitHub 35782aeedb convert : BailingMoE : avoid setting rope_dim to 0 (#12678) 2025-03-31 23:09:48 +02:00
Sigbjørn SkjæretandGitHub 403fbacbbc convert : Qwerky : use lora_rank_tokenshift and lora_rank_decay if present (#12667) 2025-03-31 16:36:25 +02:00
Sigbjørn SkjæretandGitHub 1a85949067 llava : proper description fix (#12668) 2025-03-31 11:28:30 +02:00
Sigbjørn SkjæretandGitHub f52d59d771 llava : fix clip loading GGUFs with missing description (#12660) 2025-03-31 11:07:07 +02:00
Sigbjørn SkjæretandGitHub 2c3f8b850a llama : support BailingMoE (Ling) (#12634) 2025-03-30 22:21:03 +02:00
Sigbjørn SkjæretandGitHub 3714c3ee1a llama : fix incorrect Qwen2Moe ffn_moe_out graph callback (#12631) 2025-03-28 22:13:02 +01:00
Sigbjørn SkjæretandGitHub 53af4dba42 convert: fix Mistral3/Gemma3 model hparams init (#12571) 
* Fix Mistral3/Gemma3 model hparams init

* set positional args correctly

* use existing hparams if passed
 2025-03-25 23:03:10 +01:00
Sigbjørn SkjæretandGitHub 960e726077 chore : cleanup llama_model_loader::TENSOR_ usage (#12492) 2025-03-21 10:21:36 +01:00
Sigbjørn SkjæretandGitHub dbb3a4739e llama : make Qwen2MoE QKV bias optional (#12477) 2025-03-20 12:49:59 +01:00
Sigbjørn SkjæretandGitHub 108e53c2f1 llama : add support for GPT2, Bloom and CodeShell tied word embeddings (#12456) 
* Add support for GPT2, Bloom and CodeShell tied word embeddings

* Deduplicate tied word embeddings weights

* Workaround for incorrect weight map

It appears transformer.wte.weight is in the weight map even though the weights are not there, remove it if output weights are encountered first.

* check++

* fatfingers--
 2025-03-19 09:08:49 +01:00
Sigbjørn SkjæretandGitHub a686171ea7 convert : Support chat_template.json (#12460) 2025-03-19 08:58:13 +01:00
Sigbjørn SkjæretandGitHub 60c902926c docs : bring llama-cli conversation/template docs up-to-date (#12426) 2025-03-17 21:14:32 +01:00
Sigbjørn SkjæretandGitHub 8ba95dca20 llama : fix OLMo-2-0325-32B-Instruct K-norm size (#12400) 2025-03-16 19:46:36 +02:00
Sigbjørn SkjæretandGitHub 774973b8f3 main : add -sysf / --system-prompt-file (#12249) (#12250) 
* add system_prompt_file

* add -sysf / --system-prompt-file

* remove system_prompt_file
 2025-03-14 16:57:05 +01:00
Sigbjørn SkjæretandGitHub 8fad3c7a7c server : Log original chat template parsing error (#12233) 2025-03-07 11:15:33 +01:00
Sigbjørn SkjæretandGitHub 56d7a9f812 main: allow preloading conversation with -p and add -st / --single-turn (#12145) 
* Add chat template formatting to -no-cnv

* only enable prompt formatting if explicitly enabled

* add -st / --single-turn

* add --single-turn and -p in conversation mode

* fix -sys + -p

* reword warning

* small readability change and fix (long) outdated example usage

* only activate single turn in conversation mode
 2025-03-04 12:19:39 -04:00
Sigbjørn SkjæretGitHubXuan Son Nguyen
 		14dec0c2f2 main: use jinja chat template system prompt by default (#12118) 
* Use jinja chat template system prompt by default

* faster conditional order

* remove nested ternary

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
 2025-03-02 14:53:48 +01:00
Sigbjørn SkjæretGitHubXuan-Son Nguyen
 		1782cdfed6 main: update outdated system prompt message (followup to #12131) (#12132) 
* Update outdated message

* wording

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>

---------

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
 2025-03-01 15:22:27 +01:00
Sigbjørn SkjæretGitHubXuan-Son Nguyen
 		45a8e76745 common : add --system-prompt parameter, replace behavior of -p in conversation mode (#12131) 
* Add --system-prompt parameter

* use user defined system prompt

* clarify

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>

* add warning

* clarify

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>

---------

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
 2025-03-01 13:56:45 +01:00
Sigbjørn SkjæretandGitHub 69050a11be Refactor gguf scripts to improve metadata handling (#11909) 
* Refactor gguf scripts to improve metadata handling

Added contents method to ReaderField class
Added endianess property to GGUFReader class

* update scripts

* fix import

* remove unused import

* attempt to work around flake and pyright errors

* second attempt

* give up, ignore type

* bump version

* apply newbyteorder fixes
 2025-02-26 08:04:48 -05:00
Sigbjørn SkjæretandGitHub 722ec1eb51 perplexity : do not escape input data by default (#9548) 2024-09-20 09:38:10 +03:00
Sigbjørn SkjæretandGitHub eca0fab44e imatrix : disable prompt escape by default (#9543) 2024-09-19 10:58:14 +03:00
Sigbjørn SkjæretGitHubFrancis Couture-Harpin
 		b72c20b85c Fix conversion of unnormalized BF16->BF16 weights (#7843) 
* add truncate_bf16

* truncate intermediate fp32 if converting bf16 to bf16

* fix masking in __compute_fp32_to_bf16

* np.int16 no longer used

* missing cast and additional numpy 2.x fix

* ggml-impl : do not flush bf16 subnormals to zero

* ggml : add reference fp32 to bf16 conversion

The fast version is no longer equivalent for all platforms
because of the handling of subnormal values.

* gguf-py : remove flush to zero for bf16 subnormals

* gguf-py : remove float32 truncation to bf16

Rounding achieves the same thing in the cases where this was used.

* missed prototype update in merge

* merge cleanup

---------

Co-authored-by: Francis Couture-Harpin <git@compilade.net>
 2024-08-02 15:11:39 -04:00
Sigbjørn SkjæretandGitHub 38373cfbab Add SPM infill support (#8016) 
* add --spm-infill option

* support --spm-infill

* support --spm-infill
 2024-06-28 12:53:43 +02:00
Sigbjørn SkjæretandGitHub 6030c61281 Add Qwen2MoE 57B-A14B model identifier (#8158) 
* Add Qwen2MoE 57B-A14B

* Add Qwen2MoE 57B-A14B
 2024-06-27 16:27:41 +02:00
Sigbjørn SkjæretandGitHub 911e35bb8b llama : fix CodeLlama FIM token checks (#8144) 
* account for space prefix character

* use find instead
 2024-06-27 10:46:41 +03:00
Sigbjørn SkjæretandGitHub 91c188d6c2 Only use FIM middle token if it exists (#7648) 
* Only use FIM middle if it exists

* Only use FIM middle if it exists
 2024-06-18 22:19:45 +10:00
Sigbjørn SkjæretandGitHub 6fcd1331ef llama : more checks before assuming FIM tokens (#7644) 
* More checks before assuming FIM tokens for Llama arch

* extensive token check
 2024-06-14 13:20:04 +03:00