nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 20 Packages Projects Releases Wiki Activity
6,818 Commits 1 Branch 0 Tags
d8eaa26e4d9228df3aa46a930db60c8eaab67c1b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Sigbjørn SkjæretandGitHub 84bf3c6778 model : add BailingMoeV2 support (#16063) 
* add BailingMoeV2 support

* update llm types

* undo

* undo

* update llm types

* add model collection link

* update

* almost working

* correct group selection and rename n_group_exp

* avoid large top_k and use argmax instead for now

if we had something like argmax2 that would be equivalent, but this works fine until then

* poke

* skip group selection when there are no tokens

* fix 1T conversion

* hopefully fixed expert group selection

third time's the charm?

* make expert group selection generally available

The new LLaDA2Moe model uses this method too, make it generally available regardless of architecture.

* allow n_expert_groups to be 1 (Kimi K2)

* address review suggestions
 2025-10-20 21:38:20 +02:00
Sigbjørn SkjæretandGitHub cec5edbcae ci : avoid manual updates of docs/ops.md (#16663) 2025-10-19 14:03:25 +02:00
Sigbjørn SkjæretandGitHub 3a002afafa ci : refactor sdk caching to minimize storage (#16414) 
* refactor sdk caching to minimize storage

* use correct action

* add myself as owner to /.github/actions/ [no ci]
 2025-10-06 17:40:21 +02:00
Sigbjørn SkjæretandGitHub 946f71ed9a llama : fix shapes for bert/mpt q/k norm (#16409) 2025-10-03 14:40:25 +02:00
Sigbjørn SkjæretGitHubJeff Bolz
 		d64c8104f0 test-barrier : do not use more threads than physically available (#16389) 
* do not use more threads than physically available

* ensure n_threads > 0

Co-authored-by: Jeff Bolz <jbolz@nvidia.com>

---------

Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
 2025-10-02 20:10:12 +02:00
Sigbjørn SkjæretandGitHub 72ee736c44 ci : fix ubuntu-latest-cmake-rpc (disable ccache) (#16388) 2025-10-02 13:51:36 +02:00
Sigbjørn SkjæretandGitHub 1104ca1a1c ci : use registry cache for docker builds (#16366) 2025-10-01 14:09:52 +02:00
Sigbjørn SkjæretandGitHub b2ba81dbe0 ci : fix ccache key for ubuntu-cpu-cmake (#16355) 
* fix ccache key for ubuntu-cpu-cmake

* set it for release as well [no ci]
 2025-09-30 21:41:42 +02:00
Sigbjørn SkjæretandGitHub 2df5bcf357 ci : disable ccache for android (#16348) 2025-09-30 15:38:01 +02:00
Sigbjørn SkjæretandGitHub adc76347d7 ggml : check cuda and metal argsort limits and add test (#16323) 
* check cuda argsort limits and add test

* add metal check
 2025-09-29 11:09:00 +02:00
Sigbjørn SkjæretandGitHub b887d2f341 ggml : fix GGML_F32_VEC_FMA argument order in ggml_vec_mad1_f32 (#16307) 
* fix GGML_F32_VEC_FMA argument order in ggml_vec_mad1_f32

* add test that fails on simd
 2025-09-28 23:15:03 +02:00
Sigbjørn SkjæretandGitHub 835b2b915c model : add GroveMoE support (#15510) 
* add GroveMoE support

* remove constexpr that fails on certain compilers

* revert crude scalar div implementation, use cast

* build_attn_inp_kv_unified -> build_attn_inp_kv

* fix build_attn

* re-apply ffn_exps regex changes
 2025-09-25 19:50:28 +02:00
Sigbjørn SkjæretandGitHub c0c59c1157 codeowners : match all requirements files (#16214) 2025-09-24 08:53:20 +02:00
Sigbjørn SkjæretandGitHub f6b4af3d04 ggml : fix uninitialized is_on_grid in quantize_row_iq3_xxs_impl (#15928) 
* fix uninitialized is_on_grid in quantize_row_iq3_xxs_impl

* change initialization to true
 2025-09-23 10:25:20 +02:00
Sigbjørn SkjæretGitHubGeorgi GerganovJeff Bolz
 		3ecb2f671a ggml : implement set_rows with i32 index (#16159) 
* implement set_rows with i32 index

* template fix

* test quantized path

warnings--

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* forgotten name change

* deduplicate cuda/sycl and test-fix

* indent++

* vulkan: support set_rows with i32 index type (#16162)

* disable i32 index for webgpu for now

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
 2025-09-22 19:13:00 +02:00
Sigbjørn SkjæretandGitHub b2d980fce0 codeowners : claim responsibility for ci, models, gguf-py and convert (#16124) 
* claim responsibility for ci, gguf-py and convert

* add myself to various src/llama- files
 2025-09-22 10:59:05 +03:00
Sigbjørn SkjæretandGitHub ad6bd9083b cuda : add missing F32<->I32 entries in ggml_cuda_cpy_fn (#16060) 2025-09-18 13:28:22 +02:00
Sigbjørn SkjæretandGitHub b8e09f08b9 model : add grok-2 support (#15539) 
* add grok-2 support

* type fix

* type fix

* type fix

* "fix" vocab for invalid sequences

* fix expert tensor mapping and spaces in vocab

* add chat template

* fix norm tensor mapping

* rename layer_out_norm to ffn_post_norm

* ensure ffn_post_norm is mapped

* fix experts merging

* remove erroneous FFN_GATE entry

* concatenate split tensors and add more metadata

* process all expert layers and try cat instead of hstack

* add support for community BPE vocab

* fix expert feed forward length and ffn_down concat

* commit this too

* add ffn_up/gate/down, unsure if sequence is right

* add ffn_gate/down/up to tensor names

* correct residual moe (still not working)

* mess--

* fix embedding scale being applied twice

* add built in chat template

* change beta fast for grok if default value

* remove spm vocab in favor of community bpe vocab

* change attention temp length metadata type to integer

* update attention temp length metadata

* remove comment

* replace M_SQRT2 with std::sqrt(2)

* add yarn metadata, move defaults to hparams
 2025-09-14 23:00:59 +02:00
Sigbjørn SkjæretandGitHub 6c019cb04e server : only attempt to enable thinking if using jinja (#15967) 2025-09-14 21:17:04 +02:00
Sigbjørn SkjæretGitHubGeorgi Gerganov
 		6ab397e12b graph : support non-contiguous Q in build_attn_mha (#15908) 
* support non-contiguous Q in build_attn_mha

* Update src/llama-graph.cpp

ggml-ci

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-09-10 19:08:59 +02:00
Sigbjørn SkjæretandGitHub 5ef22d281d CUDA: non-contiguous src0 not supported for PAD (#15869) 2025-09-08 12:55:44 +03:00
Sigbjørn SkjæretandGitHub 4281c7b315 ci : exempt correct research label (#15825) 2025-09-06 01:21:15 +02:00
Sigbjørn SkjæretandGitHub 7d3c9f2b21 ci : explicitly set fa off or on (#15692) 2025-08-31 15:30:20 +02:00
Sigbjørn SkjæretandGitHub 84ab83cc0b model : jina-embeddings-v3 support (#13693) 
* initial jina-embeddings-v3 support

* initial jina-embeddings-v3 support

* initial jina-embeddings-v3 support

* fix vocab parsing with only tokenizer.json

* set mask token lstrip attribute

* additional unk_token_id fallback just in case [no ci]

* revert vocab_size() change [no ci]

* merge tensor loading into general bert

* rope

* add lora embedding and loading (non-functional)

* export separate lora ggufs instead

* add adapter metadata api

* use std::string

* convert_hf_to_lora compatibility

* fix assert

* apply suggestions from review

* apply suggestion from review
 2025-08-28 15:49:50 +02:00
Sigbjørn SkjæretandGitHub 39842a7f73 gguf-py : remove erroneous FFN_GATE entry (#15583) 2025-08-26 09:08:08 +02:00
Sigbjørn SkjæretandGitHub 0fd90db585 metal : remove contiguous assertion for src0 in IM2COL (#15577) 
* remove contiguous assertion for src0 in IM2COL

* add contiguous check in supports_op
 2025-08-26 09:51:43 +03:00
Sigbjørn SkjæretandGitHub baa9255a45 llama : merge conts and reshapes and remove unnecessary cont (#15380) 
* remove unnecessary conts and merge reshapes

* restore necessary conts

* merge more conts and reshapes

* merge even more conts and reshapes
 2025-08-18 19:30:17 +02:00
Sigbjørn SkjæretandGitHub 4d196981d4 convert : force patch_embd weights to F16 or F32 to avoid broken GGUFs (#15367) 
* force patch_embd weights to f32

* use MmprojModel base tensor_force_quant instead
 2025-08-17 14:47:42 +02:00
Sigbjørn SkjæretandGitHub b143fbc87a ci : fix hang in windows-hip build/release (#15365) 
* fix hang in windows-latest-cmake-hip

* apply fix to release as well
 2025-08-17 13:30:23 +02:00
Sigbjørn SkjæretandGitHub d3248d9b65 ci : fix ios-xcode-build (#15324) 
* fix ios-xcode-build

* use xcode-select with fixed version

* switch to macos-15 to get xcode 16.4
 2025-08-15 14:02:39 +02:00
Sigbjørn SkjæretandGitHub 4ebd0c125b cuda : fix GGML_CUDA_GRAPHS=OFF (#15300) 
* fix USE_CUDA_GRAPH=OFF

ggml-ci

* check capture status

* completely disable capturing check instead
 2025-08-14 13:22:07 +03:00
Sigbjørn SkjæretandGitHub b3e16665e1 server : enable -td and -tbd parameters (#15172) 2025-08-13 15:43:00 +02:00
Sigbjørn SkjæretandGitHub 07aa869a91 ci : add more python requirements to copilot-setup-steps (#15289) 
* ci : add flake8 and pyright to copilot-setup-steps.yml

* add tools/server/tests/requirements.txt
 2025-08-13 11:30:45 +02:00
Sigbjørn SkjæretandGitHub bc5182272c ci : add copilot-setup-steps.yml (#15214) 2025-08-13 09:07:13 +02:00
Sigbjørn SkjæretandGitHub 50e81bdf5d convert : fix merge conflicts (#15229) 2025-08-11 11:15:44 +02:00
Sigbjørn SkjæretandGitHub 65c797c4fa chat : fix yandex chat template (#15116) 2025-08-06 13:26:49 +02:00
Sigbjørn SkjæretandGitHub f324a3b715 chat : only remove double bos/eos if added (#15086) 
* only remove double bos/eos if added

* fix tests
 2025-08-05 20:43:36 +02:00
Sigbjørn SkjæretandGitHub e5bebe5251 gguf-py : add --chat-template-file to gguf_new_metadata (#15075) 2025-08-04 21:01:48 +02:00
Sigbjørn SkjæretandGitHub 2721257e3e quantize : fix confusing error message if ftype is invalid (#15071) 2025-08-04 18:11:02 +02:00
Sigbjørn SkjæretandGitHub 2bf3fbf0b5 ci : check that pre-tokenizer hashes are up-to-date (#15032) 
* torch is not required for convert_hf_to_gguf_update

* add --check-missing parameter

* check that pre-tokenizer hashes are up-to-date
 2025-08-02 14:39:01 +02:00
Sigbjørn SkjæretandGitHub 138b288b59 cuda : add softcap fusion (#14907) 2025-07-29 14:22:03 +02:00
Sigbjørn SkjæretandGitHub 221c0e0c58 ci : correct label refactor->refactoring (#14832) 2025-07-23 14:27:54 +02:00
Sigbjørn SkjæretandGitHub e28c0b80c2 cuda : implement bf16 cpy ops and enable bf16 cont (#14763) 
* implement bf16 cpy ops and enable bf16 cont

* deduplicate copy functions

* deduplicate checks
 2025-07-22 12:33:10 +02:00
Sigbjørn SkjæretandGitHub 38d3af1b73 opencl: fix im2col when KW!=KH (#14803) 2025-07-21 13:55:10 -07:00
Sigbjørn SkjæretandGitHub 1ba45d4982 ci : disable failing vulkan crossbuilds (#14723) 2025-07-16 20:52:08 -03:00
Sigbjørn SkjæretandGitHub 19e5943d9e convert : make hf token optional (#14717) 
* make hf token optional

* fail if we can't get necessary tokenizer config
 2025-07-16 23:17:43 +02:00
Sigbjørn SkjæretandGitHub 4b91d6f71f convert : only check for tokenizer folder if we need it (#14704) 2025-07-16 08:52:04 +02:00
Sigbjørn SkjæretandGitHub cf91f217f1 convert : add pre-computed hashes first to prevent order mishaps (#14701) 2025-07-16 08:51:12 +02:00
Sigbjørn SkjæretandGitHub 923e3ea2e3 cuda : add set rows for bf16 (#14664) 2025-07-13 15:01:24 +02:00
Sigbjørn SkjæretandGitHub 105554595f llama : remove unintended whitespace (#14592) 2025-07-09 10:19:50 +02:00
Sigbjørn SkjæretandGitHub e1a7059053 llama : fix incorrect minicpm3 v_states shape (#14571) 2025-07-07 23:35:35 +02:00
Sigbjørn SkjæretandGitHub 12f55c302b llama : remove ggml_cont where possible (#14568) 2025-07-07 21:35:08 +02:00
Sigbjørn SkjæretandGitHub ddef99522d server : fix assistant prefilling when content is an array (#14360) 2025-07-05 09:17:14 +02:00
Sigbjørn SkjæretandGitHub 6681688146 opencl: add GELU_ERF (#14476) 2025-07-04 23:24:56 -07:00
Sigbjørn SkjæretandGitHub 28657a8229 ggml : implement GEGLU_ERF and GEGLU_QUICK ops (#14445) 2025-07-03 23:07:22 +02:00
Sigbjørn SkjæretandGitHub e75ba4c043 gguf-py : add support for chat template jinja files (#14508) 
* add support for chat template jinja files

* remove gemma3n hack
 2025-07-02 21:02:35 +02:00
Sigbjørn SkjæretandGitHub 611ba4b264 ci : add OpenCL to labeler workflow (#14496) 2025-07-02 09:02:51 +02:00
Sigbjørn SkjæretandGitHub eff5e45443 add GELU_ERF (#14455) 2025-07-01 10:14:21 +02:00
Sigbjørn SkjæretandGitHub a5d1fb6212 ggml : fix unmerged GGML_FPxx_TO_FPxx refactoring (#14443) 2025-06-29 14:38:10 +02:00
Sigbjørn SkjæretGitHubGeorgi Gerganov0cc4mAkarshanJeff Bolz
 		a0535ffa0d ggml : implement REGLU/GEGLU/SWIGLU ops (#14158) 
* implement unary REGLU/GEGLU/SWIGLU cpu ops

* relax constraints

* duplicate shape of source

* fix ggml_vec_geglu_f16

* special case gated ops

* implement unary REGLU/GEGLU/SWIGLU cuda ops

* tighten constraints again

* refactor into GGML_GLU_OP

* metal : add glu kernels

ggml-ci

* add CUDA_GLU_BLOCK_SIZE [no ci]

* more constraints and use 64bit ints

ggml-ci

* 64bit multiplication [no ci]

* implement swapped variants (cpu/cuda)

* update comment [no ci]

ggml-ci

* Vulkan: Add GLU ops and shaders

* SYCL: Implement fused kernel GEGLU, SWIGLU and REGLU for single up+gate

* ggml : implement GLU for split up/gate (#14181)

* implement GLU for split up/gate

* add tests for ggml_glu_split

* Vulkan: Implement glu_split logic and shader support

* add split to logging [no ci]

* SYCL: refactor element_size ops and add split up and gate support to gated kernels

* SYCL: switch GEGLU to use tanh approximation

---------

Co-authored-by: 0cc4m <picard12@live.de>
Co-authored-by: Akarshan <akarshan@menlo.ai>

* GGML: increase OP count in assertion

* Refactor: Optimize SYCL element-wise operations with unary function inlining

This commit refactors the SYCL element-wise operations to improve performance by:

- Inlining unary operations (sgn, abs, elu, gelu, silu, etc.) to reduce kernel launch overhead.
- Introducing helper functions `op_xxx` for each unary operation to encapsulate the logic.
- Replacing direct kernel calls with calls to these inlined functions.
- Using `__dpct_inline__` to encourage compiler inlining.
- Minor code cleanup and consistency improvements.

The changes aim to reduce kernel launch overhead and improve the overall efficiency of element-wise operations on SYCL devices.

* vulkan: Increase workgroup size for GLU, for performance (#14345)

* vulkan: Increase workgroup size for GLU, for performance

* vulkan: change GLU shaders to do one element per invocation rather than one row per workgroup

* merge fix

* metal : add support for split and swap

ggml-ci

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: 0cc4m <picard12@live.de>
Co-authored-by: Akarshan <akarshan@menlo.ai>
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
 2025-06-29 11:04:10 +02:00
Sigbjørn SkjæretandGitHub 6609507a91 ci : fix windows build and release (#14431) 2025-06-28 09:57:07 +02:00
Sigbjørn SkjæretandGitHub f667f1e624 convert : fix broken sentencepiece vocab (#14416) 2025-06-27 10:42:19 +02:00
Sigbjørn SkjæretandGitHub b25346221d llama : return mistral-v7-tekken as default template only (#14390) 2025-06-26 15:01:14 +02:00
Sigbjørn SkjæretandGitHub b193d53069 ggml : do not output unprintable characters on GGUF load failure (#14381) 2025-06-25 23:26:51 +02:00
Sigbjørn SkjæretandGitHub abf241045d main : honor --verbose-prompt on interactive prompts (#14350) 2025-06-24 09:31:00 +02:00
Sigbjørn SkjæretandGitHub 238005c2dc gguf-py : fix SpecialVocab parsing when post_processor is null (#14330) 2025-06-22 19:46:17 +02:00
Sigbjørn SkjæretandGitHub 40bfa04c95 common : use std::string_view now that we target c++17 (#14319) 2025-06-22 08:37:43 +03:00
Sigbjørn SkjæretandGitHub aa0ef5c578 gguf-py : fix Qwen3-Embedding eos token (#14314) 2025-06-21 18:12:05 +02:00
Sigbjørn SkjæretandGitHub 58cba76a9a gguf-py : fix TemplateProcessing pair when bos/eos is missing (#14312) 2025-06-21 07:33:21 +02:00
Sigbjørn SkjæretandGitHub 22015b2092 lint : remove trailing whitepace (#14304) 2025-06-20 16:37:44 +02:00
Sigbjørn SkjæretandGitHub 88fc854b4b llama : improve sep token handling (#14272) 2025-06-20 14:04:09 +02:00
Sigbjørn SkjæretandGitHub 3865cff4f5 convert : fix null head_dim AutoConfig regression (#14248) 2025-06-18 09:52:07 +02:00
Sigbjørn SkjæretandGitHub e434e69183 common : suggest --jinja when autodetection fails (#14222) 2025-06-16 21:58:42 +02:00
Sigbjørn SkjæretandGitHub d4e0d95cf5 chore : clean up relative source dir paths (#14128) 2025-06-11 19:04:23 +02:00
Sigbjørn SkjæretandGitHub cc66a7f78f tests : add test-tokenizers-repo (#14017) 2025-06-11 17:16:32 +02:00
Sigbjørn SkjæretandGitHub 55f6b9fa65 convert : fix duplicate key DeepSeek-R1 conversion error (#14103) 2025-06-10 23:29:52 +02:00
Sigbjørn SkjæretandGitHub 3678b838bb llama : support GEGLU for jina-bert-v2 (#14090) 2025-06-10 18:02:08 +02:00
Sigbjørn SkjæretandGitHub 0974ad7a7c llama : fix llama_model_chat_template with template name (LLM_KV with suffix) (#14050) 2025-06-07 14:13:12 +02:00
Sigbjørn SkjæretandGitHub d17a809ef0 llama : support multiple classifier outputs and labels (#13940) 2025-06-06 09:03:25 +02:00
Sigbjørn SkjæretandGitHub 1caae7fc6c gguf-py : add add_classifier_output_labels method to writer (#14031) 
* add add_classifier_output_labels

* use add_classifier_output_labels
 2025-06-05 17:42:31 +02:00
Sigbjørn SkjæretandGitHub 9f47fa5792 vocab : warn about missing mask token (#14022) 2025-06-05 09:29:18 +02:00
Sigbjørn SkjæretandGitHub 5e1c3aed40 convert : fix nomic-bert-moe mask token (#13757) 2025-06-01 18:07:21 +02:00
Sigbjørn SkjæretandGitHub c496fe0b1d convert : fix vocab padding code for bert models (#13954) 2025-06-01 17:23:11 +02:00
Sigbjørn SkjæretandGitHub db38704f01 convert : fix rwkv bos/eos token (#13844) 2025-05-30 14:50:43 +02:00
Sigbjørn SkjæretandGitHub e83ba3e460 llama : add support for jina-reranker-v2 (#13900) 2025-05-29 21:42:31 +02:00
Sigbjørn SkjæretandGitHub 2b131621e6 gguf-py : add support for sub_type (in arrays) in GGUFWriter add_key_value method (#13561) 2025-05-29 15:36:05 +02:00
Sigbjørn SkjæretandGitHub 5ca82fc1d7 convert : workaround for AutoConfig dummy labels (#13881) 2025-05-29 10:00:57 +02:00
Sigbjørn SkjæretandGitHub 6385b843a8 llama : add RobertaForSequenceClassification reranker support (#13875) 2025-05-29 08:15:01 +02:00
Sigbjørn SkjæretandGitHub aa50ba462f tests : improve UGM tokenizer test coverage (#13773) 2025-05-25 16:22:29 +02:00
Sigbjørn SkjæretandGitHub c3a2624339 vocab : fix ugm tokenizer precision (#13743) 2025-05-24 12:29:09 +02:00
Sigbjørn SkjæretandGitHub 5be24af73d gguf-py : correct charsmap parameter typing (#13701) 2025-05-22 14:25:05 +02:00
Sigbjørn SkjæretandGitHub 2aa777d86d examples : switch retrieval to llama_encode (#13685) 
* switch retrieval to llama_encode

* enable --no-warmup for retrieval
 2025-05-21 16:57:38 +02:00
Sigbjørn SkjæretandGitHub 759e37b0d8 tests : avoid github urls due to throttling (#13654) 2025-05-20 12:03:17 +02:00
Sigbjørn SkjæretandGitHub 7c07ac244d ci : add ppc64el to build-linux-cross (#13575) 2025-05-16 14:54:23 +02:00
Sigbjørn SkjæretandGitHub f5170c1d7a editorconfig : fix trailing whitespace from #13542 (#13546) 2025-05-14 21:22:49 +03:00
Sigbjørn SkjæretandGitHub be1d4a13db scripts : fix compare-llama-bench.py show parameter (#13514) 2025-05-14 08:41:01 +02:00
Sigbjørn SkjæretandGitHub bf79371120 scripts : support arbitrary input file formats in compare-llama-bench.py (#13455) 2025-05-13 15:31:12 +02:00
Sigbjørn SkjæretandGitHub 09232370fc scripts : exit compare-llama-bench.py gracefully when there's nothing to compare (#13451) 2025-05-11 16:20:39 +02:00
Sigbjørn SkjæretandGitHub d2a4ef05c6 vocab : add ByteDance-Seed/Seed-Coder (#13423) 2025-05-10 22:08:07 +02:00
Sigbjørn SkjæretandGitHub 43dfd741a5 llguidance : set tokenizer slices to default (#13424) 2025-05-10 17:19:52 +02:00