nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 29 Packages Projects Releases Wiki Activity
9,897 Commits 1 Branch 0 Tags
nik
Commit Graph
2700 Commits
Author SHA1 Message Date
Johannes GäßlerandGitHub b69f1647f9 CUDA: skip fully masked-out KV in FA vec kernel (#13584) 
* CUDA: skip fully masked-out KV in FA vec kernel
 2025-05-20 14:45:07 +02:00
Svetlozar GeorgievandGitHub 4245e622e0 sycl: disable reorder for sycl mulmat (#13536) 2025-05-20 11:34:15 +02:00
Georgi GerganovandGitHub c00a2634be metal : fix typo in FA kernel comments (#13651) 2025-05-20 10:41:40 +03:00
Nicolò ScipioneandGitHub f7c9429c85 sycl : Overcoming workaround for mmap() allocation on Windows (#13482) 
* Remove mmap workaround on windows

After some testing I found that mmap is supported on windows and for
many GPUs on Linux. Therefore I remove the workaround for windows since
it is not necessary.

* Update llama-bench README

SYCL backend introduced a workaround that allows execution of
llama-bench also without specifying `--mmp 0` flag
 2025-05-20 08:54:43 +08:00
0cc4mandGitHub 8960efd0a6 Vulkan: Add f32 accumulator support to quantized mul mat to fix GLM4 32B incoherence (#13607) 2025-05-19 17:54:08 +02:00
Johannes GäßlerandGeorgi Gerganov 6c35981a64 mnist: fix segmentation fault (ggml/1227) 2025-05-19 13:29:56 +03:00
Diego DevesaandGeorgi Gerganov 8b5e19aea6 ggml : fix apple OS check in ggml_print_backtrace (ggml/1229) 2025-05-19 13:29:56 +03:00
Daniel TangandGeorgi Gerganov 60aea028b5 ggml : Fix missing backtrace on Linux (ggml/1228) 
* Modern Linux defaults /proc/sys/kernel/yama/ptrace_scope to 1
* Fixed lldb attach
* Simplify by having the child do ggml_print_backtrace_symbols
 2025-05-19 13:29:56 +03:00
Chenguang LiandGitHub 33d7aed4a8 CANN: Support MOE Model MUL_MAT_ID (#13042) 
Signed-off-by: noemotiovon <757486878@qq.com>
 2025-05-19 14:21:17 +08:00
Gilad S.andGitHub e3a7cf6c5b cmake: use the current build config for vulkan-shaders-gen (#13595) 
* fix: use the current build config for `vulkan-shaders-gen`

* fix: only pass a valid build type to `--config`
 2025-05-17 15:26:43 -03:00
Jeff BolzandGitHub 2f5a4e1e09 vulkan: move common FA code to flash_attn_base.comp (#13556) 
* vulkan: move common FA code to flash_attn_base.comp

* vulkan: move common FA index/stride setup code to flash_attn_base.comp

* build fix
 2025-05-17 09:14:55 +02:00
Jeff BolzandGitHub 4f41ee11d6 vulkan: use scalar FA rather than coopmat2 when N==1 (#13554) 2025-05-17 08:35:47 +02:00
Georgi GerganovandGitHub 654a67794f metal : add FA-vec kernel for head size 64 (#13583) 
ggml-ci
 2025-05-16 20:32:58 +03:00
Łukasz ŚlusarczykandGitHub 0a338ed013 sycl : fixed compilation warnings (#13582) 2025-05-16 18:15:29 +08:00
Diego DevesaandGitHub c6a2c9e741 gguf : use ggml log system (#13571) 
* gguf : use ggml log system

* llama : remove unnecessary new lines in exception messages
 2025-05-15 19:13:11 +02:00
Atharva DubeyandGitHub 02cdd2d8b0 sycl: simplify bin_bcast_kernel (#13383) 2025-05-15 17:39:52 +02:00
Svetlozar GeorgievandGitHub 64bb51cf90 sycl: reordered Q4_K MMVQ (#13109) 2025-05-15 17:35:44 +02:00
Łukasz ŚlusarczykandGitHub 9c404ed54c sycl: use oneDNN for matrices multiplication (#12972) 2025-05-15 16:53:41 +02:00
Yibo CaiandGitHub 5ab5d5fb25 arm64: optimize q6_k_q8_k kernel with i8mm (#13519) 
This PR improves q6_k_q8_k gemm kernel with arm64 i8mm instruction.

Tested on neoverse-n2 with llama3 8b q6_k quantization model.
- 40% ~ 54% S_PP uplift for all batch sizes
- 16% ~ 47% S_TG uplift for batch size 4 and above

Perplexity doesn't change with this PR.

```
// tested on neoverse-n2
$ llama-batched-bench \
      -m Meta-Llama-3-8B-Instruct-Q6_K.gguf \
      --no-mmap -fa \
      -c 8192 -b 4096 -ub 512 -npp 128 -ntg 128 \
      -npl 1,2,4,8,16,32 \
      -t 64

---------------------------------------------------------------------
|    PP |     TG |    B |       S_PP t/s      |       S_TG t/s      |
|       |        |      | original |  this pr | original |  this pr |
|-------|--------|------|----------|----------|----------|----------|
|   128 |    128 |    1 |    78.52 |   109.18 |    18.63 |    18.88 |
|   128 |    128 |    2 |    84.62 |   123.94 |    34.54 |    36.92 |
|   128 |    128 |    4 |    84.36 |   122.49 |    52.65 |    61.32 |
|   128 |    128 |    8 |    90.52 |   138.87 |    63.46 |    84.41 |
|   128 |    128 |   16 |    90.11 |   138.56 |    71.04 |   101.33 |
|   128 |    128 |   32 |    89.81 |   137.79 |    75.14 |   110.47 |
---------------------------------------------------------------------
```
 2025-05-14 21:53:52 +02:00
Johannes GäßlerandGitHub 4696d56749 CUDA: fix crash on large batch size for quant. MoE (#13537) 2025-05-14 16:41:02 +02:00
Johannes GäßlerandGitHub 6da34fa276 CUDA: faster Deepseek FA, add Turing support (#13435) 2025-05-14 16:08:20 +02:00
bandotiandGitHub 09d13d94fb cmake: simplify vulkan shader test logic (#13263) 2025-05-14 07:53:57 -03:00
Jeff BolzandGitHub 24e86cae72 vulkan: KHR_coopmat flash attention (#13506) 
This shader uses coopmat1 to do the Q*K^T multiply. The P*V multiply is more
difficult for various reasons so I haven't done it. Performance for this
shader is around 2.5x better than for the scalar shader when doing prompt
processing. Some of the benefit may be from other optimizations like staging
through shared memory, or splitting by rows.
 2025-05-14 11:55:26 +02:00
Jeff BolzandGitHub ab3971f2a0 vulkan: workaround FA compile failures on macos (#13517) 2025-05-14 06:15:50 +02:00
Georgi GerganovandGitHub f0995d28ce metal : use FA-vec kernel up to batch size 20 (#13496) 
* batched-bench : fix pp batch contents

* metal : optimize multi-sequence FA vec kernel

ggml-ci

* metal : use FA-vec kernel up to batch size 20

ggml-ci
 2025-05-13 18:04:39 +03:00
Georgi GerganovandGitHub c252e0c409 metal : optimize multi-sequence FA vec kernel (#13493) 
* batched-bench : fix pp batch contents

* metal : optimize multi-sequence FA vec kernel

ggml-ci
 2025-05-13 18:04:00 +03:00
Dan JohanssonandGitHub 4f711afed5 ggml-cpu: Update KleidiAI to v1.6 and fix include directives (#13509) 
Signed-off-by: Dan Johansson <dan.johansson@arm.com>
 2025-05-13 18:02:28 +03:00
lhezandGitHub f0d46ef157 opencl: remove unnecessary assert for add (#13257) 2025-05-12 13:13:49 -07:00
Johannes GäßlerandGitHub 10d2af0eaa llama/ggml: add LLM training support (#10544) 
* llama/ggml: add LLM training support

more compact progress bar

llama_save_model_to_file

llama_opt_param_filter

ggml_graph_dup force_grads

refactor ggml_opt, fix test-opt

* remove logits_all

* refactor CUDA implementation for ACC

* reset graph at beginning of opt period
 2025-05-12 14:44:49 +02:00
Dan JohanssonGitHubCharles Xu
 		a71a4075cd ggml-cpu: Integrate fp32=bf16xbf16 SME KleidiAI kernel (#13053) 
* ggml-cpu: Integrate fp32=bf16xbf16 SME KleidiAI kernel

Signed-off-by: Dan Johansson <dan.johansson@arm.com>

* * code review fixes

Signed-off-by: Dan Johansson <dan.johansson@arm.com>

* * adds a comment that clarifies barrier usage

Signed-off-by: Dan Johansson <dan.johansson@arm.com>

---------

Signed-off-by: Dan Johansson <dan.johansson@arm.com>
Co-authored-by: Charles Xu <charles.xu@arm.com>
 2025-05-12 13:06:19 +02:00
Johannes GäßlerandGitHub 95e18884fc CUDA: fix misaligned synchronization in FA (#13469) 2025-05-12 10:51:21 +02:00
Xuan-Son NguyenandGitHub df8491922f ggml : add mrope kernel for metal (#13457) 2025-05-12 10:29:13 +02:00
Atharva DubeyandGitHub 14492144c2 enable dpcpp nightly builds with libraries (#13406) 2025-05-12 13:15:32 +08:00
Johannes GäßlerandGitHub 7474e00b34 CUDA: fix crash with partial offloading of MoE (#13439) 2025-05-11 16:09:33 +02:00
David HuangandGitHub 7f323a589f Add --no-op-offload to improve -ot pp perf in MoE models like llama4 400B (#13386) 2025-05-11 14:18:39 +02:00
Johannes GäßlerandGitHub 0208355f42 CUDA: fix race conditions FlashAttention kernels (#13438) 2025-05-10 22:22:48 +02:00
Johannes GäßlerandGitHub d8919424f1 CUDA: fix FlashAttention on Turing (#13415) 2025-05-10 09:16:52 +02:00
Jeff BolzandGitHub dc1d2adfc0 vulkan: scalar flash attention implementation (#13324) 
* vulkan: scalar flash attention implementation

* vulkan: always use fp32 for scalar flash attention

* vulkan: use vector loads in scalar flash attention shader

* vulkan: remove PV matrix, helps with register usage

* vulkan: reduce register usage in scalar FA, but perf may be slightly worse

* vulkan: load each Q value once. optimize O reduction. more tuning

* vulkan: support q4_0/q8_0 KV in scalar FA

* CI: increase timeout to accommodate newly-supported tests

* vulkan: for scalar FA, select between 1 and 8 rows

* vulkan: avoid using Float16 capability in scalar FA
 2025-05-10 08:07:07 +02:00
Alberto Cabrera PérezGitHubromain.biessy
 		17512a94d6 sycl : implementation of reordered Q4_0 MMVQ for Intel GPUs (#12858) 
* sycl : Implemented reorder Q4_0 mmvq

Signed-off-by: Alberto Cabrera <alberto.cabrera@codeplay.com>

* sycl : Fixed mmvq being called when reorder is disabled

* sycl : Improved comments in the quants header

Signed-off-by: Alberto Cabrera <alberto.cabrera@codeplay.com>

* Use static_assert

* safe_div -> ceil_div

* Clarify qi comment

* change the reorder tensor from init to execute OP

* dbg

* Undo changes to test-backend-ops

* Refactor changes on top of q4_0 reorder fix

* Missing Reverts

* Refactored opt_for_reorder logic to simplify code path

* Explicit inlining and unroll

* Renamed mul_mat_algo enum for consistency

---------

Signed-off-by: Alberto Cabrera <alberto.cabrera@codeplay.com>
Co-authored-by: romain.biessy <romain.biessy@codeplay.com>
 2025-05-09 16:34:08 +01:00
Georgi GerganovandGitHub 611aa914ef metal : optimize MoE for large batches (#13388) 
ggml-ci
 2025-05-09 15:14:56 +03:00
Johannes GäßlerandGitHub 0cf6725e9f CUDA: FA support for Deepseek (Ampere or newer) (#13306) 
* CUDA: FA support for Deepseek (Ampere or newer)

* do loop unrolling via C++ template
 2025-05-09 13:34:58 +02:00
Johannes GäßlerandGitHub 5c86c9ed3e CUDA: fix crash on large batch size for MoE models (#13384) 2025-05-09 12:14:04 +02:00
Radoslav GerganovandGitHub b486ba05bf rpc : add rpc_msg_set_tensor_hash_req (#13353) 
* rpc : add rpc_msg_set_tensor_hash_req

Use a dedicated struct for the request of RPC_CMD_SET_TENSOR_HASH which
makes the code cleaner.

* fix
 2025-05-09 10:31:07 +03:00
Jeff BolzandGitHub 02115dcd9a vulkan: Allow up to 4096 elements for mul_mat_id row_ids (#13326) 
This assert fired running Qwen_Qwen3-30B-A3B-Q2_K.gguf:

GGML_ASSERT(nei0 * nei1 <= 3072);

The tensor is 8 x 512. Increase this array size to accommodate.
 2025-05-09 09:23:41 +02:00
Alberto Cabrera PérezandGitHub 8733e0cf6e sycl: addressing non-contiguous src1 mul_mats (nc and batched) (#13343) 
* sycl: fixed non-contiguous src1 mul_mats (nc and batched)

* Fixed wrong static_cast inside kernel
 2025-05-08 10:08:01 +01:00
Daniel BeveniusandGeorgi Gerganov 13b0a04597 whisper: remove MSVC warnings pragmas (whisper/3090) 
* ggml : remove MSVC warnings pragmas

This commit removes the MSVC-specific pragmas as these are now handled
in ggml/CMakeLists.txt.

* whisper : remove MSVC warning pragmas

This commit removes the MSVC-specific pragmas. These are now handled in
the ggml/CMakeLists.txt file.
 2025-05-07 17:28:36 +03:00
Jared TweedandGeorgi Gerganov bba9d945c1 cmake : removed stdc++fs (whisper/3097) 
* removed stdc++fs

* kept line, but removed stdc++fs
 2025-05-07 17:28:36 +03:00
R0CKSTARandGitHub 1f73301b63 cuda : remove nrows_x in mul_mat_q_process_tile (#13325) 
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
 2025-05-07 09:48:23 +02:00
Johannes GäßlerandGitHub 141a908a59 CUDA: mix virt/real CUDA archs for GGML_NATIVE=OFF (#13135) 2025-05-06 23:35:51 +02:00
Akarshan BiswasandGitHub 1e333d5bba SYCL: Disable reorder optimize by default and stop setting tensor extras when optimize is disabled (#13254) 
* SYCL: Do not set tensor extras when reorder optimize is disabled

* SYCL: Disable reorder optimize by default
 2025-05-06 20:27:06 +05:30