nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity
7,839 Commits 1 Branch 0 Tags
8f80d1b254aef70a0959e314be368d05debe7294
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Johannes GäßlerandGitHub 0c21677e43 CUDA: faster FA for GQA > 1 but not power of 2 (#19092) 2026-01-25 21:19:47 +01:00
Johannes GäßlerandGitHub e9fd8dcab4 llama-fit-params: keep explicit --ctx-size 0 (#19070) 2026-01-24 22:13:08 +01:00
Johannes GäßlerandGitHub 4e5b83b226 GGUF: check that tensor size is representable (#19072) 2026-01-24 21:57:51 +01:00
Johannes GäßlerandGitHub 8f91ca54ec CUDA: re-use MLA K data for V in MMA FA (#19057) 2026-01-24 10:09:36 +01:00
Johannes GäßlerandGitHub e2baf02162 CUDA: fix alignment check for FA (#19023) 2026-01-22 20:39:25 +01:00
Johannes GäßlerandGitHub 5c662d21a3 CUDA: fix allignment on register spill for FA (#18815) 2026-01-15 15:14:50 +01:00
Johannes GäßlerandGitHub c1e79e610f doc: ban AI-generated PR descriptions [no ci] (#18765) 2026-01-13 13:43:12 +01:00
Johannes GäßlerandGitHub d2ff4e23ac HIP: adjust RDNA3.5 MMQ kernel selction logic (#18666) 2026-01-10 17:19:01 +01:00
Johannes GäßlerandGitHub 64848deb18 llama-fit-params: free memory target per device (#18679) 2026-01-08 10:07:58 +01:00
Johannes GäßlerandGitHub 68b4d516c3 llama-params-fit: fix last devices with low VRAM (#18494) 2026-01-06 20:02:30 +01:00
Johannes GäßlerandGitHub df17a4c94f CUDA: fix FA FP16 accumulator overflow for Granite (#18614) 2026-01-05 19:51:13 +01:00
Johannes GäßlerandGitHub 0f2e42ca1d CUDA: only allocate FA tmp buffer if needed (#18564) 2026-01-03 13:55:53 +01:00
Johannes GäßlerandGitHub ecc343de63 CUDA: fix KQ max calculation (#18487) 2025-12-31 09:37:00 +01:00
Johannes GäßlerandGitHub 0bd1212a43 CUDA: fix replacment of bad archs in CMake (#18457) 2025-12-29 17:58:20 +01:00
Johannes GäßlerandGitHub e70e640db3 CUDA: Blackwell features for non-native builds (#18436) 2025-12-29 09:35:42 +01:00
Johannes GäßlerandGitHub f8d561eb87 llama-fit-params: fix step size for last device (#18415) 2025-12-28 10:52:09 +01:00
Johannes GäßlerGitHubSigbjørn Skjæret
 		e59efe6a78 github: update issue templates [no ci] (#18410) 
* github: update issue templates [no ci]

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2025-12-28 10:50:56 +01:00
Johannes GäßlerandGitHub a4bf35889e llama-fit-params: fix overflow check (#18354) 2025-12-27 20:20:45 +01:00
Johannes GäßlerandGitHub 026d2ad472 llama: fix magic number of 999 for GPU layers (#18266) 
* llama: fix magic number of 999 for GPU layers

* use strings for -ngl, -ngld

* enacapsulate n_gpu_layers, split_mode
 2025-12-27 20:18:35 +01:00
Johannes GäßlerandGitHub a52dc60ba3 llama_fit_params: return enum for fail vs. error (#18374) 2025-12-27 09:59:19 +01:00
Johannes GäßlerandGitHub 9045c9afe5 llama-fit-params: fix Gemma 3 calculation (#18372) 2025-12-27 09:56:04 +01:00
Johannes GäßlerandGitHub 147a521636 tool/ex/tests: consistently free ctx, then model (#18168) 2025-12-22 11:00:37 +01:00
Johannes GäßlerandGitHub 0e1ccf15c7 llama: fix RPC for -fit on (#18233) 2025-12-21 19:33:08 +01:00
Johannes GäßlerandGitHub 57c1e05643 llama: offload output layer to GPU first (#18148) 2025-12-18 08:12:18 +01:00
Johannes GäßlerandGitHub 8dcc3662a2 llama-fit-params: fix memory print (#18136) 2025-12-17 21:10:03 +01:00
Johannes GäßlerandGitHub a2c199e479 common: clarify instructions for bug reports (#18134) 2025-12-17 18:44:13 +01:00
Johannes GäßlerandGitHub 6f1f6a961a Github: ask for -v logs for params_fit [no ci] (#18128) 2025-12-17 13:46:48 +01:00
Johannes GäßlerandGitHub d0794e89d9 llama-fit-params: force disable mlock (#18103) 2025-12-17 00:50:12 +01:00
Johannes GäßlerandGitHub 9dcac6cf9f llama-fit-params: lower ctx size for multi GPU (#18101) 2025-12-17 00:49:34 +01:00
Johannes GäßlerandGitHub 0e49a7b8b4 llama-fit-params: fix underflow for dense models (#18095) 2025-12-17 00:47:37 +01:00
Johannes GäßlerandGitHub 4164596c76 llama-fit-params: QoL impr. for prints/errors (#18089) 2025-12-17 00:03:19 +01:00
Johannes GäßlerandGitHub ec98e20021 llama: fix early stop in params_fit if ctx is set (#18070) 2025-12-16 14:24:00 +01:00
Johannes GäßlerandGitHub b1f3a6e5db llama: automatically set parameters not set by the user in such a way that maximizes GPU utilization (#16653) 
* llama: automatically fit args to free memory

llama-fit-params tool

* fix CI

* hints for bug reports, ensure no reallocation

* fix segfault with Vulkan

* add llama-fit-params to CI

* fix CI

* fix CI

* fix CI

* minor adjustments

* fix assignment of 1 dense layer

* fix logger not being reset on model load failure

* remove --n-gpu-layer hint on model load failure

* fix llama-fit-params verbosity

* fix edge case

* fix typo [no ci]
 2025-12-15 09:24:59 +01:00
Johannes GäßlerandGitHub 482211438d CUDA: fix overflow in MMA kernel without stream-k (#17939) 2025-12-12 17:43:58 +01:00
Johannes GäßlerandGitHub 17f7f4baad CUDA: fix unpadded strides in MMA FA kernel (#17891) 2025-12-10 12:39:56 +01:00
Johannes GäßlerandGitHub 48f47565a7 docs: clarify that CPU support should be first (#17886) 2025-12-09 20:10:36 +01:00
Johannes GäßlerandGitHub 0cdce38a97 CUDA: fix FP16 overflow in tile FA kernel (#17875) 2025-12-09 09:34:02 +01:00
Johannes GäßlerandGitHub f334b79494 HIP: fix RDNA3 FP16/BF16 matrix multiplication (#17817) 2025-12-06 13:45:36 +01:00
Johannes GäßlerandGitHub 6016d0bd41 HIP : fix RDNA4 build (#17792) 2025-12-05 13:47:52 +01:00
Johannes GäßlerandGitHub e95d0bc8fd CUDA: fix FA VKQ accumulator overflow (#17746) 2025-12-05 09:18:10 +01:00
Johannes GäßlerGitHubAman Gupta <aman>
 		2e1c9cd814 CUDA: generalized (mma) FA, add Volta support (#17505) 
* CUDA: generalized (mma) FA, add Volta support

* use struct for MMA FA kernel config

---------

Co-authored-by: Aman Gupta <aman>
 2025-12-03 16:57:05 +01:00
Johannes GäßlerandGitHub 73955f7d2a CUDA: no FP16 arithmetic for vector FA kernel (#17558) 2025-11-28 10:29:09 +01:00
Johannes GäßlerandGitHub 5d6838b74f CUDA: static assert to prevent misuse of memcpy_1 (#17198) 2025-11-12 23:13:55 +01:00
Johannes GäßlerandGitHub e14e842e87 CUDA: fix MMQ stream-k fixup ne1 indices (#17089) 2025-11-08 08:26:18 +01:00
Johannes GäßlerGitHubAman Gupta
 		6515610506 CUDA: fix should_use_mmvf for ne11 == 1 (#17085) 
* CUDA: fix should_use_mmvf for ne11 == 1

* Apply suggestion from @am17an

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
 2025-11-07 20:53:14 +01:00
Johannes GäßlerandGitHub aa374175c3 CUDA: fix crash on uneven context without FA (#16988) 2025-11-06 14:05:47 +01:00
Johannes GäßlerandGitHub 22c8c3c6ad docs: explain CUDA 11 compilation [no ci] (#16824) 2025-11-06 08:14:35 +01:00
Johannes GäßlerGitHubAman Gupta
 		31c511a968 CUDA: Volta tensor core support for MMF (#16843) 
* CUDA: Volta tensor core support for MMF

* more generic checks for hardware support

* Update ggml/src/ggml-cuda/mmf.cuh

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
 2025-10-31 15:57:19 +01:00
Johannes GäßlerandGitHub 7a0e900e36 llama: consistent ctx <-> buf order for KV cache (#16746) 2025-10-28 11:23:54 +01:00
Johannes GäßlerandGitHub 80d28f104c HIP: fix AMDGPU_TARGETS, update documentation (#16803) 2025-10-27 21:39:49 +01:00
Johannes GäßlerandGitHub 945501f5ea llama: fix leaked buffers for mmap + split files (#16765) 2025-10-27 09:17:31 +01:00
Johannes GäßlerandGitHub 0bf47a1dbb server: add memory breakdown print (#16740) 2025-10-23 21:30:17 +02:00
Johannes GäßlerandGitHub 51d1a8c997 CUDA: better error for FA kernel with 0 occupancy (#16643) 2025-10-21 15:27:53 +02:00
Johannes GäßlerandGitHub ee09828cb0 HIP: fix GPU_TARGETS (#16642) 2025-10-18 14:47:32 +02:00
Johannes GäßlerandGitHub 66b0dbcb2d llama-model: fix insonsistent ctxs <-> bufs order (#16581) 2025-10-17 17:41:09 +02:00
Johannes GäßlerandGitHub 9c7185dd28 CUDA: enable FA for FP32 KV cache (#16546) 2025-10-14 14:22:47 +02:00
Johannes GäßlerandGitHub 7049736b2d CUDA: fix numerical issues in tile FA kernel (#16540) 2025-10-13 17:29:45 +03:00
Johannes GäßlerandGitHub 11f0af5504 CUDA: faster tile FA, add oob checks, more HSs (#16492) 2025-10-11 20:54:32 +02:00
Johannes GäßlerandGitHub 75a3a6c2cd CUDA: refactor and deduplicate vector FA kernels (#16208) 
* CUDA: refactor and deduplicate vector FA kernels
 2025-09-27 18:45:07 +02:00
Johannes GäßlerandGitHub 4cdd0bb453 docs: fix typo [no ci] (#16244) 2025-09-25 12:12:27 +03:00
Johannes GäßlerandGitHub e789095502 llama: print memory breakdown on exit (#15860) 
* llama: print memory breakdown on exit
 2025-09-24 16:53:48 +02:00
Johannes GäßlerandGitHub 368560a1e3 CUDA: fix compilation on CC 6.0 (#16091) 2025-09-18 19:28:32 +02:00
Johannes GäßlerandGitHub c959b676be CUDA: fix FA occupancy, optimize tile kernel (#15982) 2025-09-17 15:32:42 +02:00
Johannes GäßlerandGitHub 0e6ff0046f CUDA: larger SRAM reads for tile FA, AMD FP16 dot (#15927) 
* CUDA: larger SRAM reads for tile FA, AMD FP16 dot

* fix logic for availability of v_dot2_f32_f16
 2025-09-11 21:19:58 +02:00
Johannes GäßlerandGitHub 17bc5a815f HIP: use v_dot2_f32_f16 instruction for FA (#15884) 2025-09-09 14:04:43 +02:00
Johannes GäßlerandGitHub 550cf726e1 CUDA: fix GET_ROWS for large tensors (#15882) 2025-09-09 08:11:01 +02:00
Johannes GäßlerandGitHub 79bc429262 CUDA: faster tile FA (Pascal/AMD), headsize 256 (#15769) 2025-09-07 00:26:28 +02:00
Johannes GäßlerandGitHub 01806e7771 ggml-cpu: document use of "free" memory [no ci] (#15834) 2025-09-06 13:28:44 +02:00
Johannes GäßlerandGitHub 5143fa895e CUDA: fastdiv, launch bounds for mmvq + q8_1 quant (#15802) 
* CUDA: fastdiv, launch bounds for mmvq + q8_1 quant
 2025-09-05 16:07:02 +02:00
Johannes GäßlerandGitHub c466abe158 llama: -fa 1/0/-1 aliases for -fa on/off/auto (#15746) 2025-09-02 18:17:26 +02:00
Johannes GäßlerandGitHub 5d804a4938 ggml-backend: raise GGML_MAX_SPLIT_INPUTS (#15722) 2025-09-01 16:14:55 -07:00
Johannes GäßlerandGitHub e81b8e4b7f llama: use FA + max. GPU layers by default (#15434) 
* llama: use max. GPU layers by default, auto -fa

* ggml-backend: abort instead of segfault
 2025-08-30 16:32:10 +02:00
Johannes GäßlerandGitHub 38ad381f9f CUDA: use FP32 arithmetic for conv2d (#15683) 2025-08-30 16:20:32 +02:00
Johannes GäßlerandGitHub 3d16b29c3b scripts: strip "AMD Instinct" from GPU name (#15668) 2025-08-29 22:04:08 +02:00
Johannes GäßlerandGitHub fbef0fad7a server: higher timeout for tests (#15621) 2025-08-27 20:58:09 +02:00
Johannes GäßlerandGitHub 8f5afa94c4 CUDA: return -1 for nonexistent compiled arch (#15587) 2025-08-26 16:01:20 +02:00
Johannes GäßlerandGitHub 5eff6ec9b1 CUDA: MoE helper in device code, better tile sizes (#15525) 
* CUDA: MoE helper in device code, better tile sizes

* reduce superfluous CUDA blocks
 2025-08-25 17:23:40 +02:00
Johannes GäßlerandGitHub 710dfc465a CUDA: fix half2 -> half conversion for HIP (#15529) 2025-08-23 21:37:06 +02:00
Johannes GäßlerandGitHub 9ef536907d scripts: fix compare-llama-bench.py (#15521) 2025-08-23 13:58:58 +03:00
Johannes GäßlerandGitHub e92734d51b test-opt: allow slight inprecision (#15503) 2025-08-22 23:47:01 +02:00
Johannes GäßlerandGitHub 13aeb7aef2 CUDA: refactor FA support/selection code (#15454) 2025-08-20 23:14:14 +02:00
Johannes GäßlerandGitHub 7a6e91ad26 CUDA: replace GGML_CUDA_F16 with CUDA arch checks (#15433) 2025-08-20 16:58:49 +02:00
Johannes GäßlerGitHubGeorgi Gerganov
 		b07791aa1d test-opt: fix backend support check (#15317) 
* test-opt: fix backend support check

* Update tests/test-opt.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-08-15 11:23:17 +02:00
Johannes GäßlerandGitHub 4227c9be42 CUDA: fix negative KV_max values in FA (#15321) 2025-08-14 23:21:24 +02:00
Johannes GäßlerGitHubSigbjørn Skjæret
 		4850b52aed server-bench: external OAI servers, sqlite (#15179) 
* server-bench: external OAI servers, sqlite

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* raise_for_status

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2025-08-08 23:04:36 +02:00
Johannes GäßlerandGitHub 1425f587a8 CUDA: attention sinks for mma FlashAttention (#15157) 2025-08-08 08:19:58 +02:00
Johannes GäßlerandGitHub 1d72c84188 CUDA: GEMM for FP32/FP16/BF16 and ne11 <= 16 (#15131) 
* CUDA: GEMM for FP32/FP16/BF16 and ne11 <= 16
 2025-08-07 10:53:21 +02:00
Johannes GäßlerandGitHub 20638e4f16 scripts: fix crash when --tool is not set (#15133) 2025-08-07 08:50:30 +02:00
Johannes GäßlerandGitHub 03d4698218 CUDA: use mma FA kernel for gqa > 4 on RTX 4000 (#15035) 2025-08-02 16:37:08 +02:00
Johannes GäßlerandGitHub f906275537 server: enable token array inputs for OAI API (#15001) 2025-08-02 10:12:41 +02:00
Johannes GäßlerandGitHub 9c35706b98 CUDA: fix MMQ nwarps for AMD with warp_size==32 (#15014) 2025-08-01 20:47:32 +02:00
Johannes GäßlerandGitHub 92b8810ec7 CUDA: skip masked KV slices for all FA kernels (#14924) 2025-07-30 15:46:13 +02:00
Johannes GäßlerGitHubSigbjørn Skjæret
 		bbd0f91779 server-bench: make seed choice configurable (#14929) 
* server-bench: make seed choice configurable

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* fix error formatting

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2025-07-29 10:40:50 +02:00
Johannes GäßlerandGitHub 946b1f6859 CUDA: fix pointer incrementation in FA (#14916) 2025-07-28 14:30:22 +02:00
Johannes GäßlerandGitHub a86f52b285 CUDA: fix overflow in FA, tune performance (#14840) 2025-07-23 21:43:25 +02:00
Johannes GäßlerandGitHub b284197df4 CUDA: fix compilation with GGML_CUDA_F16 (#14837) 2025-07-23 18:22:30 +02:00
Johannes GäßlerandGeorgi Gerganov 07a19e27a2 CUDA: fix quantized KV cache + multiple sequences (#14822) 
* CUDA: fix quantized KV cache + multiple sequences

* Update ggml/src/ggml-cuda/fattn-common.cuh

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-07-23 14:08:09 +03:00
Johannes GäßlerandGitHub 5cae766541 scripts: synthetic prompt mode for server-bench.py (#14695) 2025-07-16 09:33:28 +02:00
Johannes GäßlerandGitHub 494c5899cb scripts: benchmark for HTTP server throughput (#14668) 
* scripts: benchmark for HTTP server throughput

* fix server connection reset
 2025-07-14 13:14:30 +02:00
Johannes GäßlerandGitHub c8c4495b8d ggml: backward pass for split swiglu (#14483) 2025-07-03 17:05:18 +02:00