nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity
7,521 Commits 1 Branch 0 Tags
8e3ead6e4d1a4c455c8dac1b4690c49366769856
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Johannes GäßlerandGitHub 147a521636 tool/ex/tests: consistently free ctx, then model (#18168) 2025-12-22 11:00:37 +01:00
Johannes GäßlerandGitHub 0e1ccf15c7 llama: fix RPC for -fit on (#18233) 2025-12-21 19:33:08 +01:00
Johannes GäßlerandGitHub 57c1e05643 llama: offload output layer to GPU first (#18148) 2025-12-18 08:12:18 +01:00
Johannes GäßlerandGitHub 8dcc3662a2 llama-fit-params: fix memory print (#18136) 2025-12-17 21:10:03 +01:00
Johannes GäßlerandGitHub a2c199e479 common: clarify instructions for bug reports (#18134) 2025-12-17 18:44:13 +01:00
Johannes GäßlerandGitHub 6f1f6a961a Github: ask for -v logs for params_fit [no ci] (#18128) 2025-12-17 13:46:48 +01:00
Johannes GäßlerandGitHub d0794e89d9 llama-fit-params: force disable mlock (#18103) 2025-12-17 00:50:12 +01:00
Johannes GäßlerandGitHub 9dcac6cf9f llama-fit-params: lower ctx size for multi GPU (#18101) 2025-12-17 00:49:34 +01:00
Johannes GäßlerandGitHub 0e49a7b8b4 llama-fit-params: fix underflow for dense models (#18095) 2025-12-17 00:47:37 +01:00
Johannes GäßlerandGitHub 4164596c76 llama-fit-params: QoL impr. for prints/errors (#18089) 2025-12-17 00:03:19 +01:00
Johannes GäßlerandGitHub ec98e20021 llama: fix early stop in params_fit if ctx is set (#18070) 2025-12-16 14:24:00 +01:00
Johannes GäßlerandGitHub b1f3a6e5db llama: automatically set parameters not set by the user in such a way that maximizes GPU utilization (#16653) 
* llama: automatically fit args to free memory

llama-fit-params tool

* fix CI

* hints for bug reports, ensure no reallocation

* fix segfault with Vulkan

* add llama-fit-params to CI

* fix CI

* fix CI

* fix CI

* minor adjustments

* fix assignment of 1 dense layer

* fix logger not being reset on model load failure

* remove --n-gpu-layer hint on model load failure

* fix llama-fit-params verbosity

* fix edge case

* fix typo [no ci]
 2025-12-15 09:24:59 +01:00
Johannes GäßlerandGitHub 482211438d CUDA: fix overflow in MMA kernel without stream-k (#17939) 2025-12-12 17:43:58 +01:00
Johannes GäßlerandGitHub 17f7f4baad CUDA: fix unpadded strides in MMA FA kernel (#17891) 2025-12-10 12:39:56 +01:00
Johannes GäßlerandGitHub 48f47565a7 docs: clarify that CPU support should be first (#17886) 2025-12-09 20:10:36 +01:00
Johannes GäßlerandGitHub 0cdce38a97 CUDA: fix FP16 overflow in tile FA kernel (#17875) 2025-12-09 09:34:02 +01:00
Johannes GäßlerandGitHub f334b79494 HIP: fix RDNA3 FP16/BF16 matrix multiplication (#17817) 2025-12-06 13:45:36 +01:00
Johannes GäßlerandGitHub 6016d0bd41 HIP : fix RDNA4 build (#17792) 2025-12-05 13:47:52 +01:00
Johannes GäßlerandGitHub e95d0bc8fd CUDA: fix FA VKQ accumulator overflow (#17746) 2025-12-05 09:18:10 +01:00
Johannes GäßlerGitHubAman Gupta <aman>
 		2e1c9cd814 CUDA: generalized (mma) FA, add Volta support (#17505) 
* CUDA: generalized (mma) FA, add Volta support

* use struct for MMA FA kernel config

---------

Co-authored-by: Aman Gupta <aman>
 2025-12-03 16:57:05 +01:00
Johannes GäßlerandGitHub 73955f7d2a CUDA: no FP16 arithmetic for vector FA kernel (#17558) 2025-11-28 10:29:09 +01:00
Johannes GäßlerandGitHub 5d6838b74f CUDA: static assert to prevent misuse of memcpy_1 (#17198) 2025-11-12 23:13:55 +01:00
Johannes GäßlerandGitHub e14e842e87 CUDA: fix MMQ stream-k fixup ne1 indices (#17089) 2025-11-08 08:26:18 +01:00
Johannes GäßlerGitHubAman Gupta
 		6515610506 CUDA: fix should_use_mmvf for ne11 == 1 (#17085) 
* CUDA: fix should_use_mmvf for ne11 == 1

* Apply suggestion from @am17an

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
 2025-11-07 20:53:14 +01:00
Johannes GäßlerandGitHub aa374175c3 CUDA: fix crash on uneven context without FA (#16988) 2025-11-06 14:05:47 +01:00
Johannes GäßlerandGitHub 22c8c3c6ad docs: explain CUDA 11 compilation [no ci] (#16824) 2025-11-06 08:14:35 +01:00
Johannes GäßlerGitHubAman Gupta
 		31c511a968 CUDA: Volta tensor core support for MMF (#16843) 
* CUDA: Volta tensor core support for MMF

* more generic checks for hardware support

* Update ggml/src/ggml-cuda/mmf.cuh

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
 2025-10-31 15:57:19 +01:00
Johannes GäßlerandGitHub 7a0e900e36 llama: consistent ctx <-> buf order for KV cache (#16746) 2025-10-28 11:23:54 +01:00
Johannes GäßlerandGitHub 80d28f104c HIP: fix AMDGPU_TARGETS, update documentation (#16803) 2025-10-27 21:39:49 +01:00
Johannes GäßlerandGitHub 945501f5ea llama: fix leaked buffers for mmap + split files (#16765) 2025-10-27 09:17:31 +01:00
Johannes GäßlerandGitHub 0bf47a1dbb server: add memory breakdown print (#16740) 2025-10-23 21:30:17 +02:00
Johannes GäßlerandGitHub 51d1a8c997 CUDA: better error for FA kernel with 0 occupancy (#16643) 2025-10-21 15:27:53 +02:00
Johannes GäßlerandGitHub ee09828cb0 HIP: fix GPU_TARGETS (#16642) 2025-10-18 14:47:32 +02:00
Johannes GäßlerandGitHub 66b0dbcb2d llama-model: fix insonsistent ctxs <-> bufs order (#16581) 2025-10-17 17:41:09 +02:00
Johannes GäßlerandGitHub 9c7185dd28 CUDA: enable FA for FP32 KV cache (#16546) 2025-10-14 14:22:47 +02:00
Johannes GäßlerandGitHub 7049736b2d CUDA: fix numerical issues in tile FA kernel (#16540) 2025-10-13 17:29:45 +03:00
Johannes GäßlerandGitHub 11f0af5504 CUDA: faster tile FA, add oob checks, more HSs (#16492) 2025-10-11 20:54:32 +02:00
Johannes GäßlerandGitHub 75a3a6c2cd CUDA: refactor and deduplicate vector FA kernels (#16208) 
* CUDA: refactor and deduplicate vector FA kernels
 2025-09-27 18:45:07 +02:00
Johannes GäßlerandGitHub 4cdd0bb453 docs: fix typo [no ci] (#16244) 2025-09-25 12:12:27 +03:00
Johannes GäßlerandGitHub e789095502 llama: print memory breakdown on exit (#15860) 
* llama: print memory breakdown on exit
 2025-09-24 16:53:48 +02:00
Johannes GäßlerandGitHub 368560a1e3 CUDA: fix compilation on CC 6.0 (#16091) 2025-09-18 19:28:32 +02:00
Johannes GäßlerandGitHub c959b676be CUDA: fix FA occupancy, optimize tile kernel (#15982) 2025-09-17 15:32:42 +02:00
Johannes GäßlerandGitHub 0e6ff0046f CUDA: larger SRAM reads for tile FA, AMD FP16 dot (#15927) 
* CUDA: larger SRAM reads for tile FA, AMD FP16 dot

* fix logic for availability of v_dot2_f32_f16
 2025-09-11 21:19:58 +02:00
Johannes GäßlerandGitHub 17bc5a815f HIP: use v_dot2_f32_f16 instruction for FA (#15884) 2025-09-09 14:04:43 +02:00
Johannes GäßlerandGitHub 550cf726e1 CUDA: fix GET_ROWS for large tensors (#15882) 2025-09-09 08:11:01 +02:00
Johannes GäßlerandGitHub 79bc429262 CUDA: faster tile FA (Pascal/AMD), headsize 256 (#15769) 2025-09-07 00:26:28 +02:00
Johannes GäßlerandGitHub 01806e7771 ggml-cpu: document use of "free" memory [no ci] (#15834) 2025-09-06 13:28:44 +02:00
Johannes GäßlerandGitHub 5143fa895e CUDA: fastdiv, launch bounds for mmvq + q8_1 quant (#15802) 
* CUDA: fastdiv, launch bounds for mmvq + q8_1 quant
 2025-09-05 16:07:02 +02:00
Johannes GäßlerandGitHub c466abe158 llama: -fa 1/0/-1 aliases for -fa on/off/auto (#15746) 2025-09-02 18:17:26 +02:00
Johannes GäßlerandGitHub 5d804a4938 ggml-backend: raise GGML_MAX_SPLIT_INPUTS (#15722) 2025-09-01 16:14:55 -07:00
Johannes GäßlerandGitHub e81b8e4b7f llama: use FA + max. GPU layers by default (#15434) 
* llama: use max. GPU layers by default, auto -fa

* ggml-backend: abort instead of segfault
 2025-08-30 16:32:10 +02:00
Johannes GäßlerandGitHub 38ad381f9f CUDA: use FP32 arithmetic for conv2d (#15683) 2025-08-30 16:20:32 +02:00
Johannes GäßlerandGitHub 3d16b29c3b scripts: strip "AMD Instinct" from GPU name (#15668) 2025-08-29 22:04:08 +02:00
Johannes GäßlerandGitHub fbef0fad7a server: higher timeout for tests (#15621) 2025-08-27 20:58:09 +02:00
Johannes GäßlerandGitHub 8f5afa94c4 CUDA: return -1 for nonexistent compiled arch (#15587) 2025-08-26 16:01:20 +02:00
Johannes GäßlerandGitHub 5eff6ec9b1 CUDA: MoE helper in device code, better tile sizes (#15525) 
* CUDA: MoE helper in device code, better tile sizes

* reduce superfluous CUDA blocks
 2025-08-25 17:23:40 +02:00
Johannes GäßlerandGitHub 710dfc465a CUDA: fix half2 -> half conversion for HIP (#15529) 2025-08-23 21:37:06 +02:00
Johannes GäßlerandGitHub 9ef536907d scripts: fix compare-llama-bench.py (#15521) 2025-08-23 13:58:58 +03:00
Johannes GäßlerandGitHub e92734d51b test-opt: allow slight inprecision (#15503) 2025-08-22 23:47:01 +02:00
Johannes GäßlerandGitHub 13aeb7aef2 CUDA: refactor FA support/selection code (#15454) 2025-08-20 23:14:14 +02:00
Johannes GäßlerandGitHub 7a6e91ad26 CUDA: replace GGML_CUDA_F16 with CUDA arch checks (#15433) 2025-08-20 16:58:49 +02:00
Johannes GäßlerGitHubGeorgi Gerganov
 		b07791aa1d test-opt: fix backend support check (#15317) 
* test-opt: fix backend support check

* Update tests/test-opt.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-08-15 11:23:17 +02:00
Johannes GäßlerandGitHub 4227c9be42 CUDA: fix negative KV_max values in FA (#15321) 2025-08-14 23:21:24 +02:00
Johannes GäßlerGitHubSigbjørn Skjæret
 		4850b52aed server-bench: external OAI servers, sqlite (#15179) 
* server-bench: external OAI servers, sqlite

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* raise_for_status

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2025-08-08 23:04:36 +02:00
Johannes GäßlerandGitHub 1425f587a8 CUDA: attention sinks for mma FlashAttention (#15157) 2025-08-08 08:19:58 +02:00
Johannes GäßlerandGitHub 1d72c84188 CUDA: GEMM for FP32/FP16/BF16 and ne11 <= 16 (#15131) 
* CUDA: GEMM for FP32/FP16/BF16 and ne11 <= 16
 2025-08-07 10:53:21 +02:00
Johannes GäßlerandGitHub 20638e4f16 scripts: fix crash when --tool is not set (#15133) 2025-08-07 08:50:30 +02:00
Johannes GäßlerandGitHub 03d4698218 CUDA: use mma FA kernel for gqa > 4 on RTX 4000 (#15035) 2025-08-02 16:37:08 +02:00
Johannes GäßlerandGitHub f906275537 server: enable token array inputs for OAI API (#15001) 2025-08-02 10:12:41 +02:00
Johannes GäßlerandGitHub 9c35706b98 CUDA: fix MMQ nwarps for AMD with warp_size==32 (#15014) 2025-08-01 20:47:32 +02:00
Johannes GäßlerandGitHub 92b8810ec7 CUDA: skip masked KV slices for all FA kernels (#14924) 2025-07-30 15:46:13 +02:00
Johannes GäßlerGitHubSigbjørn Skjæret
 		bbd0f91779 server-bench: make seed choice configurable (#14929) 
* server-bench: make seed choice configurable

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* fix error formatting

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2025-07-29 10:40:50 +02:00
Johannes GäßlerandGitHub 946b1f6859 CUDA: fix pointer incrementation in FA (#14916) 2025-07-28 14:30:22 +02:00
Johannes GäßlerandGitHub a86f52b285 CUDA: fix overflow in FA, tune performance (#14840) 2025-07-23 21:43:25 +02:00
Johannes GäßlerandGitHub b284197df4 CUDA: fix compilation with GGML_CUDA_F16 (#14837) 2025-07-23 18:22:30 +02:00
Johannes GäßlerandGeorgi Gerganov 07a19e27a2 CUDA: fix quantized KV cache + multiple sequences (#14822) 
* CUDA: fix quantized KV cache + multiple sequences

* Update ggml/src/ggml-cuda/fattn-common.cuh

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-07-23 14:08:09 +03:00
Johannes GäßlerandGitHub 5cae766541 scripts: synthetic prompt mode for server-bench.py (#14695) 2025-07-16 09:33:28 +02:00
Johannes GäßlerandGitHub 494c5899cb scripts: benchmark for HTTP server throughput (#14668) 
* scripts: benchmark for HTTP server throughput

* fix server connection reset
 2025-07-14 13:14:30 +02:00
Johannes GäßlerandGitHub c8c4495b8d ggml: backward pass for split swiglu (#14483) 2025-07-03 17:05:18 +02:00
Johannes GäßlerandGeorgi Gerganov 12a81af45f CUDA: broadcasting for FlashAttention mask (#14500) 2025-07-02 15:48:33 +03:00
Johannes GäßlerandGitHub defe2158dd CUDA: mul_mat_v support for batch sizes > 1 (#14262) 
* CUDA: mul_mat_v support for batch sizes > 1

* use 64 bit math for initial offset calculation
 2025-06-23 13:11:31 +02:00
Johannes GäßlerandGitHub 0b4be4c435 CUDA: fix FTZ in FA for Gemma 3 (#13991) 2025-06-04 08:57:05 +02:00
Johannes GäßlerandGitHub 7675c555a1 gguf: fix failure on version == 0 (#13956) 2025-06-01 18:08:05 +02:00
Johannes GäßlerandGitHub e562eece7c CUDA: fix typo in FlashAttention code (#13926) 2025-05-30 21:22:03 +02:00
Johannes GäßlerandGitHub a68247439b CUDA: fix FA tg at long context for CC >= 8.9 (#13852) 2025-05-28 13:33:37 +02:00
Johannes GäßlerandGitHub ffd0eae60b CUDA: fix race condition in FA vector kernels (#13742) 2025-05-24 11:46:19 +02:00
Johannes GäßlerandGitHub b69f1647f9 CUDA: skip fully masked-out KV in FA vec kernel (#13584) 
* CUDA: skip fully masked-out KV in FA vec kernel
 2025-05-20 14:45:07 +02:00
Johannes GäßlerandGeorgi Gerganov 6c35981a64 mnist: fix segmentation fault (ggml/1227) 2025-05-19 13:29:56 +03:00
Johannes GäßlerandGitHub 4696d56749 CUDA: fix crash on large batch size for quant. MoE (#13537) 2025-05-14 16:41:02 +02:00
Johannes GäßlerandGitHub 6da34fa276 CUDA: faster Deepseek FA, add Turing support (#13435) 2025-05-14 16:08:20 +02:00
Johannes GäßlerandGitHub 10d2af0eaa llama/ggml: add LLM training support (#10544) 
* llama/ggml: add LLM training support

more compact progress bar

llama_save_model_to_file

llama_opt_param_filter

ggml_graph_dup force_grads

refactor ggml_opt, fix test-opt

* remove logits_all

* refactor CUDA implementation for ACC

* reset graph at beginning of opt period
 2025-05-12 14:44:49 +02:00
Johannes GäßlerandGitHub 95e18884fc CUDA: fix misaligned synchronization in FA (#13469) 2025-05-12 10:51:21 +02:00
Johannes GäßlerandGitHub 7474e00b34 CUDA: fix crash with partial offloading of MoE (#13439) 2025-05-11 16:09:33 +02:00
Johannes GäßlerandGitHub 0208355f42 CUDA: fix race conditions FlashAttention kernels (#13438) 2025-05-10 22:22:48 +02:00
Johannes GäßlerandGitHub d8919424f1 CUDA: fix FlashAttention on Turing (#13415) 2025-05-10 09:16:52 +02:00
Johannes GäßlerandGitHub 0cf6725e9f CUDA: FA support for Deepseek (Ampere or newer) (#13306) 
* CUDA: FA support for Deepseek (Ampere or newer)

* do loop unrolling via C++ template
 2025-05-09 13:34:58 +02:00
Johannes GäßlerandGitHub 5c86c9ed3e CUDA: fix crash on large batch size for MoE models (#13384) 2025-05-09 12:14:04 +02:00
Johannes GäßlerandGitHub 141a908a59 CUDA: mix virt/real CUDA archs for GGML_NATIVE=OFF (#13135) 2025-05-06 23:35:51 +02:00
Johannes GäßlerandGitHub 2356fb1d53 CUDA: fix bad asserts for partial offload (#13337) 2025-05-06 13:58:51 +02:00
Johannes GäßlerandGitHub 15a28ec8c7 CUDA: fix --split-mode row for MMQ (#13323) 2025-05-06 08:36:46 +02:00