nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity
6,268 Commits 1 Branch 0 Tags
0d5a470223fc90b6b6807921d68011ff06ae7f9e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Johannes GäßlerandGitHub 710dfc465a CUDA: fix half2 -> half conversion for HIP (#15529) 2025-08-23 21:37:06 +02:00
Johannes GäßlerandGitHub 9ef536907d scripts: fix compare-llama-bench.py (#15521) 2025-08-23 13:58:58 +03:00
Johannes GäßlerandGitHub e92734d51b test-opt: allow slight inprecision (#15503) 2025-08-22 23:47:01 +02:00
Johannes GäßlerandGitHub 13aeb7aef2 CUDA: refactor FA support/selection code (#15454) 2025-08-20 23:14:14 +02:00
Johannes GäßlerandGitHub 7a6e91ad26 CUDA: replace GGML_CUDA_F16 with CUDA arch checks (#15433) 2025-08-20 16:58:49 +02:00
Johannes GäßlerGitHubGeorgi Gerganov
 		b07791aa1d test-opt: fix backend support check (#15317) 
* test-opt: fix backend support check

* Update tests/test-opt.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-08-15 11:23:17 +02:00
Johannes GäßlerandGitHub 4227c9be42 CUDA: fix negative KV_max values in FA (#15321) 2025-08-14 23:21:24 +02:00
Johannes GäßlerGitHubSigbjørn Skjæret
 		4850b52aed server-bench: external OAI servers, sqlite (#15179) 
* server-bench: external OAI servers, sqlite

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* raise_for_status

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2025-08-08 23:04:36 +02:00
Johannes GäßlerandGitHub 1425f587a8 CUDA: attention sinks for mma FlashAttention (#15157) 2025-08-08 08:19:58 +02:00
Johannes GäßlerandGitHub 1d72c84188 CUDA: GEMM for FP32/FP16/BF16 and ne11 <= 16 (#15131) 
* CUDA: GEMM for FP32/FP16/BF16 and ne11 <= 16
 2025-08-07 10:53:21 +02:00
Johannes GäßlerandGitHub 20638e4f16 scripts: fix crash when --tool is not set (#15133) 2025-08-07 08:50:30 +02:00
Johannes GäßlerandGitHub 03d4698218 CUDA: use mma FA kernel for gqa > 4 on RTX 4000 (#15035) 2025-08-02 16:37:08 +02:00
Johannes GäßlerandGitHub f906275537 server: enable token array inputs for OAI API (#15001) 2025-08-02 10:12:41 +02:00
Johannes GäßlerandGitHub 9c35706b98 CUDA: fix MMQ nwarps for AMD with warp_size==32 (#15014) 2025-08-01 20:47:32 +02:00
Johannes GäßlerandGitHub 92b8810ec7 CUDA: skip masked KV slices for all FA kernels (#14924) 2025-07-30 15:46:13 +02:00
Johannes GäßlerGitHubSigbjørn Skjæret
 		bbd0f91779 server-bench: make seed choice configurable (#14929) 
* server-bench: make seed choice configurable

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* fix error formatting

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2025-07-29 10:40:50 +02:00
Johannes GäßlerandGitHub 946b1f6859 CUDA: fix pointer incrementation in FA (#14916) 2025-07-28 14:30:22 +02:00
Johannes GäßlerandGitHub a86f52b285 CUDA: fix overflow in FA, tune performance (#14840) 2025-07-23 21:43:25 +02:00
Johannes GäßlerandGitHub b284197df4 CUDA: fix compilation with GGML_CUDA_F16 (#14837) 2025-07-23 18:22:30 +02:00
Johannes GäßlerandGeorgi Gerganov 07a19e27a2 CUDA: fix quantized KV cache + multiple sequences (#14822) 
* CUDA: fix quantized KV cache + multiple sequences

* Update ggml/src/ggml-cuda/fattn-common.cuh

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-07-23 14:08:09 +03:00
Johannes GäßlerandGitHub 5cae766541 scripts: synthetic prompt mode for server-bench.py (#14695) 2025-07-16 09:33:28 +02:00
Johannes GäßlerandGitHub 494c5899cb scripts: benchmark for HTTP server throughput (#14668) 
* scripts: benchmark for HTTP server throughput

* fix server connection reset
 2025-07-14 13:14:30 +02:00
Johannes GäßlerandGitHub c8c4495b8d ggml: backward pass for split swiglu (#14483) 2025-07-03 17:05:18 +02:00
Johannes GäßlerandGeorgi Gerganov 12a81af45f CUDA: broadcasting for FlashAttention mask (#14500) 2025-07-02 15:48:33 +03:00
Johannes GäßlerandGitHub defe2158dd CUDA: mul_mat_v support for batch sizes > 1 (#14262) 
* CUDA: mul_mat_v support for batch sizes > 1

* use 64 bit math for initial offset calculation
 2025-06-23 13:11:31 +02:00
Johannes GäßlerandGitHub 0b4be4c435 CUDA: fix FTZ in FA for Gemma 3 (#13991) 2025-06-04 08:57:05 +02:00
Johannes GäßlerandGitHub 7675c555a1 gguf: fix failure on version == 0 (#13956) 2025-06-01 18:08:05 +02:00
Johannes GäßlerandGitHub e562eece7c CUDA: fix typo in FlashAttention code (#13926) 2025-05-30 21:22:03 +02:00
Johannes GäßlerandGitHub a68247439b CUDA: fix FA tg at long context for CC >= 8.9 (#13852) 2025-05-28 13:33:37 +02:00
Johannes GäßlerandGitHub ffd0eae60b CUDA: fix race condition in FA vector kernels (#13742) 2025-05-24 11:46:19 +02:00
Johannes GäßlerandGitHub b69f1647f9 CUDA: skip fully masked-out KV in FA vec kernel (#13584) 
* CUDA: skip fully masked-out KV in FA vec kernel
 2025-05-20 14:45:07 +02:00
Johannes GäßlerandGeorgi Gerganov 6c35981a64 mnist: fix segmentation fault (ggml/1227) 2025-05-19 13:29:56 +03:00
Johannes GäßlerandGitHub 4696d56749 CUDA: fix crash on large batch size for quant. MoE (#13537) 2025-05-14 16:41:02 +02:00
Johannes GäßlerandGitHub 6da34fa276 CUDA: faster Deepseek FA, add Turing support (#13435) 2025-05-14 16:08:20 +02:00
Johannes GäßlerandGitHub 10d2af0eaa llama/ggml: add LLM training support (#10544) 
* llama/ggml: add LLM training support

more compact progress bar

llama_save_model_to_file

llama_opt_param_filter

ggml_graph_dup force_grads

refactor ggml_opt, fix test-opt

* remove logits_all

* refactor CUDA implementation for ACC

* reset graph at beginning of opt period
 2025-05-12 14:44:49 +02:00
Johannes GäßlerandGitHub 95e18884fc CUDA: fix misaligned synchronization in FA (#13469) 2025-05-12 10:51:21 +02:00
Johannes GäßlerandGitHub 7474e00b34 CUDA: fix crash with partial offloading of MoE (#13439) 2025-05-11 16:09:33 +02:00
Johannes GäßlerandGitHub 0208355f42 CUDA: fix race conditions FlashAttention kernels (#13438) 2025-05-10 22:22:48 +02:00
Johannes GäßlerandGitHub d8919424f1 CUDA: fix FlashAttention on Turing (#13415) 2025-05-10 09:16:52 +02:00
Johannes GäßlerandGitHub 0cf6725e9f CUDA: FA support for Deepseek (Ampere or newer) (#13306) 
* CUDA: FA support for Deepseek (Ampere or newer)

* do loop unrolling via C++ template
 2025-05-09 13:34:58 +02:00
Johannes GäßlerandGitHub 5c86c9ed3e CUDA: fix crash on large batch size for MoE models (#13384) 2025-05-09 12:14:04 +02:00
Johannes GäßlerandGitHub 141a908a59 CUDA: mix virt/real CUDA archs for GGML_NATIVE=OFF (#13135) 2025-05-06 23:35:51 +02:00
Johannes GäßlerandGitHub 2356fb1d53 CUDA: fix bad asserts for partial offload (#13337) 2025-05-06 13:58:51 +02:00
Johannes GäßlerandGitHub 15a28ec8c7 CUDA: fix --split-mode row for MMQ (#13323) 2025-05-06 08:36:46 +02:00
Johannes GäßlerandGitHub 9070365020 CUDA: fix logic for clearing padding with -ngl 0 (#13320) 2025-05-05 22:32:13 +02:00
Johannes GäßlerandGitHub 93c4e23905 CUDA: fix race condition in MMQ stream-k fixup (#13299) 2025-05-04 14:16:39 +02:00
Johannes GäßlerandGitHub 8afbd96818 CUDA: fix race condition in MMQ ids_dst (#13294) 2025-05-04 13:58:38 +02:00
Johannes GäßlerandGitHub 3e959f0976 imatrix: fix oob writes if src1 is not contiguous (#13286) 2025-05-04 00:50:37 +02:00
Johannes GäßlerandGitHub b0ecbd434b test: non-cont. b in test-backend-ops -o MUL_MAT (#13187) 2025-05-01 20:18:56 +02:00
Johannes GäßlerandGitHub e1e8e0991f CUDA: batched+noncont MMQ, refactor bs>1 MoE code (#13199) 2025-04-30 23:12:59 +02:00
Johannes GäßlerandGitHub 19e899ce21 scripts: n_depth for compare-llama-bench [no ci] (#13201) 2025-04-29 23:32:04 +02:00
Johannes GäßlerandGitHub cdf76586b2 CUDA: fix non-cont. inputs for batched mat mul (#13155) 2025-04-29 16:00:27 +02:00
Johannes GäßlerandGitHub 69699be48a CUDA: fix q_nope_absorbed prec for DS 2 Lite f16 (#13137) 2025-04-28 09:29:26 +02:00
Johannes GäßlerandGitHub b10d8bfdb1 CUDA: use switch statements in constexpr functions (#13095) 2025-04-24 15:57:10 +02:00
Johannes GäßlerandGitHub 658987cfc9 CUDA: noncont MMVQ + batched bs1 MUL_MAT_ID (#13014) 
* CUDA: noncont MMVQ + batched bs1 MUL_MAT_ID

* fix logic for RoPE support, CUDA graphs
 2025-04-22 21:27:40 +02:00
Johannes GäßlerandGitHub dd373dd3bf llama: fix error on bad grammar (#12628) 2025-03-28 18:08:52 +01:00
Johannes GäßlerandGitHub 5220a16d18 CUDA: fix FA logic for PTX 7.0 and CC >= 7.5 (#12222) 2025-03-06 18:45:09 +01:00
Johannes GäßlerandGitHub 9c42b1718c CUDA: fix logic for V100 + GGML_CUDA_FORCE_MMQ (#12098) 2025-02-28 09:26:43 +01:00
Johannes GäßlerandGitHub a28e0d5eb1 CUDA: app option to compile without FlashAttention (#12025) 2025-02-22 20:44:34 +01:00
Johannes GäßlerandGitHub 5fa07c2f93 CUDA: optimize FA for GQA + large batches (#12014) 2025-02-22 12:20:17 +01:00
Johannes GäßlerandGitHub 586d5fe6eb doc: update contributing guidelines [no ci] (#11969) 2025-02-21 12:51:25 +01:00
Johannes GäßlerandGitHub d04e7163c8 doc: add links to ggml examples [no ci] (#11958) 2025-02-19 20:45:17 +01:00
Johannes GäßlerGitHubDiego Devesa
 		73e2ed3ce3 CUDA: use async data loading for FlashAttention (#11894) 
* CUDA: use async data loading for FlashAttention

---------

Co-authored-by: Diego Devesa <slarengh@gmail.com>
 2025-02-17 14:03:24 +01:00
Johannes GäßlerandGitHub 6dde178248 scripts: fix compare-llama-bench commit hash logic (#11891) 2025-02-15 20:23:22 +01:00
Johannes GäßlerandGitHub c3d6af7cd2 CUDA: fix CUDART_VERSION checks (#11821) 2025-02-12 13:16:39 +01:00
Johannes GäßlerGitHubDiego Devesa
 		b9ab0a4d0b CUDA: use arch list for compatibility check (#11775) 
* CUDA: use arch list for feature availability check

---------

Co-authored-by: Diego Devesa <slarengh@gmail.com>
 2025-02-11 00:17:22 +01:00
Johannes GäßlerandGitHub d80be897ac CUDA: fix min. version for movmatrix (#11751) 2025-02-08 10:46:07 +01:00
Johannes GäßlerandGitHub fa62da9b2d CUDA: support for mat. mul. with ne03 != ne13 (#11656) 2025-02-05 08:58:31 +01:00
Johannes GäßlerGitHubGeorgi Gerganov
 		fd08255d0d CUDA: non-contiguous (RMS) norm support (#11659) 
* CUDA: non-contiguous (RMS) norm support

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-02-04 22:21:42 +01:00
Johannes GäßlerandGitHub 21c84b5d2d CUDA: fix Volta FlashAttention logic (#11615) 2025-02-03 14:25:56 +02:00
Johannes GäßlerandGitHub 6eecde3cc8 HIP: fix flash_attn_stream_k_fixup warning (#11604) 2025-02-02 23:48:29 +01:00
Johannes GäßlerGitHubDiego Devesa
 		864a0b67a6 CUDA: use mma PTX instructions for FlashAttention (#11583) 
* CUDA: use mma PTX instructions for FlashAttention

* __shfl_sync workaround for movmatrix

* add __shfl_sync to HIP

Co-authored-by: Diego Devesa <slarengh@gmail.com>
 2025-02-02 19:31:09 +01:00
Johannes GäßlerandGitHub df984e0147 llama: refactor llama_decode_impl (#11381) 2025-01-27 12:07:12 +01:00
Johannes GäßlerandGitHub c5d9effb49 CUDA: fix FP16 cuBLAS GEMM (#11396) 2025-01-24 21:02:43 +01:00
Johannes GäßlerandGitHub 8137b4bb2b CPU/CUDA: fix (GQA) mul mat back, add CUDA support (#11380) 2025-01-24 12:38:31 +01:00
Johannes GäßlerandGitHub 9c8dcefe17 CUDA: backwards pass for misc. ops, add tests (#11257) 
* CUDA: backwards pass for misc. ops, add tests

* remove restrict from pointers
 2025-01-16 16:43:38 +01:00
Johannes GäßlerandGitHub 432df2d5f9 RoPE: fix back, CUDA support for back + noncont. (#11240) 
* RoPE: fix back, CUDA support for back + noncont.

* fix comments reg. non-cont. RoPE support [no-ci]
 2025-01-15 12:51:37 +01:00
Johannes GäßlerandGitHub 53ff6b9b9f GGUF: C++ refactor, backend support, misc fixes (#11030) 
* GGUF: C++ refactor, backend support, misc fixes

remove ggml_tensor.backend

update CODEOWNERS [no ci]

remove gguf_get_data from API

revise GGUF API data types
 2025-01-07 18:01:58 +01:00
Johannes GäßlerandGitHub 46e3556e01 CUDA: add BF16 support (#11093) 
* CUDA: add BF16 support
 2025-01-06 02:33:52 +01:00
Johannes GäßlerandGitHub cd920d0ac3 tests: disable GGUF test for bad value size (#10886) 2024-12-19 08:53:58 +01:00
Johannes GäßlerandGitHub 081b29bd2a tests: add tests for GGUF (#10830) 2024-12-17 19:09:35 +01:00
Johannes GäßlerandGitHub 26a8406ba9 CUDA: fix shared memory access condition for mmv (#10740) 2024-12-09 20:07:12 +01:00
Johannes GäßlerandGitHub 890719311b common: fix warning message when no GPU found (#10564) 2024-11-28 18:15:25 +01:00
Johannes GäßlerandGitHub 1f922254f0 Github: update issue templates [no ci] (#10489) 2024-11-25 19:18:37 +01:00
Johannes GäßlerandGitHub 599b3e0cd4 GitHub: ask for more info in issue templates (#10426) 
* GitHub: ask for more info in issues [no ci]

* refactor issue templates to be component-specific

* more understandable issue description

* add dropdown for llama.cpp module
 2024-11-22 08:32:40 +01:00
Johannes GäßlerandGeorgi Gerganov 02e4eaf22f ggml-opt: fix data corruption (ggml/1022) 2024-11-21 09:22:02 +02:00
Johannes GäßlerandGitHub 75207b3a88 docker: use GGML_NATIVE=OFF (#10368) 2024-11-18 00:21:53 +01:00
Johannes GäßlerandGitHub 76e9e58b78 CUDA: fix MMV kernel being used for FP16 src1 (#10357) 2024-11-17 23:20:42 +01:00
Johannes GäßlerandGitHub ce2e59ba10 CMake: fix typo in comment [no ci] (#10360) 2024-11-17 12:59:38 +01:00
Johannes GäßlerandGitHub c3ea58aca4 CUDA: remove DMMV, consolidate F16 mult mat vec (#10318) 2024-11-17 09:09:55 +01:00
Johannes GäßlerandGitHub 467576b6cc CMake: default to -arch=native for CUDA build (#10320) 2024-11-17 09:06:34 +01:00
Johannes GäßlerandGeorgi Gerganov 8a43e940ab ggml: new optimization interface (ggml/988) 2024-11-17 08:30:29 +02:00
Johannes GäßlerandGitHub 4e54be0ec6 llama/ex: remove --logdir argument (#10339) 2024-11-16 23:00:41 +01:00
Johannes GäßlerandGitHub 4047be74da scripts: update compare-llama-bench.py (#10319) 2024-11-15 21:19:03 +01:00
Johannes GäßlerandGitHub 4a8ccb37ad CUDA: no -sm row for very small matrices (#10185) 2024-11-14 13:00:15 +01:00
Johannes GäßlerandGitHub 167a515651 CUDA: fix insufficient buffer clearing for MMQ (#10032) 2024-10-24 14:40:23 +02:00
Johannes GäßlerandGitHub c39665f589 CUDA: fix MMQ for non-contiguous src0, add tests (#10021) 
* CUDA: fix MMQ for non-contiguous src0, add tests

* revise test code
 2024-10-24 11:09:36 +02:00
Johannes GäßlerandGeorgi Gerganov 80273a306d CUDA: fix 1D im2col, add tests (ggml/993) 2024-10-23 16:50:02 +03:00
Johannes GäßlerandGeorgi Gerganov fabdc3bda3 ggml/ex: calculate accuracy in graph, adapt MNIST (ggml/980) 2024-10-03 21:17:26 +03:00
Johannes GäßlerandGeorgi Gerganov eee39bdc96 ggml: refactor cross entropy loss CPU impl. (ggml/976) 2024-10-03 21:17:26 +03:00