nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 30 Packages Projects Releases Wiki Activity
4,695 Commits 1 Branch 0 Tags
fef0cbeadf5c8e221f832158cb4006ade39ef786
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Johannes GäßlerandGitHub c3d6af7cd2 CUDA: fix CUDART_VERSION checks (#11821) 2025-02-12 13:16:39 +01:00
Johannes GäßlerGitHubDiego Devesa
 		b9ab0a4d0b CUDA: use arch list for compatibility check (#11775) 
* CUDA: use arch list for feature availability check

---------

Co-authored-by: Diego Devesa <slarengh@gmail.com>
 2025-02-11 00:17:22 +01:00
Johannes GäßlerandGitHub d80be897ac CUDA: fix min. version for movmatrix (#11751) 2025-02-08 10:46:07 +01:00
Johannes GäßlerandGitHub fa62da9b2d CUDA: support for mat. mul. with ne03 != ne13 (#11656) 2025-02-05 08:58:31 +01:00
Johannes GäßlerGitHubGeorgi Gerganov
 		fd08255d0d CUDA: non-contiguous (RMS) norm support (#11659) 
* CUDA: non-contiguous (RMS) norm support

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-02-04 22:21:42 +01:00
Johannes GäßlerandGitHub 21c84b5d2d CUDA: fix Volta FlashAttention logic (#11615) 2025-02-03 14:25:56 +02:00
Johannes GäßlerandGitHub 6eecde3cc8 HIP: fix flash_attn_stream_k_fixup warning (#11604) 2025-02-02 23:48:29 +01:00
Johannes GäßlerGitHubDiego Devesa
 		864a0b67a6 CUDA: use mma PTX instructions for FlashAttention (#11583) 
* CUDA: use mma PTX instructions for FlashAttention

* __shfl_sync workaround for movmatrix

* add __shfl_sync to HIP

Co-authored-by: Diego Devesa <slarengh@gmail.com>
 2025-02-02 19:31:09 +01:00
Johannes GäßlerandGitHub df984e0147 llama: refactor llama_decode_impl (#11381) 2025-01-27 12:07:12 +01:00
Johannes GäßlerandGitHub c5d9effb49 CUDA: fix FP16 cuBLAS GEMM (#11396) 2025-01-24 21:02:43 +01:00
Johannes GäßlerandGitHub 8137b4bb2b CPU/CUDA: fix (GQA) mul mat back, add CUDA support (#11380) 2025-01-24 12:38:31 +01:00
Johannes GäßlerandGitHub 9c8dcefe17 CUDA: backwards pass for misc. ops, add tests (#11257) 
* CUDA: backwards pass for misc. ops, add tests

* remove restrict from pointers
 2025-01-16 16:43:38 +01:00
Johannes GäßlerandGitHub 432df2d5f9 RoPE: fix back, CUDA support for back + noncont. (#11240) 
* RoPE: fix back, CUDA support for back + noncont.

* fix comments reg. non-cont. RoPE support [no-ci]
 2025-01-15 12:51:37 +01:00
Johannes GäßlerandGitHub 53ff6b9b9f GGUF: C++ refactor, backend support, misc fixes (#11030) 
* GGUF: C++ refactor, backend support, misc fixes

remove ggml_tensor.backend

update CODEOWNERS [no ci]

remove gguf_get_data from API

revise GGUF API data types
 2025-01-07 18:01:58 +01:00
Johannes GäßlerandGitHub 46e3556e01 CUDA: add BF16 support (#11093) 
* CUDA: add BF16 support
 2025-01-06 02:33:52 +01:00
Johannes GäßlerandGitHub cd920d0ac3 tests: disable GGUF test for bad value size (#10886) 2024-12-19 08:53:58 +01:00
Johannes GäßlerandGitHub 081b29bd2a tests: add tests for GGUF (#10830) 2024-12-17 19:09:35 +01:00
Johannes GäßlerandGitHub 26a8406ba9 CUDA: fix shared memory access condition for mmv (#10740) 2024-12-09 20:07:12 +01:00
Johannes GäßlerandGitHub 890719311b common: fix warning message when no GPU found (#10564) 2024-11-28 18:15:25 +01:00
Johannes GäßlerandGitHub 1f922254f0 Github: update issue templates [no ci] (#10489) 2024-11-25 19:18:37 +01:00
Johannes GäßlerandGitHub 599b3e0cd4 GitHub: ask for more info in issue templates (#10426) 
* GitHub: ask for more info in issues [no ci]

* refactor issue templates to be component-specific

* more understandable issue description

* add dropdown for llama.cpp module
 2024-11-22 08:32:40 +01:00
Johannes GäßlerandGeorgi Gerganov 02e4eaf22f ggml-opt: fix data corruption (ggml/1022) 2024-11-21 09:22:02 +02:00
Johannes GäßlerandGitHub 75207b3a88 docker: use GGML_NATIVE=OFF (#10368) 2024-11-18 00:21:53 +01:00
Johannes GäßlerandGitHub 76e9e58b78 CUDA: fix MMV kernel being used for FP16 src1 (#10357) 2024-11-17 23:20:42 +01:00
Johannes GäßlerandGitHub ce2e59ba10 CMake: fix typo in comment [no ci] (#10360) 2024-11-17 12:59:38 +01:00
Johannes GäßlerandGitHub c3ea58aca4 CUDA: remove DMMV, consolidate F16 mult mat vec (#10318) 2024-11-17 09:09:55 +01:00
Johannes GäßlerandGitHub 467576b6cc CMake: default to -arch=native for CUDA build (#10320) 2024-11-17 09:06:34 +01:00
Johannes GäßlerandGeorgi Gerganov 8a43e940ab ggml: new optimization interface (ggml/988) 2024-11-17 08:30:29 +02:00
Johannes GäßlerandGitHub 4e54be0ec6 llama/ex: remove --logdir argument (#10339) 2024-11-16 23:00:41 +01:00
Johannes GäßlerandGitHub 4047be74da scripts: update compare-llama-bench.py (#10319) 2024-11-15 21:19:03 +01:00
Johannes GäßlerandGitHub 4a8ccb37ad CUDA: no -sm row for very small matrices (#10185) 2024-11-14 13:00:15 +01:00
Johannes GäßlerandGitHub 167a515651 CUDA: fix insufficient buffer clearing for MMQ (#10032) 2024-10-24 14:40:23 +02:00
Johannes GäßlerandGitHub c39665f589 CUDA: fix MMQ for non-contiguous src0, add tests (#10021) 
* CUDA: fix MMQ for non-contiguous src0, add tests

* revise test code
 2024-10-24 11:09:36 +02:00
Johannes GäßlerandGeorgi Gerganov 80273a306d CUDA: fix 1D im2col, add tests (ggml/993) 2024-10-23 16:50:02 +03:00
Johannes GäßlerandGeorgi Gerganov fabdc3bda3 ggml/ex: calculate accuracy in graph, adapt MNIST (ggml/980) 2024-10-03 21:17:26 +03:00
Johannes GäßlerandGeorgi Gerganov eee39bdc96 ggml: refactor cross entropy loss CPU impl. (ggml/976) 2024-10-03 21:17:26 +03:00
Johannes GäßlerandGeorgi Gerganov e98c1c188e test: fix OPT_STEP_ADAMW for test-backend-ops (ggml/974) 2024-10-01 16:07:40 +03:00
Johannes GäßlerandGeorgi Gerganov 7254cdf7e8 ggml: fix gradient allocation logic (ggml/966) 
* ggml: fix gradient allocation logic

* gradient allocation in ggml_build_backward_expand

* fixup

* fix test-backend-ops grad

* suggestions by slaren

* fix test1.c

* fix legacy opt API

* fix test-grad0

* remove keep arg
 2024-10-01 16:07:38 +03:00
Johannes GäßlerandGeorgi Gerganov aaa4099925 CUDA: remove bad assert (ggml/972) 2024-09-29 21:15:37 +03:00
Johannes GäßlerandGitHub a5b57b08ce CUDA: enable Gemma FA for HIP/Pascal (#9581) 2024-09-22 09:34:52 +02:00
Johannes GäßlerGeorgi Gerganovslaren
 		424c5d00a9 ggml/examples: add backend support for numerical optimization (ggml/949) 
* CUDA eval works

* stochastic gradient descent op

* Adam except decay

* CUDA CROSS_ENTROPY_LOSS_BACK

* CUDA mnist-fc training works

* backend CLI arg

* refactor gguf load

* remove sched from opt_step_adam

* implement l1 regularization (weight decay)

* extra call to add optimizer

* initialize gradients with ggml_graph_reset

* gradient accumulation

* increment iter per eval instead of epoch

* adjust backend interfaces

* fix ggml_graph_reset without backend

* fix ggml graph export/import

* fixup

* rename

* revert ggml_opt changes

* more general CUDA repeat_back

* update documentation, fix CNN

* validation split

* add clarifying comment

* optimize PyTorch training

* adjust buffer size, thread count

* fix 0.0f validation split

* Update examples/mnist/mnist-common.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* fix gradient accumulation

* tensor flag for accumulators -> tensor hash set

* Update include/ggml.h

Co-authored-by: slaren <slarengh@gmail.com>

* Update tests/test-backend-ops.cpp

Co-authored-by: slaren <slarengh@gmail.com>

* Update tests/test-backend-ops.cpp

Co-authored-by: slaren <slarengh@gmail.com>

* fix test prints

* Update src/ggml-backend.c

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* better CUDA support for noncontiguous out_prod

* add comment

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: slaren <slarengh@gmail.com>
 2024-09-20 21:15:05 +03:00
Johannes GäßlerandGitHub 5cb12f6839 CUDA: fix sum.cu compilation for CUDA < 11.7 (#9562) 2024-09-20 18:35:35 +02:00
Johannes GäßlerandGitHub 5af118efda CUDA: fix --split-mode row race condition (#9413) 2024-09-11 10:22:40 +02:00
Johannes GäßlerandGitHub 8e6e2fbe14 CUDA: fix variable name conflict for Windows build (#9382) 2024-09-09 14:22:53 +02:00
Johannes GäßlerandGeorgi Gerganov 202084d31d tests: add gradient tests for all backends (ggml/932) 
* tests: add gradient checking to test-backend-ops

* remove old comment

* reorder includes

* adjust SIN/COS parameters

* add documentation, use supports_op if possible
 2024-09-08 11:05:55 +03:00
Johannes GäßlerandGeorgi Gerganov dbbebcab33 ggml: fix ggml_graph_cpy undefined behavior (ggml/943) 2024-09-08 11:05:55 +03:00
Johannes GäßlerandGitHub f91fc5639b CUDA: fix Gemma 2 numerical issues for FA (#9166) 2024-08-25 22:11:48 +02:00
Johannes GäßlerandGitHub e11bd856d5 CPU/CUDA: Gemma 2 FlashAttention support (#8542) 
* CPU/CUDA: Gemma 2 FlashAttention support

* apply logit_softcap to scale in kernel

* disable logit softcapping tests on Metal

* remove metal check
 2024-08-24 21:34:59 +02:00
Johannes GäßlerandGitHub a8dbc6f753 CUDA/HIP: fix tests/test-backend-ops (#8896) 2024-08-07 09:07:52 +02:00
Johannes GäßlerandGitHub 641f5dd2a6 CUDA: fix padding logic for FP16/FP32 (#8884) 2024-08-06 17:13:55 +02:00
Johannes GäßlerandGitHub 6eeaeba126 cmake: use 1 more thread for non-ggml in CI (#8740) 2024-07-28 22:32:44 +02:00
Johannes GäßlerandGitHub 69c487f4ed CUDA: MMQ code deduplication + iquant support (#8495) 
* CUDA: MMQ code deduplication + iquant support

* 1 less parallel job for CI build
 2024-07-20 22:25:26 +02:00
Johannes GäßlerandGitHub a15ef8f8a0 CUDA: fix partial offloading for ne0 % 256 != 0 (#8572) 2024-07-18 23:48:47 +02:00
Johannes GäßlerandGitHub e02b597be3 lookup: fibonacci hashing, fix crashes (#8548) 2024-07-17 23:35:44 +02:00
Johannes GäßlerandGitHub 5e116e8dd5 make/cmake: add missing force MMQ/cuBLAS for HIP (#8515) 2024-07-16 21:20:59 +02:00
Johannes GäßlerandGitHub 808aba3916 CUDA: optimize and refactor MMQ (#8416) 
* CUDA: optimize and refactor MMQ

* explicit q8_1 memory layouts, add documentation
 2024-07-11 16:47:47 +02:00
Johannes GäßlerandGitHub a03e8dd99d make/cmake: LLAMA_NO_CCACHE -> GGML_NO_CCACHE (#8392) 2024-07-09 17:11:07 +02:00
Johannes GäßlerandGitHub 8e558309dc CUDA: MMQ support for iq4_nl, iq4_xs (#8278) 2024-07-05 09:06:31 +02:00
Johannes GäßlerandGitHub bcefa03bc0 CUDA: fix MMQ stream-k rounding if ne00 % 128 != 0 (#8311) 2024-07-05 09:05:34 +02:00
Johannes GäßlerandGitHub cb5fad4c6c CUDA: refactor and optimize IQ MMVQ (#8215) 
* CUDA: refactor and optimize IQ MMVQ

* uint -> uint32_t

* __dp4a -> ggml_cuda_dp4a

* remove MIN_CC_DP4A checks

* change default

* try CI fix
 2024-07-01 20:39:06 +02:00
Johannes GäßlerandGitHub 85a267daaa CUDA: fix MMQ stream-k for --split-mode row (#8167) 2024-06-27 16:26:05 +02:00
Johannes GäßlerandGitHub c8771ab5f8 CUDA: fix misaligned shared memory read (#8123) 2024-06-26 08:28:02 +02:00
Johannes GäßlerandGitHub 2df373ac40 CUDA: fix matrix multiplication algorithm choice (#8102) 2024-06-25 01:22:33 +02:00
Johannes GäßlerandGitHub 3b099bcd9c CUDA: fix MMQ writeback for int8 tensor cores (#8100) 2024-06-24 22:15:33 +02:00
Johannes GäßlerandGitHub a818f3028d CUDA: use MMQ instead of cuBLAS by default (#8075) 2024-06-24 17:43:42 +02:00
Johannes GäßlerandGitHub 9a590c8226 CUDA: optimize MMQ int8 tensor core performance (#8062) 
* CUDA: optimize MMQ int8 tensor core performance

* only a single get_mma_tile_x_k function

* simplify code, make functions constexpr
 2024-06-24 12:41:23 +02:00
Johannes GäßlerGitHubslaren
 		abd894ad96 common: fix warning (#8036) 
* common: fix warning

* Update common/common.cpp

Co-authored-by: slaren <slarengh@gmail.com>

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2024-06-20 16:40:13 +02:00
Johannes GäßlerandGitHub d50f8897a7 CUDA: stream-k decomposition for MMQ (#8018) 
* CUDA: stream-k decomposition for MMQ

* fix undefined memory reads for small matrices
 2024-06-20 14:39:21 +02:00
Johannes GäßlerandGitHub 76d66ee0be CUDA: faster q2_K, q3_K MMQ + int8 tensor cores (#7921) 
* CUDA: faster q2_K, q3_K MMQ + int8 tensor cores

* try CI fix

* try CI fix

* try CI fix

* fix data race

* rever q2_K precision related changes
 2024-06-14 18:41:49 +02:00
Johannes GäßlerandGitHub 963552903f CUDA: fix broken oob check for FA vec f32 kernel (#7904) 2024-06-12 17:41:51 +02:00
Johannes GäßlerandGitHub 148995e5e5 llama-bench: more compact markdown tables (#7879) 2024-06-11 14:45:40 +02:00
Johannes GäßlerandGitHub bdcb8f4222 CUDA: int8 tensor cores for MMQ (q4_K, q5_K, q6_K) (#7860) 2024-06-11 08:26:07 +02:00
Johannes GäßlerandGitHub 1f0dabda8d CUDA: use tensor cores for MMQ (#7676) 
* CUDA: int8 tensor cores for MMQ (legacy quants)

* fix out-of-bounds writes

* __builtin_assume -> GGML_CUDA_ASSUME

* fix writeback returning too early
 2024-06-10 11:45:13 +02:00
Johannes GäßlerandGitHub 42b53d192f CUDA: revise q8_1 data layout for mul_mat_q (#7824) 2024-06-09 09:42:25 +02:00
Johannes GäßlerandGitHub 7027b27d76 server: update cache_prompt documentation [no ci] (#7745) 2024-06-07 11:15:49 +02:00
Johannes GäßlerandGitHub 7d1a378b8f CUDA: refactor mmq, dmmv, mmvq (#7716) 
* CUDA: refactor mmq, dmmv, mmvq

* fix out-of-bounds write

* struct for qk, qr, qi

* fix cmake build

* mmq_type_traits
 2024-06-05 16:53:00 +02:00
Johannes GäßlerandGitHub 0b832d53ba make: fix debug options not being applied to NVCC (#7714) 2024-06-03 16:28:58 +02:00
Johannes GäßlerandGitHub e141ce624a Fix FlashAttention debug test, FP32 assert (#7684) 2024-06-01 23:26:10 +02:00
Johannes GäßlerandGitHub 750f60c03e CUDA: fix Pascal FA, deq. KV to FP16 for batch > 8 (#7681) 2024-06-01 15:47:04 +02:00
Johannes GäßlerandGitHub 9b596417af CUDA: quantized KV support for FA vec (#7527) 
* CUDA: quantized KV support for FA vec

* try CI fix

* fix commented-out kernel variants

* add q8_0 q4_0 tests

* fix nwarps > batch size

* split fattn compile via extern templates

* fix flake8

* fix metal tests

* fix cmake

* make generate_cu_files.py executable

* add autogenerated .cu files

* fix AMD

* error if type_v != FP16 and not flash_attn

* remove obsolete code
 2024-06-01 08:44:14 +02:00
Johannes GäßlerandGitHub c8047d538f scripts: update compare_llama_bench.py [no ci] (#7673) 2024-05-31 16:26:21 +02:00
Johannes GäßlerandGitHub 972b555ab9 README: explain parallel build [no ci] (#7618) 2024-05-30 09:52:39 +02:00
Johannes GäßlerandGitHub 10b1e45876 make: add --device-debug to NVCC debug flags (#7542) 2024-05-27 19:34:40 +02:00
Johannes GäßlerandGitHub cd93a28cb1 CUDA: fix FA out-of-bounds reads (#7479) 2024-05-23 00:31:20 +02:00
Johannes GäßlerandGitHub 38c03478a3 CUDA: fix FA out-of-bounds writes (#7465) 2024-05-22 17:58:25 +02:00
Johannes GäßlerandGitHub 95fb0aefab CUDA: remove incorrect precision check (#7454) 2024-05-22 10:24:29 +02:00
Johannes GäßlerandGitHub fcf6538ba6 CUDA: fix unused warning in mmq.cu (#7442) 2024-05-21 20:27:12 +03:00
Johannes GäßlerandGitHub d8ee902227 CUDA: deduplicate mmq code (#7397) 2024-05-21 16:02:12 +02:00
Johannes GäßlerandGitHub 20385cebcc perplexity: update README FP16 results [no ci] (#7413) 2024-05-20 18:15:38 +02:00
Johannes GäßlerandGitHub 5ca49cbecd ggml: implement quantized KV cache for FA (#7372) 2024-05-19 16:46:13 +02:00
Johannes GäßlerandGitHub 1b01f06db0 server: add test for token probs (#7347) 2024-05-19 16:26:02 +02:00
Johannes GäßlerandGitHub 41858392e1 server: fix seed being reported back (#7382) 2024-05-19 17:06:33 +03:00
Johannes GäßlerandGitHub 133d99c599 CUDA: deduplicate FlashAttention code (#7352) 2024-05-18 12:36:25 +02:00
Johannes GäßlerandGitHub cb42c29427 server: correct --threads documentation [no ci] (#7362) 2024-05-18 11:10:47 +02:00
Johannes GäßlerandGitHub 0fc1e820a9 CUDA: faster large batch FA without tensor cores (#7314) 2024-05-17 18:54:52 +02:00
Johannes GäßlerandGitHub 29c60d8cdd tokenization: add warning for double BOS (#7332) 2024-05-17 09:59:57 +02:00
Johannes GäßlerandGitHub 583fd6b000 server bench: fix bench not waiting for model load (#7284) 2024-05-15 08:44:16 +02:00
Johannes GäßlerandGitHub 1c570d8bee perplexity: add BF16 vs. FP16 results (#7150) 2024-05-13 13:03:27 +02:00
Johannes GäßlerandGitHub dc685be466 CUDA: add FP32 FlashAttention vector kernel (#7188) 
* CUDA: add FP32 FlashAttention vector kernel

* fixup! CUDA: add FP32 FlashAttention vector kernel

* fixup! fixup! CUDA: add FP32 FlashAttention vector kernel

* fixup! fixup! fixup! CUDA: add FP32 FlashAttention vector kernel
 2024-05-12 19:40:45 +02:00
Johannes GäßlerandGitHub 5ae3426b0b server: fix reported top tokens for temperature 0 (#7203) 2024-05-11 10:11:28 +02:00