nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity
2,930 Commits 1 Branch 0 Tags
854d365abab7194b5013a523f72da19860c3c550
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Johannes GäßlerandGitHub 133d99c599 CUDA: deduplicate FlashAttention code (#7352) 2024-05-18 12:36:25 +02:00
Johannes GäßlerandGitHub cb42c29427 server: correct --threads documentation [no ci] (#7362) 2024-05-18 11:10:47 +02:00
Johannes GäßlerandGitHub 0fc1e820a9 CUDA: faster large batch FA without tensor cores (#7314) 2024-05-17 18:54:52 +02:00
Johannes GäßlerandGitHub 29c60d8cdd tokenization: add warning for double BOS (#7332) 2024-05-17 09:59:57 +02:00
Johannes GäßlerandGitHub 583fd6b000 server bench: fix bench not waiting for model load (#7284) 2024-05-15 08:44:16 +02:00
Johannes GäßlerandGitHub 1c570d8bee perplexity: add BF16 vs. FP16 results (#7150) 2024-05-13 13:03:27 +02:00
Johannes GäßlerandGitHub dc685be466 CUDA: add FP32 FlashAttention vector kernel (#7188) 
* CUDA: add FP32 FlashAttention vector kernel

* fixup! CUDA: add FP32 FlashAttention vector kernel

* fixup! fixup! CUDA: add FP32 FlashAttention vector kernel

* fixup! fixup! fixup! CUDA: add FP32 FlashAttention vector kernel
 2024-05-12 19:40:45 +02:00
Johannes GäßlerandGitHub 5ae3426b0b server: fix reported top tokens for temperature 0 (#7203) 2024-05-11 10:11:28 +02:00
Johannes GäßlerandGitHub a743d76a01 CUDA: generalize FP16 fattn vec kernel (#7061) 
* CUDA: generalize FP16 fattn vec kernel

* disable unsupported head sizes for AMD in test

* try AMD fix

* fix batch size 2-8

* partially revert changes
 2024-05-09 14:32:02 +02:00
Johannes GäßlerandGitHub c12452c7ae JSON: [key] -> .at(key), assert() -> GGML_ASSERT (#7143) 2024-05-08 21:53:08 +02:00
Johannes GäßlerandGitHub af0a5b6163 server: fix incorrectly reported token probabilities (#7125) 
* server: normalize token probabilities

* fix temperature == 0.0f
 2024-05-07 23:07:58 +02:00
Johannes GäßlerandGitHub 3ea0d36000 Server: add tests for batch size, different seeds (#6950) 2024-05-01 17:52:55 +02:00
Johannes GäßlerandGitHub 1613ef8d8e CUDA: CUDART < 11.7 workaround for __hmax, __hmax2 (#7019) 2024-05-01 14:46:37 +02:00
Johannes GäßlerandGitHub a8f9b07631 perplexity: more statistics, added documentation (#6936) 
* perplexity: more statistics, added documentation

* add LLaMA 3 8b scoreboard
 2024-04-30 23:36:27 +02:00
Johannes GäßlerandGitHub c4f708a93f llama : fix typo LAMMAFILE -> LLAMAFILE (#6974) 2024-04-29 15:36:22 +03:00
Johannes GäßlerandGitHub 784e11dea1 README: add graphic for matrix multiplication (#6881) 2024-04-24 21:29:13 +02:00
Johannes GäßlerandGitHub 28103f4832 Server: fix seed for multiple slots (#6835) 
* Server: add tests for consistent results

* sampling: separate rng per sampling context
 2024-04-24 11:08:36 +02:00
Johannes GäßlerandGitHub b5e7285baf CUDA: fix matrix multiplication logic for tests (#6667) 2024-04-14 00:21:55 +02:00
Johannes GäßlerandGitHub 33a5244806 compare-llama-bench.py: fix long hexsha args (#6424) 2024-04-01 13:30:43 +02:00
Johannes GäßlerandGitHub 7aed0ffe68 Fixed lookup compilation issues on Windows (#6273) 2024-03-24 14:21:17 +01:00
Johannes GäßlerandGitHub 50ccaf5eac lookup: complement data from context with general text statistics (#5479) 
* lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens
 2024-03-23 01:24:36 +01:00
Johannes GäßlerandGitHub 47bb7b48c7 CUDA: fix DEBUG_CUDA_MALLOC (#5729) 2024-02-26 15:36:38 +01:00
Johannes GäßlerGitHubJared Van Bortel
 		ad014bba97 make: add error message for bad CUDA version (#5444) 
* make: add error message for bad CUDA version

* Update Makefile

Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com>

---------

Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com>
 2024-02-13 12:38:37 +01:00
Johannes GäßlerGitHubslaren
 		3bdc4cd0f5 CUDA: mul_mat_vec_q tiling, refactor mul mat logic (#5434) 
* CUDA: mul_mat_vec_q tiling, refactor mul mat logic

Co-authored-by: slaren <slarengh@gmail.com>

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2024-02-11 19:08:39 +01:00
Johannes GäßlerandGitHub e4640d8fdf lookup: add print for drafting performance (#5450) 2024-02-11 12:44:51 +01:00
Johannes GäßlerandGitHub 8e6a9d2de0 CUDA: more warps for mmvq on NVIDIA (#5394) 2024-02-08 21:56:40 +01:00
Johannes GäßlerandGitHub b7b74cef36 fix trailing whitespace (#5407) 2024-02-08 11:36:54 +01:00
Johannes GäßlerGitHubGeorgi Gerganov
 		26d4efd11e sampling: fix top_k <= 0 (#5388) 
* sampling: fix top_k <= 0

* Update llama.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-02-08 09:46:30 +01:00
Johannes GäßlerandGitHub aa7ab99be2 CUDA: fixed mmvq kernel for bs 2,3,4 and -sm row (#5386) 2024-02-07 12:40:26 +01:00
Johannes GäßlerandGitHub 17c97fb062 CUDA: mul_mat_vec_q max. batch size 8 -> 4 (#5370) 2024-02-06 19:43:06 +02:00
Johannes GäßlerandGitHub 2c516611f1 CUDA: mul_mat_vec_q for batch sizes > 1 (#5351) 2024-02-06 14:44:06 +01:00
Johannes GäßlerandGitHub 098f6d737b make: Use ccache for faster compilation (#5318) 
* make: Use ccache for faster compilation
 2024-02-05 19:33:00 +01:00
Johannes GäßlerGitHubGeorgi Gerganov
 		78b00dda6c README: updated introduction (#5343) 
* README: updated introduction

* readme : update

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-02-05 15:55:10 +01:00
Johannes GäßlerandGitHub 3c0d25c475 make: add nvcc info print (#5310) 2024-02-03 20:15:13 +01:00
Johannes GäßlerandGitHub 3cc5ed353c make: fix nvcc optimization flags for host code (#5309) 2024-02-03 20:14:59 +01:00
Johannes GäßlerandGitHub 9241c3a2ac Apply min_p to unsorted tokens (#5115) 2024-01-28 09:59:49 +01:00
Johannes GäßlerandGitHub b2b2bf988c Tests for min_p, sampling queue (#5147) 2024-01-28 09:35:14 +01:00
Johannes GäßlerandGitHub 9ecdd12e95 CUDA: more info when no device code (#5088) 2024-01-23 13:31:56 +01:00
Johannes GäßlerandGitHub c71d608ce7 ggml: cache sin/cos for RoPE (#4908) 2024-01-13 21:41:37 +01:00
Johannes GäßlerandGitHub 7dc78764e2 compare-llama-bench: tweak output format (#4910) 2024-01-13 15:52:53 +01:00
Johannes GäßlerandGitHub 3fe81781e3 CUDA: faster q8_0 -> f16 dequantization (#4895) 2024-01-12 20:38:54 +01:00
Johannes GäßlerandGitHub 1b280c9fff CUDA: fix softmax compile for old CUDA versions (#4862) 2024-01-12 12:30:41 +01:00
Johannes GäßlerandGitHub 4f56458d34 Python script to compare commits with llama-bench (#4844) 2024-01-10 01:04:33 +01:00
Johannes GäßlerandGitHub 8f900abfc0 CUDA: faster softmax via shared memory + fp16 math (#4742) 2024-01-09 08:58:55 +01:00
Johannes GäßlerandGitHub d5a410e855 CUDA: fixed redundant value dequantization (#4809) 2024-01-07 17:24:08 +01:00
Johannes GäßlerandGitHub a91928014f Print backend name on test-backend-ops failure (#4751) 2024-01-04 09:43:23 +01:00
Johannes GäßlerandGitHub 39d8bc71ed CUDA: fixed tensor cores not being used on RDNA3 (#4697) 2023-12-30 13:52:01 +01:00
Johannes GäßlerandGitHub a20f3c7465 CUDA: fix tensor core logic for Pascal and HIP (#4682) 2023-12-29 23:12:53 +01:00
Johannes GäßlerandGitHub e0a4002273 CUDA: fixed row rounding for 0 tensor splits (#4594) 2023-12-23 09:16:33 +01:00
Johannes GäßlerandGitHub 9154494808 CUDA: mul_mat_id always on GPU for batches >= 32 (#4553) 2023-12-21 18:42:59 +01:00
Johannes GäßlerandGitHub d3223afdad llama : disable per-tensor info prints on model load (#4562) 2023-12-21 18:34:17 +02:00
Johannes GäßlerGitHubslaren
 		799fc22689 CUDA: Faster Mixtral prompt processing (#4538) 
* CUDA: make MoE tensors contiguous for batch size>1

* Update ggml-cuda.cu

Co-authored-by: slaren <slarengh@gmail.com>

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-12-20 15:41:22 +01:00
Johannes GäßlerandGitHub 8185710a80 CUDA: use only 1 thread if fully offloaded (#2915) 2023-09-21 11:43:53 +03:00
Johannes GäßlerandGitHub ee66942d7e CUDA: fix peer access logic (#3231) 2023-09-17 23:35:20 +02:00
Johannes GäßlerandGitHub 111163e246 CUDA: enable peer access between devices (#2470) 2023-09-17 16:37:53 +02:00
Johannes GäßlerandGitHub 578d8c8f5c CUDA: fix scratch malloced on non-main device (#3220) 2023-09-17 14:16:22 +02:00
Johannes GäßlerGitHubHenri Vasserman
 		0a5eebb45d CUDA: mul_mat_q RDNA2 tunings (#2910) 
* CUDA: mul_mat_q RDNA2 tunings

* Update ggml-cuda.cu

Co-authored-by: Henri Vasserman <henv@hot.ee>

---------

Co-authored-by: Henri Vasserman <henv@hot.ee>
 2023-09-13 11:20:24 +02:00
Johannes GäßlerandGitHub 4f7cd6ba9c CUDA: fix LoRAs (#3130) 2023-09-13 00:15:33 +02:00
Johannes GäßlerandGitHub 89e89599fd CUDA: fix mul_mat_q not used for output tensor (#3127) 2023-09-11 22:58:41 +02:00
Johannes GäßlerandGitHub d54a4027a6 CUDA: lower GPU latency + fix Windows performance (#3110) 2023-09-11 19:55:51 +02:00
Johannes GäßlerandGitHub 8a4ca9af56 CUDA: add device number to error messages (#3112) 2023-09-11 13:00:24 +02:00
Johannes GäßlerandGitHub 8afe228000 CUDA: mul_mat_q=true llama_context_params default (#2912) 2023-08-30 21:46:19 +02:00
Johannes GäßlerandGitHub 6b73ef1201 YAML result logging + preset script (#2657) 2023-08-28 17:59:39 +02:00
Johannes GäßlerandGitHub 92b1bbd2ec CUDA: fix RoPE asserts, block sizes (#2833) 2023-08-28 14:23:55 +03:00
Johannes GäßlerandGitHub c63bb1d16a CUDA: use mul_mat_q kernels by default (#2683) 2023-08-22 22:47:05 +02:00
Johannes GäßlerandGitHub 0992a7b8b1 README: fix LLAMA_CUDA_MMV_Y documentation (#2647) 2023-08-17 23:57:59 +02:00
Johannes GäßlerandGitHub 1cd06fa25e CUDA: launch_bounds, small q4_K, q5_K mmq refactor (#2596) 2023-08-14 10:41:22 +02:00
Johannes GäßlerandGitHub f64d44a9b9 CUDA: Fixed OpenLLaMA 3b mmq, reduced compile time (#2590) 2023-08-13 00:24:45 +02:00
Johannes GäßlerandGitHub 25d43e0eb5 CUDA: tuned mul_mat_q kernels (#2546) 2023-08-09 09:42:34 +02:00
Johannes GäßlerandGitHub acfc5478ff CUDA: tighter VRAM scratch size for 65b/70b (#2551) 2023-08-08 14:38:16 +02:00
Johannes GäßlerandGitHub 3d9a551816 Fixed mmap prefetch for GPU offloading (#2529) 2023-08-07 10:09:40 +02:00
Johannes GäßlerandGitHub f514d1b306 CUDA: faster k-quant mul_mat_q kernels (#2525) 2023-08-05 18:20:44 +02:00
Johannes GäßlerandGitHub 468ea24fb4 CUDA: faster non k-quant mul_mat_q kernels (#2483) 2023-08-02 18:04:04 +02:00
Johannes GäßlerandGitHub 4f6b60c776 CUDA: Fix models with output size != 32000 (#2480) 2023-08-02 16:48:10 +02:00
Johannes GäßlerandGitHub 49e7cb5bb1 CUDA: fixed LLAMA_FAST compilation option (#2473) 2023-07-31 21:02:19 +02:00
Johannes GäßlerandGitHub b772bba42e CUDA: fixed cmake F16 option (#2471) 2023-07-31 19:52:22 +02:00
Johannes GäßlerandGitHub 0728c5a8b9 CUDA: mmq CLI option, fixed mmq build issues (#2453) 2023-07-31 15:44:35 +02:00
Johannes GäßlerandGitHub 1215ed7d5c CUDA: Implemented row flattening for non-glm RoPE (#2468) 2023-07-31 14:32:30 +02:00
Johannes GäßlerandGitHub 2dbf518911 CUDA: fewer memory bank conflicts for mul_mat_q (#2458) 2023-07-31 13:18:51 +02:00
Johannes GäßlerandGitHub 11f3ca06b8 CUDA: Quantized matrix matrix multiplication (#2160) 
* mmq implementation for non k-quants

* q6_K

* q2_K

* q3_k

* q4_K

* vdr

* q5_K

* faster q8_1 loading

* loop unrolling

* add __restrict__

* q2_K sc_high

* GGML_CUDA_MMQ_Y

* Updated Makefile

* Update Makefile

* DMMV_F16 -> F16

* Updated README, CMakeLists

* Fix CMakeLists.txt

* Fix CMakeLists.txt

* Fix multi GPU out-of-bounds
 2023-07-29 23:04:44 +02:00
Johannes GäßlerandGitHub 9baf9ef304 CUDA: faster multi GPU synchronization (#2448) 2023-07-29 23:04:10 +02:00
Johannes GäßlerandGitHub 70d26ac388 Fix __dp4a documentation (#2348) 2023-07-23 17:49:06 +02:00
Johannes GäßlerandGitHub b9b7d94fc1 CUDA: Fixed 7b q3_K_S with mul_mat_vec_q (#2313) 2023-07-22 21:27:34 +02:00
Johannes GäßlerandGitHub 4304bd3cde CUDA: mul_mat_vec_q kernels for k-quants (#2203) 2023-07-14 19:44:08 +02:00
Johannes GäßlerandGitHub 2b5eb72e10 Fixed __dp4a compute capability: 6.0 -> 6.1 (#2189) 2023-07-12 10:38:52 +02:00
Johannes GäßlerandGitHub 64639555ff Fixed OpenLLaMA 3b CUDA mul_mat_vec_q (#2144) 2023-07-08 20:01:44 +02:00
Johannes GäßlerandGitHub 061f5f8d21 CUDA: add __restrict__ to mul mat vec kernels (#2140) 2023-07-08 00:25:15 +02:00
Johannes GäßlerandGitHub 924dd22fd3 Quantized dot products for CUDA mul mat vec (#2067) 2023-07-05 14:19:42 +02:00
Johannes GäßlerandGitHub 9e4475f5cf Fixed OpenCL offloading prints (#2082) 2023-07-05 08:58:05 +02:00
Johannes GäßlerandGitHub 0bc2cdfc87 Better CUDA synchronization logic (#2057) 2023-07-01 21:49:44 +02:00
Johannes GäßlerandGitHub befb3a3562 Test-based VRAM scratch size + context adjustment (#2056) 2023-07-01 21:47:26 +02:00
Johannes GäßlerandGitHub 7f9753fa12 CUDA GPU acceleration for LoRAs + f16 models (#1970) 2023-06-28 18:35:54 +02:00
Johannes GäßlerandGitHub bbca06e269 cmake: revert CUDA arch default to 52, 61 if f16 (#1959) 2023-06-21 23:49:25 +02:00
Johannes GäßlerandGitHub 16b9cd1939 Convert vector to f16 for dequantize mul mat vec (#1913) 
* Convert vector to f16 for dmmv

* compile option

* Added compilation option description to README

* Changed cmake CUDA_ARCHITECTURES from "OFF" to "native"
 2023-06-19 10:23:56 +02:00
Johannes GäßlerandGitHub b24c3049d9 Added tokens per second to info prints (#1928) 2023-06-18 17:41:26 +02:00
Johannes GäßlerandGitHub 0ede372a51 Fixed incorrectly applying RMS norm twice (#1925) 2023-06-18 16:07:09 +02:00
Johannes GäßlerandGitHub 2c9380dd2f Only one CUDA stream per device for async compute (#1898) 2023-06-17 19:15:02 +02:00
Johannes GäßlerandGitHub ac3b886953 llama : fix embd when offloading non-repeating layers (#1891) 2023-06-16 21:25:51 +03:00
Johannes GäßlerandGitHub a09f9195be Fixed CUDA runtime version check (#1879) 2023-06-15 21:49:08 +02:00
Johannes GäßlerandGitHub 6b8312e797 Better error when using both LoRA + GPU layers (#1861) 2023-06-15 19:06:46 +02:00