nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 29 Packages Projects Releases Wiki Activity
4,222 Commits 1 Branch 0 Tags
f0678c5ff4cb8873d6ff48801475ff270db656fa
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Georgi GerganovandGitHub f0678c5ff4 ggml : fix I8MM Q4_1 scaling factor conversion (#10562) 
ggml-ci
 2024-11-29 16:25:39 +02:00
Georgi Gerganov dc22344088 ggml : remove redundant copyright notice + update authors 2024-11-28 20:46:40 +02:00
Georgi Gerganov 4c0a95b107 llama : add missing model types 2024-11-28 20:45:07 +02:00
Georgi GerganovandGitHub 76b27d29c2 ggml : fix row condition for i8mm kernels (#10561) 
ggml-ci
 2024-11-28 14:56:37 +02:00
Georgi GerganovandGitHub eea986f215 cmake : fix ARM feature detection (#10543) 
ggml-ci
 2024-11-28 14:56:23 +02:00
Georgi Gerganov 9e2301f4a4 metal : fix group_norm support condition (#0) 2024-11-27 11:22:14 +02:00
Georgi Gerganov fee824a1a1 sync : ggml 2024-11-27 11:10:42 +02:00
Georgi GerganovandGitHub ab96610b1e cmake : enable warnings in llama (#10474) 
* cmake : enable warnings in llama

ggml-ci

* cmake : add llama_get_flags and respect LLAMA_FATAL_WARNINGS

* cmake : get_flags -> ggml_get_flags

* speculative-simple : fix warnings

* cmake : reuse ggml_get_flags

ggml-ci

* speculative-simple : fix compile warning

ggml-ci
 2024-11-26 14:18:08 +02:00
Georgi GerganovandGitHub 84e1c33cde server : fix parallel speculative decoding (#10513) 
ggml-ci
 2024-11-26 13:36:40 +02:00
Georgi GerganovandGitHub 811872a59d speculative : simplify the implementation (#10504) 
ggml-ci
 2024-11-26 12:29:38 +02:00
Georgi Gerganov 9fd8c2687f server : add more information about error (#10455) 2024-11-25 22:28:59 +02:00
Georgi GerganovandGitHub 47f931c8f9 server : enable cache_prompt by default (#10501) 
ggml-ci
 2024-11-25 21:50:07 +02:00
Georgi GerganovandGitHub 106964e3d2 metal : enable mat-vec kernels for bs <= 4 (#10491) 2024-11-25 21:49:31 +02:00
Georgi GerganovandGitHub 9ca2e67762 server : add speculative decoding support (#10455) 
* server : add speculative decoding support

ggml-ci

* server : add helper function slot.can_speculate()

ggml-ci
 2024-11-25 16:31:38 +02:00
Georgi Gerganov f6d12e7df8 tests : fix compile warning 2024-11-25 15:17:32 +02:00
Georgi Gerganov b756441104 metal : minor code formatting 2024-11-25 15:08:04 +02:00
Georgi GerganovandGitHub d9d54e498d speculative : refactor and add a simpler example (#10362) 
* speculative : refactor and add a simpler example

ggml-ci

* speculative : clean-up and add comments and TODOs [no ci]

* speculative : manage context in common_speculative

ggml-ci

* speculative : simplify

ggml-ci

* speculative : simplify (cont)

ggml-ci

* speculative : add --draft-min CLI arg

* speculative : minor fixup

* make : build fixes

* speculative : do not redraft previous drafts

ggml-ci

* speculative : fix the draft sampling

ggml-ci

* speculative : fix compile warning

* common : refactor args

ggml-ci

* common : change defaults [no ci]

* common : final touches

ggml-ci
 2024-11-25 09:58:41 +02:00
Georgi GerganovGitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
 		cce5a90075 flake.lock: Update (#10470) 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/5e4fbfb6b3de1aa2872b76d49fafc942626e2add?narHash=sha256-OZiZ3m8SCMfh3B6bfGC/Bm4x3qc1m2SVEAlkV6iY7Yg%3D' (2024-11-15)
  → 'github:NixOS/nixpkgs/23e89b7da85c3640bbc2173fe04f4bd114342367?narHash=sha256-y/MEyuJ5oBWrWAic/14LaIr/u5E0wRVzyYsouYY3W6w%3D' (2024-11-19)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-11-24 08:03:25 -08:00
Georgi GerganovandGitHub 1bb30bf28c llama : handle KV shift for recurrent models (#10402) 
ggml-ci
 2024-11-21 10:22:47 +02:00
Georgi Gerganov 87a533be57 sync : ggml 2024-11-21 09:22:11 +02:00
Georgi Gerganov 9fe0fb0626 sync : ggml 2024-11-19 20:03:21 +02:00
Georgi GerganovandGitHub 8e752a777b llama : add check for KV cache shifts (#10401) 
ggml-ci
 2024-11-19 13:29:26 +02:00
Georgi GerganovGitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
 		2eb76b2a5e flake.lock: Update (#10346) 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/4aa36568d413aca0ea84a1684d2d46f55dbabad7?narHash=sha256-Zwl8YgTVJTEum%2BL%2B0zVAWvXAGbWAuXHax3KzuejaDyo%3D' (2024-11-05)
  → 'github:NixOS/nixpkgs/5e4fbfb6b3de1aa2872b76d49fafc942626e2add?narHash=sha256-OZiZ3m8SCMfh3B6bfGC/Bm4x3qc1m2SVEAlkV6iY7Yg%3D' (2024-11-15)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-11-18 06:08:20 -08:00
Georgi Gerganov 20a780c7b6 gitignore : ignore local run scripts [no ci] 2024-11-17 13:12:22 +02:00
Georgi GerganovandGitHub cf32a9b93a metal : refactor kernel args into structs (#10238) 
* metal : add kernel arg structs (wip)

* metal : fattn args

ggml-ci

* metal : cont + avoid potential int overflow [no ci]

* metal : mul mat struct (wip)

* cont : mul mat vec

* cont : pass by reference

* cont : args is first argument

* cont : use char ptr

* cont : shmem style

* cont : thread counters style

* cont : mul mm id

ggml-ci

* cont : int safety + register optimizations

ggml-ci

* metal : GGML_OP_CONCAT

ggml-ci

* metal : GGML_OP_ADD, GGML_OP_SUB, GGML_OP_MUL, GGML_OP_DIV

* metal : GGML_OP_REPEAT

* metal : GGML_OP_CPY

* metal : GGML_OP_RMS_NORM

* metal : GGML_OP_NORM

* metal : add TODOs for rest of ops

* ggml : add ggml-metal-impl.h

ggml-ci
 2024-11-17 11:23:01 +02:00
Georgi Gerganov 24203e9dd7 ggml : inttypes.h -> cinttypes (#0) 
ggml-ci
 2024-11-17 08:30:29 +02:00
Georgi Gerganov 5d9e59979c ggml : adapt AMX to tensor->grad removal (#0) 
ggml-ci
 2024-11-17 08:30:29 +02:00
Georgi Gerganov a4200cafad make : add ggml-opt (#0) 
ggml-ci
 2024-11-17 08:30:29 +02:00
Georgi Gerganov 84274a10c3 tests : remove test-grad0 2024-11-17 08:30:29 +02:00
Georgi Gerganov 68fcb4759c ggml : fix compile warnings (#0) 
ggml-ci
 2024-11-17 08:30:29 +02:00
Georgi Gerganov 5c9a8b22b1 scripts : update sync 2024-11-17 08:30:29 +02:00
Georgi Gerganov db4cfd5dbc llamafile : fix include path (#0) 
ggml-ci
 2024-11-16 20:36:26 +02:00
Georgi Gerganov 8ee0d09ae6 make : auto-determine dependencies (#0) 2024-11-16 20:36:26 +02:00
Georgi GerganovandGitHub f245cc28d4 scripts : fix missing key in compare-llama-bench.py (#10332) 2024-11-16 10:32:50 +02:00
Georgi Gerganov 09ecbcb596 cmake : fix ppc64 check (whisper/0) 
ggml-ci
 2024-11-15 15:44:06 +02:00
Georgi Gerganov cbf5541a82 sync : ggml 2024-11-15 15:44:06 +02:00
Georgi Gerganov 4802ad350b scripts : fix regex in sync [no ci] 2024-11-15 08:38:43 +02:00
Georgi GerganovandGitHub 2a82891a85 speculative : fix out-of-bounds access (#10289) 2024-11-14 11:44:15 +02:00
Georgi Gerganov 5ea926dad7 sync : ggml 2024-11-13 18:11:54 +02:00
Georgi GerganovandGitHub b0cefea58a metal : more precise Q*K in FA vec kernel (#10247) 2024-11-11 08:39:13 +02:00
Georgi GerganovandGitHub b141e5f6ef server : enable KV cache defrag by default (#10233) 
ggml-ci
 2024-11-11 08:38:43 +02:00
Georgi GerganovGitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
 		4b3a9212b6 flake.lock: Update (#10243) 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/807e9154dcb16384b1b765ebe9cd2bba2ac287fd?narHash=sha256-l253w0XMT8nWHGXuXqyiIC/bMvh1VRszGXgdpQlfhvU%3D' (2024-10-29)
  → 'github:NixOS/nixpkgs/4aa36568d413aca0ea84a1684d2d46f55dbabad7?narHash=sha256-Zwl8YgTVJTEum%2BL%2B0zVAWvXAGbWAuXHax3KzuejaDyo%3D' (2024-11-05)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-11-10 11:45:25 -08:00
Georgi GerganovandGitHub 6423c65aa8 metal : reorder write loop in mul mat kernel + style (#10231) 
* metal : reorder write loop

* metal : int -> short, style

ggml-ci
 2024-11-09 11:53:13 +02:00
Georgi GerganovandGitHub 39a334a9aa metal : fix build and some more comments (#10229) 2024-11-09 11:53:02 +02:00
Georgi GerganovandGitHub bb38cdd8ba metal : fix F32 accumulation in FA vec kernel (#10232) 2024-11-09 11:52:45 +02:00
Georgi Gerganov f018acba22 llama : fix Qwen model type strings 2024-11-09 11:26:34 +02:00
Georgi Gerganov 46323fa9ef metal : hide debug messages from normal log 2024-11-09 11:21:49 +02:00
Georgi GerganovandGitHub ec450d3bbf metal : opt-in compile flag for BF16 (#10218) 
* metal : opt-in compile flag for BF16

ggml-ci

* ci : use BF16

ggml-ci

* swift : switch back to v12

* metal : has_float -> use_float

ggml-ci

* metal : fix BF16 check in MSL

ggml-ci
 2024-11-08 21:59:46 +02:00
Georgi Gerganov 695ad752b2 metal : improve clarity (minor) (#10171) 2024-11-08 18:37:41 +02:00
Georgi GerganovandGitHub 841f27abdb metal : optimize FA kernels (#10171) 
* ggml : add ggml_flash_attn_ext_get_prec

* metal : use F16 precision in FA kernels

ggml-ci

* metal : minor clean-up

* metal : compile-guard bf16 FA kernels

ggml-ci

* build : remove obsolete compile flag [no ci]

* metal : prevent int overflows [no ci]

* cuda : disable BF16 FA

ggml-ci

* metal : fix BF16 requirement for FA kernels

ggml-ci

* make : clean-up [no ci]
 2024-11-08 13:47:22 +02:00
Georgi Gerganov eec4d71737 scripts : add amx to sync-ggml.sh [no ci] 2024-11-07 23:11:36 +02:00
Georgi Gerganov 3b08828674 sync : ggml 2024-11-07 23:08:24 +02:00
Georgi Gerganov a2c6fd747c scripts : sync update 2024-11-07 23:07:55 +02:00
Georgi GerganovandGitHub 5c333e0140 metal : add BF16 support (#8439) 
* ggml : add initial BF16 support

ggml-ci

* metal : add mul_mat_id BF16 support

ggml-ci

* metal : check for bfloat support on the Metal device

ggml-ci

* metal : better var names [no ci]

* metal : do not build bfloat kernels when not supported

ggml-ci

* metal : try to fix BF16 support check

ggml-ci

* metal : this should correctly check bfloat support
 2024-11-06 19:53:51 +02:00
Georgi GerganovandGitHub b11f9ba9b8 server : remove hack for extra parallel slot (#10187) 
ggml-ci
 2024-11-06 13:29:01 +02:00
Georgi GerganovandGitHub 1dc04b2dee ggml : adjust is_first_call init value (#10193) 
ggml-ci
 2024-11-06 11:20:10 +02:00
Georgi GerganovandGitHub a1eaf6a960 metal : add quantized FA support (#10149) 
* metal : add quantized FA (vec) support

ggml-ci

* metal : add quantized FA (non-vec) support

* metal : fix support check

ggml-ci

* metal : clean-up

* metal : clean-up (cont)

* metal : fix shared memory calc + reduce smem + comments

* metal : float-correctness

* metal : minor [no ci]
 2024-11-06 10:24:23 +02:00
Georgi Gerganov 05697f670b metal : simplify f16 and f32 dequant kernels (#0) 2024-11-04 13:49:34 +02:00
Georgi Gerganov f8e58135cf metal : move dequantize templates to beginning of MSL source (#0) 2024-11-04 13:44:06 +02:00
Georgi Gerganov ce027adfb3 sync : ggml 2024-11-04 10:33:37 +02:00
Georgi GerganovandGitHub 08828a6d7d metal : minor fixup in FA kernel (#10143) 
* metal : minor fixup in FA kernel

ggml-ci

* metal : use the unrolled loop variable

* metal : remove unused var
 2024-11-03 15:18:40 +02:00
Georgi GerganovandGitHub 1839f69130 flake.lock: Update (#10146) 2024-11-03 05:14:15 -08:00
Georgi GerganovandGitHub 45950415ed server : fix endpoint checks (#10135) 
ggml-ci
 2024-11-02 18:34:00 +02:00
Georgi GerganovandGitHub 1926d6e39d llama : adjust default context size + print warnings (#10136) 
* llama : adjust default context size + print warnings

ggml-ci

* ggml-ci : add missing gpu-layers + adjust context sizes
 2024-11-02 15:18:56 +02:00
Georgi GerganovandGitHub ba6f62eb79 readme : update hot topics 2024-11-01 17:31:51 +02:00
Georgi GerganovandGitHub 1804adb0cf ggml : remove ggml_scratch (#10121) 
ggml-ci
 2024-11-01 12:58:45 +02:00
Georgi Gerganov 815fe72adc sync : ggml 2024-11-01 10:28:24 +02:00
Georgi Gerganov f221d56220 ggml : alloc ggml_contexts on the heap (whisper/2525) 2024-11-01 10:24:50 +02:00
Georgi GerganovandGitHub 8d8ff71536 llama : remove Tail-Free sampling (#10071) 
ggml-ci
 2024-10-29 10:42:05 +02:00
Georgi GerganovGitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
 		07028f9d74 flake.lock: Update (#10063) 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/4c2fcb090b1f3e5b47eaa7bd33913b574a11e0a0?narHash=sha256-/uilDXvCIEs3C9l73JTACm4quuHUsIHcns1c%2BcHUJwA%3D' (2024-10-18)
  → 'github:NixOS/nixpkgs/2768c7d042a37de65bb1b5b3268fc987e534c49d?narHash=sha256-AlcmCXJZPIlO5dmFzV3V2XF6x/OpNWUV8Y/FMPGd8Z4%3D' (2024-10-23)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-10-28 08:41:24 -07:00
Georgi GerganovandGitHub 8125e6cbfc server : don't overfill the batch during infill (#10018) 
ggml-ci
 2024-10-28 08:49:32 +02:00
Georgi GerganovandGitHub 8841ce3f43 llama : switch KQ multiplication to F32 precision by default (#10015) 
ggml-ci
 2024-10-27 20:59:58 +02:00
Georgi Gerganov cc2983d375 sync : ggml 2024-10-26 10:34:08 +03:00
Georgi Gerganov 9e4a2563ea scripts : fix amx sync [no ci] 2024-10-26 10:33:31 +03:00
Georgi GerganovandGitHub 668750357e metal : support permuted matrix multiplicaions (#10033) 
* metal : support permuted matrix multiplicaions

ggml-ci

* cont : use nb01 directly for row steps

ggml-ci

* cont : add comments [no ci]

* metal : minor refactor

* metal : minor
 2024-10-25 22:26:15 +03:00
Georgi GerganovandGitHub bc5ba007b2 server : check that the prompt fits in the slot's context (#10030) 
ggml-ci
 2024-10-25 10:13:46 +03:00
Georgi Gerganov 40f2555797 ci : fix cmake flags for SYCL 2024-10-24 21:23:33 +03:00
Georgi Gerganov 190a37d797 sync : ggml 2024-10-23 17:23:55 +03:00
Georgi Gerganov 2d3aba9ee8 llama.vim : bump generation time limit to 3s [no ci] 2024-10-23 17:16:56 +03:00
Georgi Gerganov e94a138d64 llama.vim : fix info text display [no ci] (#9787) 2024-10-22 00:37:55 +03:00
Georgi Gerganov e01c67affe llama.vim : move info to the right of screen [no ci] (#9787) 
'eol' messes up the rendering with nvim v0.10.2 for some reason
 2024-10-21 22:53:18 +03:00
Georgi GerganovandGitHub dbd5f2f573 llama.vim : plugin for Neovim (#9787) 2024-10-21 20:25:02 +03:00
Georgi GerganovandGitHub f594bc80ba ggml : add asserts for type conversion in fattn kernels (#9971) 
ggml-ci
 2024-10-21 16:20:46 +03:00
Georgi GerganovandGitHub 55e47786e3 llama : default sampling changes + greedy update (#9897) 
* llama : deprecate softmax sampler + fix dist sampler

ggml-ci

* tests : replace macros with functions

ggml-ci

* sampling : change temperature sampler logic

For t <= 0.0f, keep the max logit intact and set the rest to -inf

* cont : no need for special "greedy" logic

top-k == 1 is the same

* tests : init prob correctly

* llama : handle temp <= 0.0 in the temp_ext sampler too

ggml-ci

* cont : avoid extra loop in temperature sampler for sub-zero temp

ggml-ci
 2024-10-21 09:46:40 +03:00
Georgi GerganovandGitHub bc21975084 speculative : fix handling of some input params (#9963) 
* speculative : fix batch sizes at initialization

ggml-ci

* speculative : handle params.n_predict == -1

* speculative : limit batch size to llama_n_batch
 2024-10-21 09:37:12 +03:00
Georgi GerganovandGitHub 8901755ba3 server : add n_indent parameter for line indentation requirement (#9929) 
ggml-ci
 2024-10-18 07:32:19 +03:00
Georgi Gerganov 17bb928080 readme : remove --memory-f32 references (#9925) 2024-10-17 23:43:05 +03:00
Georgi Gerganov 9f45fc1e99 llama : change warning to debug log 2024-10-17 23:27:42 +03:00
Georgi GerganovandGitHub 99bd4ac28c llama : infill sampling handle very long tokens (#9924) 
* llama : infill sampling handle very long tokens

ggml-ci

* cont : better indices

ggml-ci
 2024-10-17 22:32:47 +03:00
Georgi Gerganov 0e41b300ed sync : ggml 2024-10-16 11:28:14 +03:00
Georgi GerganovandGitHub 755a9b2bf0 llama : add infill sampler (#9896) 
ggml-ci
 2024-10-15 16:35:33 +03:00
Georgi GerganovandGitHub 223c25a72f server : improve infill context reuse (#9894) 
ggml-ci
 2024-10-15 16:28:55 +03:00
Georgi GerganovandGitHub dcdd535302 server : update preact (#9895) 2024-10-15 12:48:44 +03:00
Georgi GerganovandGitHub d4c19c0f5c server : accept extra_context for the infill endpoint (#9874) 
* server : accept extra_context for the infill endpoint

ggml-ci

* server : update readme [no ci]

* server : use repo-level FIM pattern if possible

ggml-ci
 2024-10-13 21:31:35 +03:00
Georgi GerganovandGitHub c7181bd294 server : reuse cached context chunks (#9866) 
ggml-ci
 2024-10-13 18:52:48 +03:00
Georgi GerganovGitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
 		92be9f1216 flake.lock: Update (#9870) 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/bc947f541ae55e999ffdb4013441347d83b00feb?narHash=sha256-NOiTvBbRLIOe5F6RbHaAh6%2B%2BBNjsb149fGZd1T4%2BKBg%3D' (2024-10-04)
  → 'github:NixOS/nixpkgs/5633bcff0c6162b9e4b5f1264264611e950c8ec7?narHash=sha256-9UTxR8eukdg%2BXZeHgxW5hQA9fIKHsKCdOIUycTryeVw%3D' (2024-10-09)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-10-12 20:11:26 -07:00
Georgi GerganovandGitHub edc265661c server : add option to time limit the generation phase (#9865) 
ggml-ci
 2024-10-12 16:14:27 +03:00
Georgi GerganovandGitHub 1bde94dd02 server : remove self-extend features (#9860) 
* server : remove self-extend

ggml-ci

* server : fix context limit check to use slot.n_past

ggml-ci
 2024-10-12 16:06:31 +03:00
Georgi GerganovandGitHub 95c76e8e92 server : remove legacy system_prompt feature (#9857) 
* server : remove legacy system_prompt feature

ggml-ci

* readme : update [no ci]

* server : fix non-transformer logic + remove response from /props
 2024-10-12 14:51:54 +03:00
Georgi GerganovandGitHub 11ac9800af llama : improve infill support and special token detection (#9798) 
* llama : improve infill support

ggml-ci

* llama : add more FIM token strings

ggml-ci

* server : update prompt on slot restore (#9800)

* gguf : deprecate old FIM token KVs
 2024-10-12 08:21:51 +03:00