nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 29 Packages Projects Releases Wiki Activity
4,308 Commits 1 Branch 0 Tags
fb18934a97425c42c8b32a1baaa94f0080eb051d
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Georgi Gerganov fb18934a97 gguf-py : bump version to 0.11.0 2024-12-11 23:13:31 +02:00
Georgi GerganovandGitHub d9c3ba2b77 ggml : disable iq4_nl interleave size 8 (#10709) 
ggml-ci
 2024-12-07 18:38:15 +02:00
Georgi GerganovGitHubXuan Son NguyenXuan Son Nguyen
 		ce4a7b8493 server : various fixes (#10704) 
* server : various fixes

ggml-ci

* server : show curent seed in slot_params

ggml-ci

* fix /slots endpoint

* Update examples/server/server.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* server : reflect endpoint response changes in the readme

ggml-ci

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
 2024-12-07 18:02:05 +02:00
Georgi GerganovandGitHub c2a16c0bdb server : fix free of spec context and batch (#10651) 
ggml-ci
 2024-12-07 11:52:44 +02:00
Georgi Gerganov c5ede3849f convert : add custom attention mapping 2024-12-06 21:33:49 +02:00
Georgi Gerganov 0cd182ebcc sync : ggml 2024-12-05 13:27:42 +02:00
Georgi GerganovandGitHub 1da7b76569 server : fix speculative decoding with context shift (#10641) 
* server : fix speculative decoding with context shift

ggml-ci

* server : take into account speculative limits

ggml-ci

* server : add tests
 2024-12-04 22:38:20 +02:00
Georgi Gerganov 1cd3df46bd scripts : remove amx sync 
ggml-ci
 2024-12-03 20:04:49 +02:00
Georgi Gerganov c505471857 sync : ggml 2024-12-03 20:04:49 +02:00
Georgi GerganovandGitHub 0115df2f65 metal : small-batch mat-mul kernels (#10581) 
* metal : small-batch mat-mul kernels

ggml-ci

* metal : add rest of types

ggml-ci

* metal : final adjustments

ggml-ci

* metal : add comments

ggml-ci
 2024-12-03 11:52:33 +02:00
Georgi Gerganov 515d4e5372 github : minify link [no ci] (revert) 
this doesn't work as expected
 2024-12-03 11:21:43 +02:00
Georgi Gerganov 844e2e1fee github : minify link [no ci] 2024-12-03 11:20:35 +02:00
Georgi GerganovandGitHub 70b98fadbc server : fix default draft model parameters (#10586) 
* server : force F16 KV cache for the draft model

ggml-ci

* server : fix draft params

ggml-ci

* server : various params fixes

ggml-ci
 2024-12-03 11:20:00 +02:00
Georgi GerganovGitHubDiego Devesa
 		8648c52101 make : deprecate (#10514) 
* make : deprecate

ggml-ci

* ci : disable Makefile builds

ggml-ci

* docs : remove make references [no ci]

* ci : disable swift build

ggml-ci

* docs : remove obsolete make references, scripts, examples

ggml-ci

* basic fix for compare-commits.sh

* update build.md

* more build.md updates

* more build.md updates

* more build.md updates

* Update Makefile

Co-authored-by: Diego Devesa <slarengh@gmail.com>

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2024-12-02 21:22:53 +02:00
Georgi GerganovandGitHub 4cb003dd8d contrib : refresh (#10593) 
* contrib : refresh

* contrib : expand [no ci]

* contrib : expand test-backend-ops instructions

* contrib : add CODEOWNERS

* prs : update template to not have checkbox [no ci]
 2024-12-02 08:53:27 +02:00
Georgi GerganovandGitHub 5e1ed95583 grammars : add English-only grammar (#10612) 2024-12-01 21:37:54 +02:00
Georgi GerganovandGitHub 6acce39710 readme : update the usage section with examples (#10596) 
* readme : update the usage section with examples

* readme : more examples
 2024-12-01 11:25:17 +02:00
Georgi GerganovandGitHub 3e0ba0e604 readme : remove old badge 2024-11-30 10:09:21 +02:00
Georgi GerganovandGitHub abadba05be readme : refresh (#10587) 
* readme : refresh

* readme : move section [no ci]

* readme : clarify [no ci]

* readme : fixes [no ci]

* readme : more fixes [no ci]

* readme : simplify [no ci]

* readme : clarify GGUF
 2024-11-30 09:47:07 +02:00
Georgi GerganovandGitHub f0678c5ff4 ggml : fix I8MM Q4_1 scaling factor conversion (#10562) 
ggml-ci
 2024-11-29 16:25:39 +02:00
Georgi Gerganov dc22344088 ggml : remove redundant copyright notice + update authors 2024-11-28 20:46:40 +02:00
Georgi Gerganov 4c0a95b107 llama : add missing model types 2024-11-28 20:45:07 +02:00
Georgi GerganovandGitHub 76b27d29c2 ggml : fix row condition for i8mm kernels (#10561) 
ggml-ci
 2024-11-28 14:56:37 +02:00
Georgi GerganovandGitHub eea986f215 cmake : fix ARM feature detection (#10543) 
ggml-ci
 2024-11-28 14:56:23 +02:00
Georgi Gerganov 9e2301f4a4 metal : fix group_norm support condition (#0) 2024-11-27 11:22:14 +02:00
Georgi Gerganov fee824a1a1 sync : ggml 2024-11-27 11:10:42 +02:00
Georgi GerganovandGitHub ab96610b1e cmake : enable warnings in llama (#10474) 
* cmake : enable warnings in llama

ggml-ci

* cmake : add llama_get_flags and respect LLAMA_FATAL_WARNINGS

* cmake : get_flags -> ggml_get_flags

* speculative-simple : fix warnings

* cmake : reuse ggml_get_flags

ggml-ci

* speculative-simple : fix compile warning

ggml-ci
 2024-11-26 14:18:08 +02:00
Georgi GerganovandGitHub 84e1c33cde server : fix parallel speculative decoding (#10513) 
ggml-ci
 2024-11-26 13:36:40 +02:00
Georgi GerganovandGitHub 811872a59d speculative : simplify the implementation (#10504) 
ggml-ci
 2024-11-26 12:29:38 +02:00
Georgi Gerganov 9fd8c2687f server : add more information about error (#10455) 2024-11-25 22:28:59 +02:00
Georgi GerganovandGitHub 47f931c8f9 server : enable cache_prompt by default (#10501) 
ggml-ci
 2024-11-25 21:50:07 +02:00
Georgi GerganovandGitHub 106964e3d2 metal : enable mat-vec kernels for bs <= 4 (#10491) 2024-11-25 21:49:31 +02:00
Georgi GerganovandGitHub 9ca2e67762 server : add speculative decoding support (#10455) 
* server : add speculative decoding support

ggml-ci

* server : add helper function slot.can_speculate()

ggml-ci
 2024-11-25 16:31:38 +02:00
Georgi Gerganov f6d12e7df8 tests : fix compile warning 2024-11-25 15:17:32 +02:00
Georgi Gerganov b756441104 metal : minor code formatting 2024-11-25 15:08:04 +02:00
Georgi GerganovandGitHub d9d54e498d speculative : refactor and add a simpler example (#10362) 
* speculative : refactor and add a simpler example

ggml-ci

* speculative : clean-up and add comments and TODOs [no ci]

* speculative : manage context in common_speculative

ggml-ci

* speculative : simplify

ggml-ci

* speculative : simplify (cont)

ggml-ci

* speculative : add --draft-min CLI arg

* speculative : minor fixup

* make : build fixes

* speculative : do not redraft previous drafts

ggml-ci

* speculative : fix the draft sampling

ggml-ci

* speculative : fix compile warning

* common : refactor args

ggml-ci

* common : change defaults [no ci]

* common : final touches

ggml-ci
 2024-11-25 09:58:41 +02:00
Georgi GerganovGitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
 		cce5a90075 flake.lock: Update (#10470) 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/5e4fbfb6b3de1aa2872b76d49fafc942626e2add?narHash=sha256-OZiZ3m8SCMfh3B6bfGC/Bm4x3qc1m2SVEAlkV6iY7Yg%3D' (2024-11-15)
  → 'github:NixOS/nixpkgs/23e89b7da85c3640bbc2173fe04f4bd114342367?narHash=sha256-y/MEyuJ5oBWrWAic/14LaIr/u5E0wRVzyYsouYY3W6w%3D' (2024-11-19)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-11-24 08:03:25 -08:00
Georgi GerganovandGitHub 1bb30bf28c llama : handle KV shift for recurrent models (#10402) 
ggml-ci
 2024-11-21 10:22:47 +02:00
Georgi Gerganov 87a533be57 sync : ggml 2024-11-21 09:22:11 +02:00
Georgi Gerganov 9fe0fb0626 sync : ggml 2024-11-19 20:03:21 +02:00
Georgi GerganovandGitHub 8e752a777b llama : add check for KV cache shifts (#10401) 
ggml-ci
 2024-11-19 13:29:26 +02:00
Georgi GerganovGitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
 		2eb76b2a5e flake.lock: Update (#10346) 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/4aa36568d413aca0ea84a1684d2d46f55dbabad7?narHash=sha256-Zwl8YgTVJTEum%2BL%2B0zVAWvXAGbWAuXHax3KzuejaDyo%3D' (2024-11-05)
  → 'github:NixOS/nixpkgs/5e4fbfb6b3de1aa2872b76d49fafc942626e2add?narHash=sha256-OZiZ3m8SCMfh3B6bfGC/Bm4x3qc1m2SVEAlkV6iY7Yg%3D' (2024-11-15)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-11-18 06:08:20 -08:00
Georgi Gerganov 20a780c7b6 gitignore : ignore local run scripts [no ci] 2024-11-17 13:12:22 +02:00
Georgi GerganovandGitHub cf32a9b93a metal : refactor kernel args into structs (#10238) 
* metal : add kernel arg structs (wip)

* metal : fattn args

ggml-ci

* metal : cont + avoid potential int overflow [no ci]

* metal : mul mat struct (wip)

* cont : mul mat vec

* cont : pass by reference

* cont : args is first argument

* cont : use char ptr

* cont : shmem style

* cont : thread counters style

* cont : mul mm id

ggml-ci

* cont : int safety + register optimizations

ggml-ci

* metal : GGML_OP_CONCAT

ggml-ci

* metal : GGML_OP_ADD, GGML_OP_SUB, GGML_OP_MUL, GGML_OP_DIV

* metal : GGML_OP_REPEAT

* metal : GGML_OP_CPY

* metal : GGML_OP_RMS_NORM

* metal : GGML_OP_NORM

* metal : add TODOs for rest of ops

* ggml : add ggml-metal-impl.h

ggml-ci
 2024-11-17 11:23:01 +02:00
Georgi Gerganov 24203e9dd7 ggml : inttypes.h -> cinttypes (#0) 
ggml-ci
 2024-11-17 08:30:29 +02:00
Georgi Gerganov 5d9e59979c ggml : adapt AMX to tensor->grad removal (#0) 
ggml-ci
 2024-11-17 08:30:29 +02:00
Georgi Gerganov a4200cafad make : add ggml-opt (#0) 
ggml-ci
 2024-11-17 08:30:29 +02:00
Georgi Gerganov 84274a10c3 tests : remove test-grad0 2024-11-17 08:30:29 +02:00
Georgi Gerganov 68fcb4759c ggml : fix compile warnings (#0) 
ggml-ci
 2024-11-17 08:30:29 +02:00
Georgi Gerganov 5c9a8b22b1 scripts : update sync 2024-11-17 08:30:29 +02:00
Georgi Gerganov db4cfd5dbc llamafile : fix include path (#0) 
ggml-ci
 2024-11-16 20:36:26 +02:00
Georgi Gerganov 8ee0d09ae6 make : auto-determine dependencies (#0) 2024-11-16 20:36:26 +02:00
Georgi GerganovandGitHub f245cc28d4 scripts : fix missing key in compare-llama-bench.py (#10332) 2024-11-16 10:32:50 +02:00
Georgi Gerganov 09ecbcb596 cmake : fix ppc64 check (whisper/0) 
ggml-ci
 2024-11-15 15:44:06 +02:00
Georgi Gerganov cbf5541a82 sync : ggml 2024-11-15 15:44:06 +02:00
Georgi Gerganov 4802ad350b scripts : fix regex in sync [no ci] 2024-11-15 08:38:43 +02:00
Georgi GerganovandGitHub 2a82891a85 speculative : fix out-of-bounds access (#10289) 2024-11-14 11:44:15 +02:00
Georgi Gerganov 5ea926dad7 sync : ggml 2024-11-13 18:11:54 +02:00
Georgi GerganovandGitHub b0cefea58a metal : more precise Q*K in FA vec kernel (#10247) 2024-11-11 08:39:13 +02:00
Georgi GerganovandGitHub b141e5f6ef server : enable KV cache defrag by default (#10233) 
ggml-ci
 2024-11-11 08:38:43 +02:00
Georgi GerganovGitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
 		4b3a9212b6 flake.lock: Update (#10243) 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/807e9154dcb16384b1b765ebe9cd2bba2ac287fd?narHash=sha256-l253w0XMT8nWHGXuXqyiIC/bMvh1VRszGXgdpQlfhvU%3D' (2024-10-29)
  → 'github:NixOS/nixpkgs/4aa36568d413aca0ea84a1684d2d46f55dbabad7?narHash=sha256-Zwl8YgTVJTEum%2BL%2B0zVAWvXAGbWAuXHax3KzuejaDyo%3D' (2024-11-05)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-11-10 11:45:25 -08:00
Georgi GerganovandGitHub 6423c65aa8 metal : reorder write loop in mul mat kernel + style (#10231) 
* metal : reorder write loop

* metal : int -> short, style

ggml-ci
 2024-11-09 11:53:13 +02:00
Georgi GerganovandGitHub 39a334a9aa metal : fix build and some more comments (#10229) 2024-11-09 11:53:02 +02:00
Georgi GerganovandGitHub bb38cdd8ba metal : fix F32 accumulation in FA vec kernel (#10232) 2024-11-09 11:52:45 +02:00
Georgi Gerganov f018acba22 llama : fix Qwen model type strings 2024-11-09 11:26:34 +02:00
Georgi Gerganov 46323fa9ef metal : hide debug messages from normal log 2024-11-09 11:21:49 +02:00
Georgi GerganovandGitHub ec450d3bbf metal : opt-in compile flag for BF16 (#10218) 
* metal : opt-in compile flag for BF16

ggml-ci

* ci : use BF16

ggml-ci

* swift : switch back to v12

* metal : has_float -> use_float

ggml-ci

* metal : fix BF16 check in MSL

ggml-ci
 2024-11-08 21:59:46 +02:00
Georgi Gerganov 695ad752b2 metal : improve clarity (minor) (#10171) 2024-11-08 18:37:41 +02:00
Georgi GerganovandGitHub 841f27abdb metal : optimize FA kernels (#10171) 
* ggml : add ggml_flash_attn_ext_get_prec

* metal : use F16 precision in FA kernels

ggml-ci

* metal : minor clean-up

* metal : compile-guard bf16 FA kernels

ggml-ci

* build : remove obsolete compile flag [no ci]

* metal : prevent int overflows [no ci]

* cuda : disable BF16 FA

ggml-ci

* metal : fix BF16 requirement for FA kernels

ggml-ci

* make : clean-up [no ci]
 2024-11-08 13:47:22 +02:00
Georgi Gerganov eec4d71737 scripts : add amx to sync-ggml.sh [no ci] 2024-11-07 23:11:36 +02:00
Georgi Gerganov 3b08828674 sync : ggml 2024-11-07 23:08:24 +02:00
Georgi Gerganov a2c6fd747c scripts : sync update 2024-11-07 23:07:55 +02:00
Georgi GerganovandGitHub 5c333e0140 metal : add BF16 support (#8439) 
* ggml : add initial BF16 support

ggml-ci

* metal : add mul_mat_id BF16 support

ggml-ci

* metal : check for bfloat support on the Metal device

ggml-ci

* metal : better var names [no ci]

* metal : do not build bfloat kernels when not supported

ggml-ci

* metal : try to fix BF16 support check

ggml-ci

* metal : this should correctly check bfloat support
 2024-11-06 19:53:51 +02:00
Georgi GerganovandGitHub b11f9ba9b8 server : remove hack for extra parallel slot (#10187) 
ggml-ci
 2024-11-06 13:29:01 +02:00
Georgi GerganovandGitHub 1dc04b2dee ggml : adjust is_first_call init value (#10193) 
ggml-ci
 2024-11-06 11:20:10 +02:00
Georgi GerganovandGitHub a1eaf6a960 metal : add quantized FA support (#10149) 
* metal : add quantized FA (vec) support

ggml-ci

* metal : add quantized FA (non-vec) support

* metal : fix support check

ggml-ci

* metal : clean-up

* metal : clean-up (cont)

* metal : fix shared memory calc + reduce smem + comments

* metal : float-correctness

* metal : minor [no ci]
 2024-11-06 10:24:23 +02:00
Georgi Gerganov 05697f670b metal : simplify f16 and f32 dequant kernels (#0) 2024-11-04 13:49:34 +02:00
Georgi Gerganov f8e58135cf metal : move dequantize templates to beginning of MSL source (#0) 2024-11-04 13:44:06 +02:00
Georgi Gerganov ce027adfb3 sync : ggml 2024-11-04 10:33:37 +02:00
Georgi GerganovandGitHub 08828a6d7d metal : minor fixup in FA kernel (#10143) 
* metal : minor fixup in FA kernel

ggml-ci

* metal : use the unrolled loop variable

* metal : remove unused var
 2024-11-03 15:18:40 +02:00
Georgi GerganovandGitHub 1839f69130 flake.lock: Update (#10146) 2024-11-03 05:14:15 -08:00
Georgi GerganovandGitHub 45950415ed server : fix endpoint checks (#10135) 
ggml-ci
 2024-11-02 18:34:00 +02:00
Georgi GerganovandGitHub 1926d6e39d llama : adjust default context size + print warnings (#10136) 
* llama : adjust default context size + print warnings

ggml-ci

* ggml-ci : add missing gpu-layers + adjust context sizes
 2024-11-02 15:18:56 +02:00
Georgi GerganovandGitHub ba6f62eb79 readme : update hot topics 2024-11-01 17:31:51 +02:00
Georgi GerganovandGitHub 1804adb0cf ggml : remove ggml_scratch (#10121) 
ggml-ci
 2024-11-01 12:58:45 +02:00
Georgi Gerganov 815fe72adc sync : ggml 2024-11-01 10:28:24 +02:00
Georgi Gerganov f221d56220 ggml : alloc ggml_contexts on the heap (whisper/2525) 2024-11-01 10:24:50 +02:00
Georgi GerganovandGitHub 8d8ff71536 llama : remove Tail-Free sampling (#10071) 
ggml-ci
 2024-10-29 10:42:05 +02:00
Georgi GerganovGitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
 		07028f9d74 flake.lock: Update (#10063) 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/4c2fcb090b1f3e5b47eaa7bd33913b574a11e0a0?narHash=sha256-/uilDXvCIEs3C9l73JTACm4quuHUsIHcns1c%2BcHUJwA%3D' (2024-10-18)
  → 'github:NixOS/nixpkgs/2768c7d042a37de65bb1b5b3268fc987e534c49d?narHash=sha256-AlcmCXJZPIlO5dmFzV3V2XF6x/OpNWUV8Y/FMPGd8Z4%3D' (2024-10-23)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-10-28 08:41:24 -07:00
Georgi GerganovandGitHub 8125e6cbfc server : don't overfill the batch during infill (#10018) 
ggml-ci
 2024-10-28 08:49:32 +02:00
Georgi GerganovandGitHub 8841ce3f43 llama : switch KQ multiplication to F32 precision by default (#10015) 
ggml-ci
 2024-10-27 20:59:58 +02:00
Georgi Gerganov cc2983d375 sync : ggml 2024-10-26 10:34:08 +03:00
Georgi Gerganov 9e4a2563ea scripts : fix amx sync [no ci] 2024-10-26 10:33:31 +03:00
Georgi GerganovandGitHub 668750357e metal : support permuted matrix multiplicaions (#10033) 
* metal : support permuted matrix multiplicaions

ggml-ci

* cont : use nb01 directly for row steps

ggml-ci

* cont : add comments [no ci]

* metal : minor refactor

* metal : minor
 2024-10-25 22:26:15 +03:00
Georgi GerganovandGitHub bc5ba007b2 server : check that the prompt fits in the slot's context (#10030) 
ggml-ci
 2024-10-25 10:13:46 +03:00
Georgi Gerganov 40f2555797 ci : fix cmake flags for SYCL 2024-10-24 21:23:33 +03:00
Georgi Gerganov 190a37d797 sync : ggml 2024-10-23 17:23:55 +03:00
Georgi Gerganov 2d3aba9ee8 llama.vim : bump generation time limit to 3s [no ci] 2024-10-23 17:16:56 +03:00
Georgi Gerganov e94a138d64 llama.vim : fix info text display [no ci] (#9787) 2024-10-22 00:37:55 +03:00
Georgi Gerganov e01c67affe llama.vim : move info to the right of screen [no ci] (#9787) 
'eol' messes up the rendering with nvim v0.10.2 for some reason
 2024-10-21 22:53:18 +03:00