Georgi Gerganov and GitHub
ab47dec3d3
security : add note about RPC and server functionality ( #13061 )
...
* security : add note about RPC functionality
* security : add note about llama-server
2025-04-22 16:16:10 +03:00
Georgi Gerganov and GitHub
7b53389c24
metal : add memory pool for temp allocs ( #12850 )
...
* metal : add memory pool for temp allocs (wip) [no ci]
* cont : free buffers from the heap
* cont : resize heap [no ci]
* cont : refactor heap [no ci]
* cont : heap for each cmd buffer [no ci]
* cont : fix free
* wip
* cont : fix alignment [no ci]
* cont : not working .. [no ci]
* cont : heap allocation now works [no ci]
* cont : use MTLHeapTypePlacement
ggml-ci
* metal : use dynamic MTLHeap allocations
ggml-ci
* metal : add comments
* metal : disable softmax use of mem_pool
ggml-ci
* metal : final touches
2025-04-22 16:15:51 +03:00
Georgi Gerganov and GitHub
2f74c354c0
graph : make FA compatible with MLA + add initial Metal kernels ( #12953 )
...
* graph : make mla compatible with FA
* metal : add exp FA kernels for DeepSeek models
ggml-ci
* llama : minor naming updates
ggml-ci
* ggml : disable FA for DS head sizes
* tests : add FA tests for MLA shapes
ggml-ci
2025-04-17 18:16:36 +03:00
Georgi Gerganov and GitHub
f8f820cc4d
metal : add FA-vec kernels for head size 96 ( #12952 )
...
ggml-ci
2025-04-15 14:45:05 +03:00
Georgi Gerganov
526739b879
sync : ggml
...
ggml-ci
2025-04-14 09:26:15 +03:00
Georgi Gerganov and GitHub
c94085df28
server : add VSCode's Github Copilot Chat support ( #12896 )
...
* server : add VSCode's Github Copilot Chat support
* cont : update handler name
2025-04-11 23:37:41 +03:00
Georgi Gerganov
47ba87d0a4
sync : ggml
2025-04-11 00:17:47 +03:00
Georgi Gerganov
1d2b613445
tests : fix init order ( #0 )
...
ggml-ci
2025-04-11 00:17:47 +03:00
Georgi Gerganov
eb420e1148
sync : ggml
...
ggml-ci
2025-04-11 00:17:47 +03:00
Georgi Gerganov
e4bf72d631
scripts : fix sync-ggml-am.sh
2025-04-11 00:17:47 +03:00
Georgi Gerganov and GitHub
47277d6d1d
readme : add rpc backend ( #12842 )
2025-04-09 10:54:42 +03:00
Georgi Gerganov and GitHub
a19b5cef16
llama : fix FA when KV cache is not used (i.e. embeddings) ( #12825 )
...
* ggml : FA supports F32 V
* graph : cast KV to F16 when the KV cache is not used
ggml-ci
* server : add test that exercises embeddings with FA enabled
ggml-ci
2025-04-08 19:54:51 +03:00
Georgi Gerganov
1a1ab7e7a4
cuda : fix HIP and MUSA BF16 ( #0 )
...
ggml-ci
2025-04-07 18:44:17 +03:00
Georgi Gerganov
a4e46e28f9
sync : ggml
...
ggml-ci
2025-04-07 18:44:17 +03:00
Georgi Gerganov
ff067dbcb9
ggml : simplify Arm fp16 CPU logic (ggml/1177)
...
* ggml : simlpify Arm fp16 CPU logic
ggml-ci
* cont : bring back CUDA/MUSA checks
ggml-ci
2025-04-07 18:44:17 +03:00
Georgi Gerganov and GitHub
3e1d29348b
kv-cache : simplify + fix warning for recurrent models ( #12756 )
...
ggml-ci
2025-04-04 21:48:10 +03:00
833e2b7409
model : print tensor size during load ( #12711 )
...
* model : print tensor size during load
* cont : fix units MB -> MiB
Co-authored-by: Diego Devesa <slarengh@gmail.com >
---------
Co-authored-by: Diego Devesa <slarengh@gmail.com >
2025-04-02 16:38:54 +03:00
Georgi Gerganov and GitHub
a10b36c91a
llama : refactor kv cache guard ( #12695 )
...
* llama : refactor kv cache guard
ggml-ci
* cont : fix comment [no ci]
* llama : fix kv_cache restore logic
ggml-ci
* context : simplify kv cache updates
ggml-ci
* cont : better name [no ci]
* llama : fix llama_decode return code when could not find KV slot
ggml-ci
* context : change log err -> warn [no ci]
* kv-cache : add comment + warning
2025-04-02 14:32:59 +03:00
Georgi Gerganov and GitHub
3fd072a540
metal : use F32 prec in FA kernels ( #12688 )
...
* metal : use F32 prec in FA kernels
ggml-ci
* cont : fix FA vec kernel
ggml-ci
2025-04-01 14:57:19 +03:00
Georgi Gerganov
1790e73157
cmake : fix whitespace ( #0 )
2025-03-31 15:07:32 +03:00
Georgi Gerganov
0114a32da0
sync : ggml
...
ggml-ci
2025-03-31 15:07:32 +03:00
Georgi Gerganov and GitHub
4663bd353c
metal : use constexpr in FA kernels + fix typedef ( #12659 )
...
* metal : use constexpr in FA kernels
ggml-ci
* cont
ggml-ci
* cont : fix typedef
ggml-ci
2025-03-30 22:04:04 +03:00
Georgi Gerganov
d3f1f0acfb
sync : ggml
...
ggml-ci
2025-03-30 08:33:31 +03:00
Georgi Gerganov and GitHub
b4ae50810e
metal : improve FA + improve MoE ( #12612 )
...
* ggml : FA with different K, V head sizes (CPU)
ggml-ci
* metal : add FA with HS=192
* metal : extend FA to support different K and V head sizes
ggml-ci
* metal : add FA vector kernels for heads K 192 and V 128
ggml-ci
* ggml : restrict op on other backends to equal head sizes
ggml-ci
* metal : optimize FA-vec kernel
ggml-ci
* metal : FA remove mq registers
* metal : improve MoE mul_mat_id condition
ggml-ci
* metal : fix comments + remove unnecessary addition
ggml-ci
* metal : avoid too much shared memory usage with mul_mat_id
ggml-ci
2025-03-28 20:21:59 +02:00
Georgi Gerganov and GitHub
2969019837
media : add SVG logo [no ci] ( #12616 )
2025-03-27 23:09:05 +02:00
Georgi Gerganov
029c693fdc
sync : ggml
...
ggml-ci
2025-03-27 10:09:29 +02:00
Georgi Gerganov
771d84371c
scripts : update sync + fix cmake merge
...
ggml-ci
2025-03-27 10:09:29 +02:00
Georgi Gerganov
df0665a483
sync : ggml
...
ggml-ci
2025-03-27 09:04:38 +02:00
Georgi Gerganov
0306aad1ca
cmake : sync/merge PowerPC build commands ( #0 )
2025-03-27 09:04:38 +02:00
Georgi Gerganov and GitHub
f28bc4c286
llama : make loras compatible with repacking ( #12593 )
...
* llama : make loras compatible with repacking
ggml-ci
* cont : simplify
ggml-ci
* cont : add TODO [no ci]
2025-03-27 08:24:10 +02:00
Georgi Gerganov and GitHub
b3298fa47a
metal : refactor mat-vec code ( #12569 )
...
* metal : refactor mat-vec code
ggml-ci
* metal : rename all_sum -> sum_all
ggml-ci
* metal : fix comments [no ci]
* metal : fix nr constant [no ci]
* metal : mv q6_K support nr0 > 1
ggml-ci
* metal : reduce register pressure
ggml-ci
* metal : fix typo [no ci]
* metal : reduce register pressure
ggml-ci
2025-03-26 21:38:38 +02:00
df4d20cd53
convert : fix squeeze for ssm_conv tensors ( #12573 )
...
* convert : fix squeeze for ssm_conv tensors
* convert : match ssm_conv tensors by type
---------
Co-authored-by: Francis Couture-Harpin <git@compilade.net >
2025-03-26 08:21:05 -04:00
Georgi Gerganov and GitHub
5ed38b6852
ggml : fix MUL_MAT_ID repack with Q8_K ( #12544 )
...
* ggml : fix MUL_MAT_ID repack with Q8_K
ggml-ci
* ggml : improve repack templates
ggml-ci
2025-03-26 13:02:00 +02:00
Georgi Gerganov and GitHub
2d77d88e70
context : fix worst-case reserve outputs ( #12545 )
...
ggml-ci
2025-03-25 09:19:23 +02:00
Georgi Gerganov and GitHub
ba932dfb50
ggml : fix quantized cpy op ( #12310 )
...
* ggml : fix quantized cpy op
ggml-ci
* tests : add cpy tests for all types
ggml-ci
* tests : add BF16 copy tests
ggml-ci
* tests : fix loop for same-type copy
ggml-ci
* tests : add option to permute the dst tensor
ggml-ci
2025-03-22 16:23:26 +02:00
Georgi Gerganov and GitHub
af04481e6b
model : do not repack if a GPU device is present ( #12498 )
...
ggml-ci
2025-03-21 16:14:29 +02:00
Georgi Gerganov and GitHub
75422e8bc4
graph : normalize Q, K, V shapes + sync cross attention ( #12449 )
...
* graph : normalize Q, K, V shapes and add comments
ggml-ci
* context : synchronize before getting cross attention data
* model : fix command-r attention norm check
2025-03-18 21:35:19 +02:00
Georgi Gerganov and GitHub
c6af2161b2
speculative : fix seg fault in certain cases ( #12454 )
2025-03-18 19:35:11 +02:00
Georgi Gerganov and GitHub
8551c44d84
context : always use non-causal attention for encoder graphs ( #12447 )
...
* context : always use non-causal attention for encoder graphs
ggml-ci
* context : move the change to llama_context::encode()
ggml-ci
2025-03-18 13:05:49 +02:00
Georgi Gerganov and GitHub
810e0af3f5
server : fix warmup draft cache type ( #12446 )
...
ggml-ci
2025-03-18 12:05:42 +02:00
Georgi Gerganov and GitHub
dc079cfdff
context : fix init of n_outputs ( #12397 )
...
ggml-ci
2025-03-16 19:29:36 +02:00
Georgi Gerganov and GitHub
c522ce4143
graph : simplify attn input build for unified KV cache ( #12381 )
...
ggml-ci
2025-03-14 10:47:44 +02:00
Georgi Gerganov and GitHub
081bee8c64
hparams : add SWA rope parameters ( #12374 )
...
ggml-ci
2025-03-14 09:03:24 +02:00
Georgi Gerganov and GitHub
84d5475541
llama : fix Gemma3 SWA KV cache shift ( #12373 )
...
* llama : fix Gemma3 SWA KV cache shift
ggml-ci
* hparams : add comment [no ci]
2025-03-13 19:08:07 +02:00
Georgi Gerganov and GitHub
e0dbec0bc6
llama : refactor llama_context, llama_kv_cache, llm_build_context ( #12181 )
...
* llama : refactor llama_context, llama_kv_cache, llm_build_context
ggml-ci
* graph : don't mutate the KV cache during defrag
ggml-ci
* context : reduce virtuals + remove test function
ggml-ci
* context : move interface implementation to source file + factory
ggml-ci
* graph : move KV cache build functions to llama_context impl
ggml-ci
* graph : remove model reference from build_pooling
ggml-ci
* graph : remove llama_model reference
ggml-ci
* kv_cache : provide rope factors
ggml-ci
* graph : rework inputs to use only unique_ptr, remove attn input abstraction
ggml-ci
* context : remove llama_context_i abstraction
ggml-ci
* context : clean-up
ggml-ci
* graph : clean-up
ggml-ci
* llama : remove redundant keywords (struct, enum)
ggml-ci
* model : adapt gemma3
ggml-ci
* graph : restore same attention ops as on master
ggml-ci
* llama : remove TODO + fix indent
ggml-ci
2025-03-13 12:35:44 +02:00
Georgi Gerganov and GitHub
e128a1bf5b
tests : fix test-quantize-fns to init the CPU backend ( #12306 )
...
ggml-ci
2025-03-10 14:07:15 +02:00
Georgi Gerganov and GitHub
1e2f78a004
server : add speculative decoding presets for FIM ( #12287 )
2025-03-09 19:08:20 +02:00
Georgi Gerganov and GitHub
0fd7ca7a21
authors : update ( #12271 )
2025-03-08 18:26:00 +02:00
Georgi Gerganov and GitHub
7ab364390f
server : infill gen ends on new line ( #12254 )
2025-03-07 20:54:30 +02:00
Georgi Gerganov
102ac1891d
sync : ggml
...
ggml-ci
2025-03-07 14:49:44 +02:00
Georgi Gerganov and GitHub
ea002810a2
ci : fix save-load test invocations ( #12245 )
2025-03-07 12:19:31 +02:00
Georgi Gerganov and GitHub
20a9b8f5e1
readme : fix roadmap link ( #12185 )
2025-03-04 18:42:44 +02:00
Georgi Gerganov
dfd6b2c0be
sync : ggml
...
ggml-ci
2025-03-03 18:18:11 +02:00
Georgi Gerganov
3d1cf3cf33
sync : ggml
...
ggml-ci
2025-03-03 18:18:11 +02:00
Georgi Gerganov
8371d44595
sync : ggml
...
ggml-ci
2025-03-03 18:18:11 +02:00
Georgi Gerganov
aede2074f6
scripts : sync-ggml-am.sh fix
2025-03-03 18:18:11 +02:00
Georgi Gerganov and GitHub
f3e64859ed
ci : fix arm upload artifacts ( #12024 )
...
* ci : fix arm upload artifacts
* cont : fix archive name to use matrix
2025-02-22 15:03:00 +02:00
Georgi Gerganov and GitHub
cf756d6e0a
server : disable Nagle's algorithm ( #12020 )
2025-02-22 11:46:31 +01:00
Georgi Gerganov and GitHub
51f311e057
llama : skip loading unused tensors ( #12004 )
...
* llama : assign unknown/unused tensors to host buffer type
ggml-ci
* llama : skip unused tensors
ggml-ci
2025-02-21 18:33:18 +02:00
Georgi Gerganov and GitHub
abd4d0bc4f
speculative : update default params ( #11954 )
...
* speculative : update default params
* speculative : do not discard the last drafted token
2025-02-19 13:29:42 +02:00
Georgi Gerganov and GitHub
c2cd24fbfd
readme : add notice about new package registry ( #11890 )
...
* readme : add notice about new package registry
* cont : fix whitespace
2025-02-15 20:29:56 +02:00
Georgi Gerganov and GitHub
68ff663a04
repo : update links to new url ( #11886 )
...
* repo : update links to new url
ggml-ci
* cont : more urls
ggml-ci
2025-02-15 16:40:57 +02:00
Georgi Gerganov and GitHub
dbc2ec59b5
docker : drop to CUDA 12.4 ( #11869 )
...
* docker : drop to CUDA 12.4
* docker : update readme [no ci]
2025-02-14 14:48:40 +02:00
Georgi Gerganov and GitHub
04045bb842
readme : minor
2025-02-14 00:16:56 +02:00
Georgi Gerganov
0fb77f821f
sync : ggml
2025-02-12 21:46:02 +02:00
Georgi Gerganov and GitHub
aaa5505307
server : minor log updates ( #11760 )
...
ggml-ci
2025-02-08 18:08:43 +02:00
Georgi Gerganov
bdcf8b6a56
cont : fix mmap flag print ( #11699 )
2025-02-08 16:49:38 +02:00
Georgi Gerganov and GitHub
ed926d8833
llama : fix defrag logic ( #11707 )
...
* llama : fix defrag logic
ggml-ci
* cont : better logic
ggml-ci
* cont : clamp fragmentation to 0.0
ggml-ci
2025-02-07 16:05:34 +02:00
Georgi Gerganov
8a59053f63
sync : ggml
2025-02-06 21:23:03 +02:00
Georgi Gerganov and GitHub
9dd7a0390f
llama : add log about loading model tensors ( #11699 )
2025-02-06 13:41:37 +02:00
Georgi Gerganov and GitHub
d774ab3acc
metal : adjust support conditions for norm operators ( #11671 )
...
cont #11659
ggml-ci
2025-02-05 10:57:42 +02:00
Georgi Gerganov
387a1598ca
authors : update
2025-02-04 13:04:10 +02:00
Georgi Gerganov
7c9e0ca520
sync : ggml
2025-02-04 12:59:21 +02:00
Georgi Gerganov
b34aedd558
ci : do not stale-close roadmap issues
2025-02-04 09:31:01 +02:00
Georgi Gerganov
8ec05832fa
sync : ggml
2025-02-03 14:57:08 +02:00
Georgi Gerganov
815857791d
sync : ggml
2025-01-29 11:25:29 +02:00
Georgi Gerganov and GitHub
178a7eb952
metal : use residency sets ( #11427 )
...
* metal : use residency sets
ggml-ci
* metal : restore commandBufferWithUnretainedReferences calls [no ci]
* metal : release descriptors
ggml-ci
* metal : check env GGML_METAL_NO_RESIDENCY
ggml-ci
* metal : fix build + clean-up
ggml-ci
2025-01-26 20:06:16 +02:00
Georgi Gerganov and GitHub
2cc9b8c32c
readme : update hot topics
2025-01-26 14:30:15 +02:00
Georgi Gerganov and GitHub
00c24acb2a
ci : fix line breaks on windows builds ( #11409 )
...
* ci : fix line breaks on windows builds
* cont : another try
* ci : fix powershell line breaks
2025-01-25 13:36:48 +02:00
Georgi Gerganov and GitHub
9755129c27
release : pack /lib in the packages ( #11392 )
...
* release : pack /lib and /include in the packages
* cmake : put libs in /bin
* TMP : push artifacts
* Revert "TMP : push artifacts"
This reverts commit 4decf2c4dfc5cdf5d96ea44c03c8f9801ab41262.
* ci : fix HIP cmake compiler options to be on first line
* ci : restore the original HIP commands
* ci : change ubuntu build from latest to 20.04
* ci : try to fix macos build rpaths
* ci : remove obsolete MacOS build
* TMP : push artifacts
* ci : change back to ubuntu latest
* ci : macos set build rpath to "@loader_path"
* ci : fix typo
* ci : change ubuntu package to 22.04
* Revert "TMP : push artifacts"
This reverts commit 537b09e70ffc604c414ee78acf3acb4c940ec597.
2025-01-24 18:41:30 +02:00
Georgi Gerganov and GitHub
16d3df7ab0
readme : add plugin links ( #11355 )
2025-01-22 19:44:26 +02:00
Georgi Gerganov and GitHub
2139667ec4
metal : fix out-of-bounds write ( #11314 )
...
ggml-ci
2025-01-21 08:48:13 +02:00
Georgi Gerganov and GitHub
80d0d6b4b7
common : add -hfd option for the draft model ( #11318 )
...
* common : add -hfd option for the draft model
* cont : fix env var
* cont : more fixes
2025-01-20 22:29:43 +02:00
Georgi Gerganov and GitHub
9f7add1cde
examples : fix add_special conditions ( #11311 )
2025-01-20 16:36:08 +02:00
Georgi Gerganov
ef6dada60c
cont : fix whitespaces ( #11305 )
2025-01-20 09:29:32 +02:00
Georgi Gerganov and GitHub
92bc493917
tests : increase timeout when sanitizers are enabled ( #11300 )
...
* tests : increase timeout when sanitizers are enabled
* tests : add DEFAULT_HTTP_TIMEOUT
2025-01-19 20:22:30 +02:00
Georgi Gerganov and GitHub
b9daaffe02
simple-chat : fix BOS being added to each message ( #11278 )
2025-01-19 18:12:09 +02:00
4dd34ff831
cmake : add sanitizer flags for llama.cpp ( #11279 )
...
* cmake : add sanitizer flags for llama.cpp
ggml-ci
* tests : fix compile warnings
ggml-ci
* cmake : move sanitizer flags to llama_add_compile_flags
ggml-ci
* cmake : move llama.cpp compile flags to top level lists
ggml-ci
* cmake : apply only sanitizer flags at top level
ggml-ci
* tests : fix gguf context use in same_tensor_data
* gguf-test: tensor data comparison
* dummy : trigger ggml-ci
* unicode : silence gcc warnings
ggml-ci
* ci : use sanitizer builds only in Debug mode
ggml-ci
* cmake : add status messages [no ci]
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
2025-01-18 16:18:15 +02:00
Georgi Gerganov and GitHub
f26c874179
scripts : restore hf.sh ( #11288 )
...
ggml-ci
2025-01-18 13:18:32 +02:00
Georgi Gerganov and GitHub
a133566d34
vocab : fix double-eos check ( #11273 )
...
ggml-ci
2025-01-17 09:28:00 +02:00
Georgi Gerganov and GitHub
f11cfdfd7f
ci : use -no-cnv in gguf-split tests ( #11254 )
...
* ci : use -no-cnv in gguf-split tests
ggml-ci
* ci : use -no-cnv in requantize tests
ggml-ci
* scripts : fix [no ci]
2025-01-15 18:28:35 +02:00
Georgi Gerganov and GitHub
bbf3e55e35
vocab : add dummy tokens for "no_vocab" type ( #11231 )
...
* vocab : add dummy tokens for "no_vocab" type
ggml-ci
* vocab : minor [no ci]
2025-01-14 11:54:58 +01:00
Georgi Gerganov
44d1e796d0
sync : ggml
2025-01-14 10:39:42 +02:00
Georgi Gerganov
a4f3f5d8e6
scripts : sync gguf (cont)
2025-01-14 09:40:52 +02:00
Georgi Gerganov
48e1ae0e61
scripts : sync gguf
2025-01-14 09:36:58 +02:00
Georgi Gerganov
d00a80e89d
scripts : sync opencl
2025-01-14 09:19:58 +02:00
Georgi Gerganov
a29f0870d4
contrib : add naming guidelines (cont) ( #11177 )
2025-01-13 15:59:26 +02:00
Georgi Gerganov and GitHub
ca001f6656
contrib : add naming guidelines (cont) ( #11177 )
2025-01-13 15:08:44 +02:00
Georgi Gerganov and GitHub
7426a26b24
contrib : add naming guidelines ( #11177 )
...
* contrib : add naming guidelines
* contrib : expand naming guidelines [no ci]
* contrib : cont [no ci]
* contrib : add `_t` suffix guideline [no ci]
* contrib : cont [no ci]
* minor [no ci]
* contrib : move coding guidelines to correct section [no ci]
* contrib : minor reword coding guidelines [no ci]
* contrib : add TODO for preprocessor directives [no ci]
* contrib : expand [no ci]
* minor [no ci]
* contrib : clarify `_context` suffix usage [no ci]
* contrib : filename guidelines [no ci]
* contrib : fix notes [no ci]
2025-01-13 14:46:36 +02:00
Georgi Gerganov and GitHub
08f10f69c3
llama : remove notion of CLS token ( #11064 )
...
ggml-ci
2025-01-12 12:15:53 +02:00
