Georgi Gerganov
2df255da3c
sync : ggml
...
ggml-ci
2025-07-24 20:27:23 +03:00
Georgi Gerganov and GitHub
e4868d16d2
context : perform output reorder lazily upon access after sync ( #14853 )
...
* context : perform output reorder after lazily upon access after sync
ggml-ci
* cont : add TODO
2025-07-24 16:31:48 +03:00
Georgi Gerganov and GitHub
065908cb09
metal : fix fusion across different encoders ( #14849 )
...
* metal : fix fusion across different encoders
ggml-ci
* cont : add assertion
ggml-ci
2025-07-24 10:24:05 +03:00
Georgi Gerganov
18f3b5ff9e
tests : add non-cont K,V FA tests
...
ggml-ci
2025-07-23 14:08:09 +03:00
Georgi Gerganov
b17230917c
sync : ggml
2025-07-19 11:46:50 +03:00
Georgi Gerganov and GitHub
bf9087f59a
metal : fuse add, mul + add tests ( #14596 )
...
ggml-ci
2025-07-18 20:37:26 +03:00
Georgi Gerganov and GitHub
9fb1042ce6
graph : fix graph reuse reset of params ( #14760 )
...
ggml-ci
2025-07-18 20:08:33 +03:00
Georgi Gerganov and GitHub
2adf8d83ac
parallel : add option for different RNG seeds ( #14757 )
...
ggml-ci
2025-07-18 17:33:41 +03:00
Georgi Gerganov and GitHub
d498af3d5a
graph : avoid huge warm-up graphs for MoE models ( #14753 )
...
* graph : avoid huge warm-up graphs for MoE models
ggml-ci
* cont : bump max nodes to 8x model tensors
2025-07-18 14:31:15 +03:00
Georgi Gerganov and GitHub
eacdeb5bfc
model : fix build after merge conflict ( #14754 )
2025-07-18 11:53:55 +03:00
Georgi Gerganov and GitHub
8f974bc1e9
graph : refactor context to not pass gf explicitly ( #14629 )
...
ggml-ci
2025-07-18 08:29:28 +03:00
Georgi Gerganov and GitHub
d6fb3f6b49
kv-cache : fix k-shift for multiple streams ( #14742 )
...
ggml-ci
2025-07-17 20:52:33 +03:00
Georgi Gerganov and GitHub
01612b7409
llama : reuse compute graphs ( #14482 )
...
* llama : reuse compute graphs
ggml-ci
* llama-bench : add graph reuse parameter
ggml-ci
* cont : remove the parameter and the sched resets
ggml-ci
* graph : rename update() to can_reuse()
ggml-ci
* params : remove is_same()
ggml-ci
* graph : set res->params in llm_graph_context constructor
ggml-ci
* graph : avoid set_max_nodes in llm_graph_result
ggml-ci
* kv-cache : reuse llama_context's graph result instance
ggml-ci
* context : reset the previous graph result upon memory updates
ggml-ci
* batch : llama_ubatch now carries its data instead of pointing to balloc
ggml-ci
* merge : fix build
ggml-ci
* graph : fix can_reuse() checks when flash-attention is disabled
* graph : move llm_graph_result impl in source file + debug env
ggml-ci
2025-07-17 19:08:33 +03:00
Georgi Gerganov and GitHub
d9b691081c
kv-cache : opt mask set input ( #14600 )
...
ggml-ci
2025-07-17 09:49:15 +03:00
Georgi Gerganov and GitHub
ad57d3edd2
batch : fix uninitialized has_cpl flag ( #14733 )
...
ggml-ci
2025-07-17 09:45:54 +03:00
225e7a1438
llama : add high-throughput mode ( #14363 )
...
* kv-cache : prepare K/V buffers for separation
ggml-ci
* batched-bench : fix oob write
ggml-ci
* llama : add "virtual sequences"
ggml-ci
* llama : use "stream" vs "virtual sequence"
ggml-ci
* graph : fix stream splitting when KV cache is not used
ggml-ci
* kv-cache : add multi-stream save/load support
ggml-ci
* llama : add "--attn-streams" flag
ggml-ci
* kv-cache : fix handling when find_slot fails
ggml-ci
* kv-cache : restore find_slot impl
ggml-ci
* kv-cache : add comments
* kv-cache : add bounds checks for sequence id
ggml-ci
* cont : add n_seq_max to batch allocr
ggml-ci
* kv-cache : perform stream copies lazily after llama_synchronize
ggml-ci
* kv-cache : avoid throwing exceptions across the C boundary
ggml-ci
* CUDA: 4D FlashAttention support (#14628 )
* CUDA: 4D FlashAttention support
* CUDA: fix WMMA FA kernel
* llama : rename attn_streams -> kv_unified
ggml-ci
* common : rename kv_split -> kv_unified
ggml-ci
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
2025-07-16 16:35:42 +03:00
64978340b0
ggml : add asserts ( #14720 )
...
* ggml : add asserts
ggml-ci
* cont : fix constant type
Co-authored-by: Diego Devesa <slarengh@gmail.com >
---------
Co-authored-by: Diego Devesa <slarengh@gmail.com >
2025-07-16 14:43:32 +03:00
Georgi Gerganov and GitHub
6ffd4e9c44
server : pre-calculate EOG logit biases ( #14721 )
...
ggml-ci
2025-07-16 14:04:12 +03:00
Georgi Gerganov and GitHub
538cc77f7f
server : fix handling of the ignore_eos flag ( #14710 )
...
ggml-ci
2025-07-16 12:13:57 +03:00
Georgi Gerganov and GitHub
05fec5bd29
ggml : add build-time message to remind about ggml_set_rows ( #14661 )
...
ggml-ci
2025-07-13 10:36:33 +03:00
Georgi Gerganov
8eff95544e
sync : ggml
2025-07-12 16:13:27 +03:00
Georgi Gerganov
3120413ccd
vulkan : remove unused vars ( #0 )
...
ggml-ci
2025-07-12 14:25:44 +03:00
Georgi Gerganov
215535701d
sync : ggml
...
ggml-ci
2025-07-12 14:25:44 +03:00
Georgi Gerganov and GitHub
aaa088d87f
readme : add hot PRs ( #14636 )
...
* readme : add hot PRs
* cont
* readme : update title
* readme : hot PRs links
* cont
2025-07-11 16:07:55 +03:00
Georgi Gerganov and GitHub
0d5375d54b
llama : move enum llama_vocab_pre_type to implementation ( #14631 )
...
ggml-ci
2025-07-11 13:46:07 +03:00
Georgi Gerganov and GitHub
4d0dcd4a06
cuda : fix rope with partial rotation and non-cont src ( #14580 )
...
* cuda : fix rope non-cont
ggml-ci
* cont : fix multi-rope + add test
ggml-ci
* sycl : try fix
ggml-ci
* cont : fix sycl + clean-up cuda
ggml-ci
2025-07-08 10:15:21 +03:00
Georgi Gerganov and GitHub
bac8bed248
eval-callback : check for empty input ( #14539 )
2025-07-05 07:18:09 +03:00
Georgi Gerganov and GitHub
ef797db357
metal : disable fast math in all quantize kernels ( #14528 )
...
ggml-ci
2025-07-04 19:19:09 +03:00
Georgi Gerganov and GitHub
67d1ef23c6
batch : add optional for sequential equal split ( #14511 )
...
ggml-ci
2025-07-04 09:08:59 +03:00
Georgi Gerganov and GitHub
7b50f7c025
graph : prepare for 4D mask ( #14515 )
...
ggml-ci
2025-07-04 09:05:36 +03:00
Georgi Gerganov and GitHub
c79184d2d1
batch : add n_used count ( #14512 )
...
ggml-ci
2025-07-04 09:04:59 +03:00
Georgi Gerganov and GitHub
a70c8a0c4b
kv-cache : use ggml_set_rows ( #14285 )
...
* kv-cache : use ggml_set_rows
ggml-ci
* graph : separate k and v indices
ggml-ci
* cont : remove redundant ifs
ggml-ci
* kv-cache : improve find_slot impl
* kv-cache : bounds-check when accessing slot_info indices
* kv-cache : add comments
ggml-ci
* ggml : add TODOs for adding GGML_OP_SET_ROWS support in the backends
ggml-ci
2025-07-03 10:53:35 +03:00
Georgi Gerganov and GitHub
9067487c44
ggml : fix FA mask dim 2 and 3 ( #14505 )
...
* ggml : fix FA mask dim 2 and 3
ggml-ci
* backends : unsupport batched FA in CUDA and Vulkan
ggml-ci
* vulkan : disable FA for mask->ne[2] != 1
2025-07-03 10:46:57 +03:00
Georgi Gerganov and GitHub
d4cdd9c1c3
ggml : remove kompute backend ( #14501 )
...
ggml-ci
2025-07-03 07:48:32 +03:00
Georgi Gerganov
e17991c466
sync : ggml
...
ggml-ci
2025-07-02 20:08:45 +03:00
Georgi Gerganov
ec68e84c32
ggml : support bcast ggml_soft_max_ext, ggml_flash_attn_ext ( #14435 )
...
ggml-ci
2025-07-02 15:48:33 +03:00
Georgi Gerganov and GitHub
d7f5f4e578
simple-chat : fix context-exceeded condition ( #14494 )
...
* simple-chat : fix context-exceeded condition
ggml-ci
* cont : fix n_ctx_used computation
ggml-ci
2025-07-02 14:12:07 +03:00
Georgi Gerganov and GitHub
de56944147
ci : disable fast-math for Metal GHA CI ( #14478 )
...
* ci : disable fast-math for Metal GHA CI
ggml-ci
* cont : remove -g flag
ggml-ci
2025-07-01 18:04:08 +03:00
Georgi Gerganov
a6a47958a1
ggml : remove trailing whitespace ( #0 )
2025-07-01 11:06:39 +03:00
Georgi Gerganov
f61c05d4b1
sync : ggml
...
ggml-ci
2025-07-01 11:06:39 +03:00
Georgi Gerganov and GitHub
745f11fed0
memory : correctly handle failure in apply() ( #14438 )
...
ggml-ci
2025-06-30 18:03:03 +03:00
Georgi Gerganov and GitHub
5dd942de59
metal : disable fast-math for some cpy kernels ( #14460 )
...
* metal : disable fast-math for some cpy kernels
ggml-ci
* cont : disable for q4_1
ggml-ci
* cont : disable for iq4_nl
ggml-ci
2025-06-30 17:04:05 +03:00
Georgi Gerganov and GitHub
72babea5de
graph : make llm_graph_context destructor virtual ( #14410 )
...
ggml-ci
2025-06-27 21:42:02 +03:00
Georgi Gerganov and GitHub
43678060c1
recurrent : call balloc split_reset() in init_batch() ( #14414 )
...
ggml-ci
2025-06-27 17:55:45 +03:00
Georgi Gerganov and GitHub
e8215dbb96
metal : add special-case mat-vec mul for ne00 == 4 ( #14385 )
...
ggml-ci
2025-06-26 15:51:19 +03:00
Georgi Gerganov and GitHub
5783ae4359
metal : batch rows copy in a single threadgroup ( #14384 )
...
* metal : batch rows copy in a single threadgroup
ggml-ci
* metal : handle some edge cases when threadgroup size is not a power of 2
ggml-ci
2025-06-26 15:50:15 +03:00
Georgi Gerganov and GitHub
62af464227
batch : fix check for empty sequences in memory ( #14364 )
...
* batch : fix check for empty sequences in memory
ggml-ci
* cont : reuse the var
ggml-ci
2025-06-24 18:26:30 +03:00
Georgi Gerganov and GitHub
7b50d589a8
kv-cells : fix tracking of seq_pos ( #14339 )
...
* kv-cells : fix tracking of seq_pos during cache reuse
ggml-ci
* cont : improve error message
ggml-ci
* cont : add more comments
2025-06-23 12:27:35 +03:00
Georgi Gerganov and GitHub
f1f5e82df6
examples : fix is_first logic for tokenization ( #14329 )
...
ggml-ci
2025-06-22 20:10:07 +03:00
Georgi Gerganov and GitHub
67ae5312e2
metal : fix thread-safety ( #14300 )
...
ggml-ci
2025-06-21 08:04:18 +03:00
Georgi Gerganov and GitHub
692e3cdd0a
memory : rename interface to llama_memory_context_i ( #14296 )
...
* memory : rename interface to llama_memory_context_i
ggml-ci
* cont : fix comments
* cont : use "mctx" for referencing a memory context
ggml-ci
2025-06-21 08:03:46 +03:00
Georgi Gerganov
06cbedfca1
sync : ggml
...
ggml-ci
2025-06-20 21:02:47 +03:00
Georgi Gerganov and GitHub
d27b3ca175
ggml : fix repack work size for mul_mat_id ( #14292 )
...
ggml-ci
2025-06-20 11:19:15 +03:00
Georgi Gerganov and GitHub
812939a9e9
model : more uniform output id handling ( #14275 )
...
* model : more uniform output id handling
ggml-ci
* cont : revert n_outputs < n_tokens optimization
ggml-ci
* cont : fix out_ids initialization
ggml-ci
2025-06-20 10:50:27 +03:00
Georgi Gerganov and GitHub
4c9fdfbe15
ubatch : new splitting logic ( #14217 )
...
ggml-ci
2025-06-20 10:14:14 +03:00
Georgi Gerganov and GitHub
ed3290ab34
metal : add mean kernel ( #14267 )
...
* metal : add mean kernel
ggml-ci
* cont : dedup implementation
ggml-ci
2025-06-19 08:05:21 +03:00
Georgi Gerganov
d03172cc79
sync : ggml
...
ggml-ci
2025-06-18 09:59:21 +03:00
Georgi Gerganov and GitHub
89fea80d29
server : fix incorrect usage of llama_get_embeddings() ( #14225 )
...
* server : fix incorrect usage of llama_get_embeddings()
ggml-ci
* cont : fix the fix
ggml-ci
2025-06-16 22:33:27 +03:00
Georgi Gerganov and GitHub
d3e64b9f49
llama : rework embeddings logic ( #14208 )
...
* llama : rework embeddings logic
ggml-ci
* cont : fix rerank
ggml-ci
* cont : engrish [no ci]
* cont : fix rerank
ggml-ci
* server : support both embeddings and completions with single model
ggml-ci
* cont : avoid embeddings_org
ggml-ci
2025-06-16 14:14:00 +03:00
Georgi Gerganov and GitHub
5fce5f948d
kv-cache : fix use-after-move of defrag info ( #14189 )
...
ggml-ci
2025-06-15 10:52:11 +03:00
Georgi Gerganov and GitHub
c311ac664d
cparams : rename LLAMA_MAX_PARALLEL_SEQUENCES to LLAMA_MAX_SEQ ( #14188 )
...
ggml-ci
2025-06-15 10:08:58 +03:00
Georgi Gerganov and GitHub
b9912ac570
batch : auto-gen positions + verify multi-sequence input ( #14177 )
...
* batch : verify multi-sequence input batches
ggml-ci
* cont : auto-gen positions + verify multi-seq input
ggml-ci
* cont : first print debug info, then perform validation
ggml-ci
* cont : fix position auto-gen + add comments
ggml-ci
2025-06-15 09:18:37 +03:00
Georgi Gerganov and GitHub
fb85a288d7
vocab : fix build ( #14175 )
...
ggml-ci
2025-06-13 20:03:05 +03:00
Georgi Gerganov and GitHub
80709b70a2
batch : add LLAMA_BATCH_DEBUG environment variable ( #14172 )
...
* batch : add LLAMA_BATCH_DEBUG environment variable
ggml-ci
* cont : improve seq_id display
2025-06-13 18:35:00 +03:00
Georgi Gerganov and GitHub
60c666347b
batch : rework llama_batch_allocr ( #14153 )
...
* batch : rework llama_batch_allocr
ggml-ci
* cont : move validation inside class
ggml-ci
* cont : move output counting to class
ggml-ci
* cont : minor
ggml-ci
* batch : add TODOs
ggml-ci
2025-06-13 13:47:55 +03:00
Georgi Gerganov and GitHub
b7cc7745e3
readme : remove survey link ( #14168 )
2025-06-13 11:55:44 +03:00
Georgi Gerganov and GitHub
ffad043973
server : fix SWA condition for full context reprocess ( #14163 )
...
ggml-ci
2025-06-13 11:18:25 +03:00
Georgi Gerganov and GitHub
c33fe8b8c4
vocab : prevent heap overflow when vocab is too small ( #14145 )
...
ggml-ci
2025-06-13 08:03:54 +03:00
Georgi Gerganov and GitHub
a681b4ba83
readme : remove project status link ( #14149 )
2025-06-12 14:43:09 +03:00
Georgi Gerganov and GitHub
7d516443dd
server : re-enable SWA speculative decoding ( #14131 )
...
ggml-ci
2025-06-12 11:51:38 +03:00
Georgi Gerganov and GitHub
f6e1a7aa87
context : simplify output counting logic during decode ( #14142 )
...
* batch : remove logits_all flag
ggml-ci
* context : simplify output counting logic during decode
ggml-ci
* cont : fix comments
2025-06-12 11:50:01 +03:00
Georgi Gerganov and GitHub
c3ee46fab4
batch : remove logits_all flag ( #14141 )
...
ggml-ci
2025-06-12 11:49:26 +03:00
e2c0b6e46a
cmake : handle whitepsaces in path during metal build ( #14126 )
...
* cmake : handle whitepsaces in path during metal build
ggml-ci
* cont : proper fix
ggml-ci
---------
Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com >
2025-06-12 10:14:24 +03:00
Georgi Gerganov and GitHub
9596506965
kv-cache : fix split_equal handling in unified implementation ( #14130 )
...
ggml-ci
2025-06-12 10:02:15 +03:00
Georgi Gerganov and GitHub
89a184fa71
kv-cache : relax SWA masking condition ( #14119 )
...
ggml-ci
2025-06-11 16:48:45 +03:00
Georgi Gerganov and GitHub
7ae2932116
kv-cache : add LLAMA_KV_CACHE_DEBUG environment variable ( #14121 )
2025-06-11 12:52:45 +03:00
Georgi Gerganov
ae92c1855b
sync : ggml
...
ggml-ci
2025-06-10 18:39:33 +03:00
Georgi Gerganov
b7ce1ad1e3
ggml : fix weak alias win32 (whisper/0)
...
ggml-ci
2025-06-10 18:39:33 +03:00
Georgi Gerganov
b8e2194efc
sync : ggml
...
ggml-ci
2025-06-10 09:21:56 +03:00
Georgi Gerganov and GitHub
1f63e75f3b
metal : use less stack memory in FA kernel ( #14088 )
...
* metal : use less stack memory in FA kernel
ggml-ci
* cont : fix BF16 variant
2025-06-09 23:05:02 +03:00
Georgi Gerganov and GitHub
40cbf571c9
kv-cache : fix shift and defrag logic ( #14081 )
...
* kv-cache : fix shift
ggml-ci
* cont : reset shift[i]
ggml-ci
* cont : fix defrag erasing cells that didn't move
ggml-ci
2025-06-09 23:04:35 +03:00
Georgi Gerganov and GitHub
201b31dc2e
graph : fix geglu ( #14077 )
...
ggml-ci
2025-06-09 17:17:31 +03:00
Georgi Gerganov and GitHub
87d34b381d
server : fix LRU check ( #14079 )
...
ggml-ci
2025-06-09 12:57:58 +03:00
Georgi Gerganov and GitHub
745aa5319b
llama : deprecate llama_kv_self_ API ( #14030 )
...
* llama : deprecate llama_kv_self_ API
ggml-ci
* llama : allow llama_memory_(nullptr)
ggml-ci
* memory : add flag for optional data clear in llama_memory_clear
ggml-ci
2025-06-06 14:11:15 +03:00
Georgi Gerganov and GitHub
487a5e0401
context : fix SWA-related warning for multiple sequences ( #14045 )
2025-06-06 13:29:18 +03:00
Georgi Gerganov and GitHub
7f37b6cf1e
memory : migrate from llama_kv_cache to more generic llama_memory ( #14006 )
...
* memory : merge llama_kv_cache into llama_memory + new `llama_memory` API
ggml-ci
* context : fix casts
ggml-ci
2025-06-05 15:29:22 +03:00
Georgi Gerganov and GitHub
9e31bec4fd
context : fix pos_min initialization upon error decode ( #14008 )
...
ggml-ci
2025-06-05 09:06:29 +03:00
Georgi Gerganov and GitHub
3e63a58ef7
kv-cache : refactor the update/defrag mechanism ( #13988 )
...
* kv-cache : refactor update mechanism
ggml-ci
* memory : improve status handling
* defrag : reset head + add comments
ggml-ci
* cont : minor fixes
ggml-ci
2025-06-04 18:58:20 +03:00
Georgi Gerganov and GitHub
e0e806f52e
kv-cache : fix unified::seq_rm to work with seq_id < 0 ( #13985 )
...
ggml-ci
2025-06-04 09:50:32 +03:00
Georgi Gerganov and GitHub
3637576288
server : disable speculative decoding for SWA models ( #13970 )
...
* server : use swa-full fo draft context
ggml-ci
* server : disable speculative decoding for SWA models
2025-06-02 21:34:40 +03:00
Georgi Gerganov and GitHub
ea394d7ab1
metal : use F32 accumulators in FA kernels ( #13975 )
...
ggml-ci
2025-06-02 21:33:40 +03:00
Georgi Gerganov and GitHub
5582c49c39
gemma : more consistent attention scaling for v2 and v3 ( #13951 )
...
* gemma : fix attn scale for 27B
* cont : apply scale before attn
* cont : consistent attention scaling
2025-06-02 20:54:26 +03:00
Georgi Gerganov
f3a4b1659c
sync : ggml
...
ggml-ci
2025-06-01 13:43:57 +03:00
Georgi Gerganov and Daniel Tang
a7b8d35f78
sync : whisper.cpp (ggml/1250)
...
* ggml : Fix backtrace breaking Windows build (whisper/3203)
* sync : whisper.cpp
ggml-ci
---------
Co-authored-by: Daniel Tang <danielzgtg.opensource@gmail.com >
2025-06-01 13:43:57 +03:00
Georgi Gerganov and GitHub
c04621711a
parallel : fix n_junk == 0 ( #13952 )
2025-06-01 11:42:16 +03:00
Georgi Gerganov and GitHub
0fc16b42e8
kv-cache : split implementation in separate sources ( #13920 )
...
ggml-ci
2025-06-01 11:39:27 +03:00
Georgi Gerganov and GitHub
803f8baf4f
llama : deprecate explicit kv_self defrag/update calls ( #13921 )
...
ggml-ci
2025-05-31 15:58:33 +03:00
Georgi Gerganov and GitHub
3600cc2886
llama : use n_swa + n_ubatch cells for SWA cache ( #13833 )
...
* llama : use n_swa + n_ubatch cells for SWA cache
ggml-ci
* llama : add warning about multi-sqeuence SWA contexts
2025-05-31 15:57:44 +03:00
Georgi Gerganov and GitHub
3f55f781f1
llama : auto-batch preparation ( #13845 )
...
* llama : auto-batch
ggml-ci
* context : simplify if branching
2025-05-31 12:55:57 +03:00
Georgi Gerganov and GitHub
12d0188c0d
kv-cache : refactor + add llama_memory_state_i ( #13746 )
...
* kv-cache : simplify the "struct llama_kv_cache" interface
ggml-ci
* kv-cache : revert the (n_swa + n_ubatch) change (for next PR)
ggml-ci
* kv-cache : some comments
ggml-ci
* context : fix graph reserve for multiple sequences
ggml-ci
* kv-cache : fix typo [no ci]
* kv-cache : fix find_slot() logic for free slots
ggml-ci
* llama : add TODO for deprecating the defrag API in the future
* kv-cache : improve find_slot() using min/max seq pos info
ggml-ci
* llama : handle aborts and compute errors
ggml-ci
* memory : extract state into llama_memory_state
ggml-ci
* kv-cache : add comments
ggml-ci
* server : update batching logic to reset n_batch on successful decode
* server : upon full re-processing, remove the sequence from the cache
* kv-cache : add TODO for doing split_equal when split_simple fails
ggml-ci
2025-05-31 10:24:04 +03:00
Georgi Gerganov