Georgi Gerganov and GitHub
053174436f
server : passthrough the /models endpoint during loading ( #13535 )
...
* server : passthrough the /models endpoint during loading
* server : update readme + return json for "meta" field
2025-05-14 15:42:10 +03:00
Georgi Gerganov and GitHub
f0995d28ce
metal : use FA-vec kernel up to batch size 20 ( #13496 )
...
* batched-bench : fix pp batch contents
* metal : optimize multi-sequence FA vec kernel
ggml-ci
* metal : use FA-vec kernel up to batch size 20
ggml-ci
2025-05-13 18:04:39 +03:00
Georgi Gerganov and GitHub
c252e0c409
metal : optimize multi-sequence FA vec kernel ( #13493 )
...
* batched-bench : fix pp batch contents
* metal : optimize multi-sequence FA vec kernel
ggml-ci
2025-05-13 18:04:00 +03:00
Georgi Gerganov and GitHub
b89d605a91
batched-bench : fix pp batch contents ( #13492 )
2025-05-13 18:01:53 +03:00
Georgi Gerganov
1e2809bc4b
sync : ggml
2025-05-13 14:02:28 +03:00
Georgi Gerganov and GitHub
064cc596ac
context : fix state io for memory-less contexts ( #13470 )
...
ggml-ci
2025-05-12 15:12:27 +03:00
Georgi Gerganov and GitHub
611aa914ef
metal : optimize MoE for large batches ( #13388 )
...
ggml-ci
2025-05-09 15:14:56 +03:00
Georgi Gerganov and GitHub
6562e5a4d6
context : allow cache-less context for embeddings ( #13108 )
...
* context : allow cache-less context for embeddings
ggml-ci
* context : enable reranking with encode()
ggml-ci
* context : encode() clears embd_seq
ggml-ci
* examples : use llama_encode() when appropriate
ggml-ci
* models : nomic bert moe does not require KV cache
* llama : update comments for llama_decode/llama_encode
ggml-ci
* context : update warning log [no ci]
2025-05-08 14:28:33 +03:00
Georgi Gerganov and GitHub
51fb96b1ff
context : remove logits_all flag ( #13284 )
...
* context : remove logits_all flag
ggml-ci
* llama : remove logits_all flag + reorder llama_context_params
ggml-ci
2025-05-08 14:26:50 +03:00
Georgi Gerganov
d879433824
sync : ggml
...
ggml-ci
2025-05-07 17:28:36 +03:00
Georgi Gerganov and GitHub
4773d7a02f
examples : remove infill ( #13283 )
...
ggml-ci
2025-05-07 10:28:02 +03:00
b34443923c
sync : ggml ( #13268 )
...
* vulkan : kernels for depthwise 2D convolution (CONV_2D_DW) (ggml/1204)
* vulkan : add kernels for depthwise 2d convolution (OP_CONV_2D_DW)
* review: remove src_x/y < 0 checks; add performance tests
* sync : ggml
ggml-ci
* vulkan : fix lint (#0 )
---------
Co-authored-by: Acly <aclysia@gmail.com >
2025-05-02 20:54:30 +03:00
Georgi Gerganov and GitHub
a75cb30dc9
context : fix reorder logic ( #13267 )
...
ggml-ci
2025-05-02 20:54:13 +03:00
Georgi Gerganov and GitHub
c642bc014c
kv-cache : separate recurrent vs non-recurrent impl ( #12799 )
...
* kv-cache : serparate recurrent vs non-recurrent impl (wip)
ggml-ci
* kv-cache : init -> contructor + add llama_memory_params
ggml-ci
* kv-cache : fix callback reference
ggml-ci
* context : llama_kv_cache -> llama_memory_i
ggml-ci
* context : move memory creation logic to model
ggml-ci
* llama : remove reference of memory during encode
ggml-ci
* kv-cache : hide padding details in the implementation
ggml-ci
* kv-cache : add ubatch_next()
ggml-ci
* context : simplify sbatch logic
ggml-ci
* kv-cache : hide defrag logic in the implementation
ggml-ci
* context : hide kv cache details in implementation
ggml-ci
* build : fix
ggml-ci
* cont : another fix
ggml-ci
* kv-cache : simplify interface (wip)
ggml-ci
* kv-cache : use separate KV cell structs for unified/recurrent
ggml-ci
* kv-cache : clean-up
ggml-ci
* model : better llama_model::create_model() signature
ggml-ci
* kv-cache : fix recurrent seq_rm()
ggml-ci
* kv-cache : replace `struct callbacks` with `llama_model &`
ggml-ci
* kv-cache : replace `struct graph_params` with `llama_context &`
ggml-ci
* kv-cache : fix offload check
ggml-ci
* context : avoid passing unique_ptr
ggml-ci
* kv-cache : avoid using the backends from the llama_context
ref #13113
ggml-ci
* kv-cache : more consistent debug logs [no ci]
* kv-cache : do not pass the full llama_context for kv graphs
ggml-ci
* kv-cache : remove comment
* kv-cache : ggml_rope_ext_inplace -> ggml_rope_ext
ggml-ci
* kv-cache : fix recurrent multi-user case
ggml-ci
* memory : remove comments [no ci]
2025-05-02 17:48:36 +03:00
Georgi Gerganov and GitHub
fab647e884
server : add cache reuse card link to help ( #13230 )
...
* server : add cache reuse card link to help
* args : use short url
2025-05-02 09:48:31 +03:00
Georgi Gerganov
b1dd4d08e8
sync : ggml
...
ggml-ci
2025-05-01 20:15:34 +03:00
Georgi Gerganov
8d33d740c3
sync : ggml
2025-05-01 10:00:39 +03:00
Georgi Gerganov
9998540149
cuda : fix unused variable compile warning (whisper/0)
...
ggml-ci
2025-05-01 09:58:44 +03:00
Georgi Gerganov and GitHub
d9d398f84f
sampling : when top-k <= 0 -> noop ( #13173 )
...
ggml-ci
2025-04-29 20:22:57 +03:00
Georgi Gerganov and GitHub
d0a417f3c7
readme : update hot topics ( #13150 )
2025-04-28 12:10:18 +03:00
Georgi Gerganov and GitHub
43f2b07193
common : fix noreturn compile warning ( #13151 )
...
ggml-ci
2025-04-28 11:57:19 +03:00
Georgi Gerganov and GitHub
226251ed56
embeddings : fix batch sizes ( #13076 )
...
ggml-ci
2025-04-24 22:29:22 +03:00
Georgi Gerganov
87616f0680
ggml : fix trailing whitespaces ( #0 )
2025-04-24 17:32:47 +03:00
Georgi Gerganov
63b4911494
sync : ggml
...
ggml-ci
2025-04-24 17:32:47 +03:00
Georgi Gerganov and GitHub
13b4548877
cmake : do not include ./src as public for libllama ( #13062 )
...
* cmake : do not include ./src as public for libllama
ggml-ci
* cmake : rework tests
ggml-ci
* llguidance : remove unicode include
ggml-ci
* cmake : make c++17 private
ggml-ci
2025-04-24 16:00:10 +03:00
Georgi Gerganov and GitHub
572b3141d3
clang-tidy : disable warning about missing math parenthesis ( #13091 )
2025-04-24 15:44:05 +03:00
Georgi Gerganov and GitHub
7604a7d6b8
metal : fix floating-point range of attention scores in FA kernels ( #13090 )
...
ggml-ci
2025-04-24 10:38:30 +03:00
Georgi Gerganov and GitHub
ab47dec3d3
security : add note about RPC and server functionality ( #13061 )
...
* security : add note about RPC functionality
* security : add note about llama-server
2025-04-22 16:16:10 +03:00
Georgi Gerganov and GitHub
7b53389c24
metal : add memory pool for temp allocs ( #12850 )
...
* metal : add memory pool for temp allocs (wip) [no ci]
* cont : free buffers from the heap
* cont : resize heap [no ci]
* cont : refactor heap [no ci]
* cont : heap for each cmd buffer [no ci]
* cont : fix free
* wip
* cont : fix alignment [no ci]
* cont : not working .. [no ci]
* cont : heap allocation now works [no ci]
* cont : use MTLHeapTypePlacement
ggml-ci
* metal : use dynamic MTLHeap allocations
ggml-ci
* metal : add comments
* metal : disable softmax use of mem_pool
ggml-ci
* metal : final touches
2025-04-22 16:15:51 +03:00
Georgi Gerganov and GitHub
2f74c354c0
graph : make FA compatible with MLA + add initial Metal kernels ( #12953 )
...
* graph : make mla compatible with FA
* metal : add exp FA kernels for DeepSeek models
ggml-ci
* llama : minor naming updates
ggml-ci
* ggml : disable FA for DS head sizes
* tests : add FA tests for MLA shapes
ggml-ci
2025-04-17 18:16:36 +03:00
Georgi Gerganov and GitHub
f8f820cc4d
metal : add FA-vec kernels for head size 96 ( #12952 )
...
ggml-ci
2025-04-15 14:45:05 +03:00
Georgi Gerganov
526739b879
sync : ggml
...
ggml-ci
2025-04-14 09:26:15 +03:00
Georgi Gerganov and GitHub
c94085df28
server : add VSCode's Github Copilot Chat support ( #12896 )
...
* server : add VSCode's Github Copilot Chat support
* cont : update handler name
2025-04-11 23:37:41 +03:00
Georgi Gerganov
47ba87d0a4
sync : ggml
2025-04-11 00:17:47 +03:00
Georgi Gerganov
1d2b613445
tests : fix init order ( #0 )
...
ggml-ci
2025-04-11 00:17:47 +03:00
Georgi Gerganov
eb420e1148
sync : ggml
...
ggml-ci
2025-04-11 00:17:47 +03:00
Georgi Gerganov
e4bf72d631
scripts : fix sync-ggml-am.sh
2025-04-11 00:17:47 +03:00
Georgi Gerganov and GitHub
47277d6d1d
readme : add rpc backend ( #12842 )
2025-04-09 10:54:42 +03:00
Georgi Gerganov and GitHub
a19b5cef16
llama : fix FA when KV cache is not used (i.e. embeddings) ( #12825 )
...
* ggml : FA supports F32 V
* graph : cast KV to F16 when the KV cache is not used
ggml-ci
* server : add test that exercises embeddings with FA enabled
ggml-ci
2025-04-08 19:54:51 +03:00
Georgi Gerganov
1a1ab7e7a4
cuda : fix HIP and MUSA BF16 ( #0 )
...
ggml-ci
2025-04-07 18:44:17 +03:00
Georgi Gerganov
a4e46e28f9
sync : ggml
...
ggml-ci
2025-04-07 18:44:17 +03:00
Georgi Gerganov
ff067dbcb9
ggml : simplify Arm fp16 CPU logic (ggml/1177)
...
* ggml : simlpify Arm fp16 CPU logic
ggml-ci
* cont : bring back CUDA/MUSA checks
ggml-ci
2025-04-07 18:44:17 +03:00
Georgi Gerganov and GitHub
3e1d29348b
kv-cache : simplify + fix warning for recurrent models ( #12756 )
...
ggml-ci
2025-04-04 21:48:10 +03:00
833e2b7409
model : print tensor size during load ( #12711 )
...
* model : print tensor size during load
* cont : fix units MB -> MiB
Co-authored-by: Diego Devesa <slarengh@gmail.com >
---------
Co-authored-by: Diego Devesa <slarengh@gmail.com >
2025-04-02 16:38:54 +03:00
Georgi Gerganov and GitHub
a10b36c91a
llama : refactor kv cache guard ( #12695 )
...
* llama : refactor kv cache guard
ggml-ci
* cont : fix comment [no ci]
* llama : fix kv_cache restore logic
ggml-ci
* context : simplify kv cache updates
ggml-ci
* cont : better name [no ci]
* llama : fix llama_decode return code when could not find KV slot
ggml-ci
* context : change log err -> warn [no ci]
* kv-cache : add comment + warning
2025-04-02 14:32:59 +03:00
Georgi Gerganov and GitHub
3fd072a540
metal : use F32 prec in FA kernels ( #12688 )
...
* metal : use F32 prec in FA kernels
ggml-ci
* cont : fix FA vec kernel
ggml-ci
2025-04-01 14:57:19 +03:00
Georgi Gerganov
1790e73157
cmake : fix whitespace ( #0 )
2025-03-31 15:07:32 +03:00
Georgi Gerganov
0114a32da0
sync : ggml
...
ggml-ci
2025-03-31 15:07:32 +03:00
Georgi Gerganov and GitHub
4663bd353c
metal : use constexpr in FA kernels + fix typedef ( #12659 )
...
* metal : use constexpr in FA kernels
ggml-ci
* cont
ggml-ci
* cont : fix typedef
ggml-ci
2025-03-30 22:04:04 +03:00
Georgi Gerganov
d3f1f0acfb
sync : ggml
...
ggml-ci
2025-03-30 08:33:31 +03:00
Georgi Gerganov and GitHub
b4ae50810e
metal : improve FA + improve MoE ( #12612 )
...
* ggml : FA with different K, V head sizes (CPU)
ggml-ci
* metal : add FA with HS=192
* metal : extend FA to support different K and V head sizes
ggml-ci
* metal : add FA vector kernels for heads K 192 and V 128
ggml-ci
* ggml : restrict op on other backends to equal head sizes
ggml-ci
* metal : optimize FA-vec kernel
ggml-ci
* metal : FA remove mq registers
* metal : improve MoE mul_mat_id condition
ggml-ci
* metal : fix comments + remove unnecessary addition
ggml-ci
* metal : avoid too much shared memory usage with mul_mat_id
ggml-ci
2025-03-28 20:21:59 +02:00
Georgi Gerganov and GitHub
2969019837
media : add SVG logo [no ci] ( #12616 )
2025-03-27 23:09:05 +02:00
Georgi Gerganov
029c693fdc
sync : ggml
...
ggml-ci
2025-03-27 10:09:29 +02:00
Georgi Gerganov
771d84371c
scripts : update sync + fix cmake merge
...
ggml-ci
2025-03-27 10:09:29 +02:00
Georgi Gerganov
df0665a483
sync : ggml
...
ggml-ci
2025-03-27 09:04:38 +02:00
Georgi Gerganov
0306aad1ca
cmake : sync/merge PowerPC build commands ( #0 )
2025-03-27 09:04:38 +02:00
Georgi Gerganov and GitHub
f28bc4c286
llama : make loras compatible with repacking ( #12593 )
...
* llama : make loras compatible with repacking
ggml-ci
* cont : simplify
ggml-ci
* cont : add TODO [no ci]
2025-03-27 08:24:10 +02:00
Georgi Gerganov and GitHub
b3298fa47a
metal : refactor mat-vec code ( #12569 )
...
* metal : refactor mat-vec code
ggml-ci
* metal : rename all_sum -> sum_all
ggml-ci
* metal : fix comments [no ci]
* metal : fix nr constant [no ci]
* metal : mv q6_K support nr0 > 1
ggml-ci
* metal : reduce register pressure
ggml-ci
* metal : fix typo [no ci]
* metal : reduce register pressure
ggml-ci
2025-03-26 21:38:38 +02:00
df4d20cd53
convert : fix squeeze for ssm_conv tensors ( #12573 )
...
* convert : fix squeeze for ssm_conv tensors
* convert : match ssm_conv tensors by type
---------
Co-authored-by: Francis Couture-Harpin <git@compilade.net >
2025-03-26 08:21:05 -04:00
Georgi Gerganov and GitHub
5ed38b6852
ggml : fix MUL_MAT_ID repack with Q8_K ( #12544 )
...
* ggml : fix MUL_MAT_ID repack with Q8_K
ggml-ci
* ggml : improve repack templates
ggml-ci
2025-03-26 13:02:00 +02:00
Georgi Gerganov and GitHub
2d77d88e70
context : fix worst-case reserve outputs ( #12545 )
...
ggml-ci
2025-03-25 09:19:23 +02:00
Georgi Gerganov and GitHub
ba932dfb50
ggml : fix quantized cpy op ( #12310 )
...
* ggml : fix quantized cpy op
ggml-ci
* tests : add cpy tests for all types
ggml-ci
* tests : add BF16 copy tests
ggml-ci
* tests : fix loop for same-type copy
ggml-ci
* tests : add option to permute the dst tensor
ggml-ci
2025-03-22 16:23:26 +02:00
Georgi Gerganov and GitHub
af04481e6b
model : do not repack if a GPU device is present ( #12498 )
...
ggml-ci
2025-03-21 16:14:29 +02:00
Georgi Gerganov and GitHub
75422e8bc4
graph : normalize Q, K, V shapes + sync cross attention ( #12449 )
...
* graph : normalize Q, K, V shapes and add comments
ggml-ci
* context : synchronize before getting cross attention data
* model : fix command-r attention norm check
2025-03-18 21:35:19 +02:00
Georgi Gerganov and GitHub
c6af2161b2
speculative : fix seg fault in certain cases ( #12454 )
2025-03-18 19:35:11 +02:00
Georgi Gerganov and GitHub
8551c44d84
context : always use non-causal attention for encoder graphs ( #12447 )
...
* context : always use non-causal attention for encoder graphs
ggml-ci
* context : move the change to llama_context::encode()
ggml-ci
2025-03-18 13:05:49 +02:00
Georgi Gerganov and GitHub
810e0af3f5
server : fix warmup draft cache type ( #12446 )
...
ggml-ci
2025-03-18 12:05:42 +02:00
Georgi Gerganov and GitHub
dc079cfdff
context : fix init of n_outputs ( #12397 )
...
ggml-ci
2025-03-16 19:29:36 +02:00
Georgi Gerganov and GitHub
c522ce4143
graph : simplify attn input build for unified KV cache ( #12381 )
...
ggml-ci
2025-03-14 10:47:44 +02:00
Georgi Gerganov and GitHub
081bee8c64
hparams : add SWA rope parameters ( #12374 )
...
ggml-ci
2025-03-14 09:03:24 +02:00
Georgi Gerganov and GitHub
84d5475541
llama : fix Gemma3 SWA KV cache shift ( #12373 )
...
* llama : fix Gemma3 SWA KV cache shift
ggml-ci
* hparams : add comment [no ci]
2025-03-13 19:08:07 +02:00
Georgi Gerganov and GitHub
e0dbec0bc6
llama : refactor llama_context, llama_kv_cache, llm_build_context ( #12181 )
...
* llama : refactor llama_context, llama_kv_cache, llm_build_context
ggml-ci
* graph : don't mutate the KV cache during defrag
ggml-ci
* context : reduce virtuals + remove test function
ggml-ci
* context : move interface implementation to source file + factory
ggml-ci
* graph : move KV cache build functions to llama_context impl
ggml-ci
* graph : remove model reference from build_pooling
ggml-ci
* graph : remove llama_model reference
ggml-ci
* kv_cache : provide rope factors
ggml-ci
* graph : rework inputs to use only unique_ptr, remove attn input abstraction
ggml-ci
* context : remove llama_context_i abstraction
ggml-ci
* context : clean-up
ggml-ci
* graph : clean-up
ggml-ci
* llama : remove redundant keywords (struct, enum)
ggml-ci
* model : adapt gemma3
ggml-ci
* graph : restore same attention ops as on master
ggml-ci
* llama : remove TODO + fix indent
ggml-ci
2025-03-13 12:35:44 +02:00
Georgi Gerganov and GitHub
e128a1bf5b
tests : fix test-quantize-fns to init the CPU backend ( #12306 )
...
ggml-ci
2025-03-10 14:07:15 +02:00
Georgi Gerganov and GitHub
1e2f78a004
server : add speculative decoding presets for FIM ( #12287 )
2025-03-09 19:08:20 +02:00
Georgi Gerganov and GitHub
0fd7ca7a21
authors : update ( #12271 )
2025-03-08 18:26:00 +02:00
Georgi Gerganov and GitHub
7ab364390f
server : infill gen ends on new line ( #12254 )
2025-03-07 20:54:30 +02:00
Georgi Gerganov
102ac1891d
sync : ggml
...
ggml-ci
2025-03-07 14:49:44 +02:00
Georgi Gerganov and GitHub
ea002810a2
ci : fix save-load test invocations ( #12245 )
2025-03-07 12:19:31 +02:00
Georgi Gerganov and GitHub
20a9b8f5e1
readme : fix roadmap link ( #12185 )
2025-03-04 18:42:44 +02:00
Georgi Gerganov
dfd6b2c0be
sync : ggml
...
ggml-ci
2025-03-03 18:18:11 +02:00
Georgi Gerganov
3d1cf3cf33
sync : ggml
...
ggml-ci
2025-03-03 18:18:11 +02:00
Georgi Gerganov
8371d44595
sync : ggml
...
ggml-ci
2025-03-03 18:18:11 +02:00
Georgi Gerganov
aede2074f6
scripts : sync-ggml-am.sh fix
2025-03-03 18:18:11 +02:00
Georgi Gerganov and GitHub
f3e64859ed
ci : fix arm upload artifacts ( #12024 )
...
* ci : fix arm upload artifacts
* cont : fix archive name to use matrix
2025-02-22 15:03:00 +02:00
Georgi Gerganov and GitHub
cf756d6e0a
server : disable Nagle's algorithm ( #12020 )
2025-02-22 11:46:31 +01:00
Georgi Gerganov and GitHub
51f311e057
llama : skip loading unused tensors ( #12004 )
...
* llama : assign unknown/unused tensors to host buffer type
ggml-ci
* llama : skip unused tensors
ggml-ci
2025-02-21 18:33:18 +02:00
Georgi Gerganov and GitHub
abd4d0bc4f
speculative : update default params ( #11954 )
...
* speculative : update default params
* speculative : do not discard the last drafted token
2025-02-19 13:29:42 +02:00
Georgi Gerganov and GitHub
c2cd24fbfd
readme : add notice about new package registry ( #11890 )
...
* readme : add notice about new package registry
* cont : fix whitespace
2025-02-15 20:29:56 +02:00
Georgi Gerganov and GitHub
68ff663a04
repo : update links to new url ( #11886 )
...
* repo : update links to new url
ggml-ci
* cont : more urls
ggml-ci
2025-02-15 16:40:57 +02:00
Georgi Gerganov and GitHub
dbc2ec59b5
docker : drop to CUDA 12.4 ( #11869 )
...
* docker : drop to CUDA 12.4
* docker : update readme [no ci]
2025-02-14 14:48:40 +02:00
Georgi Gerganov and GitHub
04045bb842
readme : minor
2025-02-14 00:16:56 +02:00
Georgi Gerganov
0fb77f821f
sync : ggml
2025-02-12 21:46:02 +02:00
Georgi Gerganov and GitHub
aaa5505307
server : minor log updates ( #11760 )
...
ggml-ci
2025-02-08 18:08:43 +02:00
Georgi Gerganov
bdcf8b6a56
cont : fix mmap flag print ( #11699 )
2025-02-08 16:49:38 +02:00
Georgi Gerganov and GitHub
ed926d8833
llama : fix defrag logic ( #11707 )
...
* llama : fix defrag logic
ggml-ci
* cont : better logic
ggml-ci
* cont : clamp fragmentation to 0.0
ggml-ci
2025-02-07 16:05:34 +02:00
Georgi Gerganov
8a59053f63
sync : ggml
2025-02-06 21:23:03 +02:00
Georgi Gerganov and GitHub
9dd7a0390f
llama : add log about loading model tensors ( #11699 )
2025-02-06 13:41:37 +02:00
Georgi Gerganov and GitHub
d774ab3acc
metal : adjust support conditions for norm operators ( #11671 )
...
cont #11659
ggml-ci
2025-02-05 10:57:42 +02:00
Georgi Gerganov
387a1598ca
authors : update
2025-02-04 13:04:10 +02:00
Georgi Gerganov
7c9e0ca520
sync : ggml
2025-02-04 12:59:21 +02:00
