nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 29 Packages Projects Releases Wiki Activity
8,401 Commits 1 Branch 0 Tags
a69d54f990d0cd88786d5943632d6426dc9660b7
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Georgi GerganovandGitHub 8cc2d81264 server : fix ctx checkpoint invalidation (#20671) 2026-03-17 15:21:14 +02:00
Georgi GerganovandGitHub 45172df4d6 ci : disable AMX jobs (#20654) 
[no ci]
 2026-03-16 22:38:59 +02:00
Georgi GerganovandGitHub 9b342d0a9f benches : add Nemotron 3 Nano on DGX Spark (#20652) 
[no ci]
 2026-03-16 21:50:43 +02:00
Georgi Gerganov f47a246a08 sync : ggml 2026-03-16 17:22:06 +02:00
Georgi Gerganov c0ccbd1f86 ggml : try fix arm build (whisper/0) 2026-03-16 17:22:06 +02:00
Georgi GerganovandGitHub 88915cb55c server : fix wait in test_cancel_requests() test (#20601) 
* server : fix wait in test_cancel_requests() test

* codeowners : add team for server tests
 2026-03-15 20:54:37 +02:00
Georgi GerganovGitHubSigbjørn Skjæret
 		9cd4ebcfb1 ci : split build.yml + server.yml (#20546) 
* ci : split build.yml

* cont : split server.yml

* cont : reduce paths

* cont : split build-android.yml + update paths

* ci : make msys workflows manual (#20588)

* ci : make cross-build workflows manual (#20585)

* cont : fix release paths

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-03-15 15:11:17 +02:00
Georgi GerganovandGitHub b30a5fdf37 metal : add FA specialization for HSK = 320, HSV = 256 (#20549) 2026-03-14 23:15:47 +02:00
Georgi GerganovandGitHub b4768955c4 ci : move self-hosted workflows to separate files (#20540) 2026-03-14 23:15:35 +02:00
Georgi GerganovandGitHub 9f774e45ee ci : reduce webgpu tests timeout to 900s (#20538) 
[no ci]
 2026-03-14 17:08:26 +02:00
Georgi GerganovGitHubClaude Opus 4.6Paul FlynnOliver Simons
 		e30f1fdf74 graph : remove redundant GDN state transposes (#20443) 
* ggml : transpose fused GDN state access for coalesced memory reads (#20436)

The fused Gated Delta Net kernel accessed the [S_v, S_v] state matrix
column-wise on row-major storage, causing strided reads (stride S_v =
128 floats = 512 bytes) that waste GPU cache bandwidth. This produced a
39% regression on Qwen3.5-9B (Metal, M4 Max) compared to the unfused
path.

Transpose the state indexing so threads read contiguously:
- Metal: s_ptr[is*S_v] -> s_ptr[is] (stride 1 vs S_v)
- CUDA:  curr_state[i*S_v+col] -> curr_state[col*S_v+i] (coalesced)
- CPU:   restructured loops for row-wise transposed access

Also add --fused-gdn [on|off|auto] CLI flag (mirrors --flash-attn) so
users can control fused GDN independently of auto-detection.

All GATED_DELTA_NET backend-ops tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* ggml : use SIMD dot products in CPU GDN kernel, couple AR/chunked fused flags

- Replace scalar inner loops with ggml_vec_dot_f32 for SIMD-optimized
  dot products in the CPU fused GDN kernel (delta and attention output)
- Couple fused_gdn_ar and fused_gdn_ch flags in auto-detection: if one
  path lacks device support, disable both to prevent state layout mismatch
  between transposed (fused) and non-transposed (unfused) formats

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* llama : rever fgdn argument changes

* graph : remove GDN state transposes

* vulkan : adapt

* cuda : remove obsolete smem code

---------

Co-authored-by: Paul Flynn <paul@arkavo.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Oliver Simons <osimons@nvidia.com>
 2026-03-13 22:12:54 +02:00
Georgi GerganovandGitHub 73c9eb8ced metal : fix l2 norm scale (#20493) 2026-03-13 11:43:20 +02:00
Georgi GerganovandGitHub 57819b8d4b llama : disable graph reuse with pipeline parallelism (#20463) 2026-03-12 21:04:13 +02:00
Georgi GerganovandGitHub e4cff0956b metal : avoid divisions in bin kernel (#20426) 
* metal : avoid modulus in bin kernel when not broadcasting

* metal : fix capture_started flag
 2026-03-12 09:42:40 +02:00
Georgi GerganovGitHubuvosAman GuptaPaul FlynnClaude Opus 4.6Oliver Simons
 		d28961d81e llama : enable chunked fused GDN path (#20340) 
* llama : enable chunked fused GDN path

* models : avoid Q and K repeats when using fused GDA

* cont : fix comment

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

* cont : fix the fix

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

* cont : fix

* metal : add GDN kernel (#20361)

* metal : add Metal backend for GGML_OP_GATED_DELTA_NET

Add a fused Metal kernel for the gated delta net recurrence op
(#19504), enabling GPU-accelerated inference for DeltaNet-based
models (Qwen3.5, etc.) on Apple Silicon.

Supports both GDA (scalar gate) and KDA (per-row gate) modes
with head_size 64 and 128. Unsupported configurations (head_size
32, non-contiguous tensors) gracefully fall back to CPU.

Performance: Qwen3.5-0.8B Q4_K_M on M4 Max
  tg128: 170 -> 213 t/s (+25%)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* metal : validate contiguity of all input tensors in supports_op

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* metal : add algorithm equivalence comment for GDA decay path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* cont : unslop + optimize

* cont : clean-up

---------

Co-authored-by: Paul Flynn <paul@arkavo.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* CUDA: AR gated delta net improvements (#20391)

* Add FastDiv to gated_delta_net_cuda

* Shard columns across warps

This reduces register pressure (avoids spill for S_v = 128) and gives
the warp-scheduler more CTAs to schedule (thus hiding data-access
latencies).

* Remove unneded include in gated_delta_net.cu

* Improve comments

* Apply code-formating

* Make sharding HIP-compatible

1. Use ggml_cuda_get_physical_warp_size() to determine warp size flexibly
2. Add test with partial warp to test sum reduction on CUDA

* Remove fastdiv_s64, as we can treat neqk1 and rq3 as uint32_t

* Rename variables

* Enable GDN also for prefill, move TODO for chunked_GDN

* Actually remove the TODO from 206890897546bd16602c3b79394fd5ea09ef199f

* Get warp size at runtime

warp_size is not known at compile time in hip host code.

* Don't expose ggml_cuda_get_physical_warp_size on host

---------

Co-authored-by: uvos <devnull@uvos.xyz>

* llama : refactor llm_build_delta_net_base API

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Paul Flynn <paul@arkavo.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Oliver Simons <osimons@nvidia.com>
Co-authored-by: uvos <devnull@uvos.xyz>
 2026-03-11 22:46:40 +02:00
Georgi GerganovandGitHub 3ca19b0e9f benches : add nemotron super (#20420) 2026-03-11 21:39:40 +02:00
Georgi GerganovandGitHub 76ea1c1c46 metal : fix capture_compute counter logic (#20410) 2026-03-11 18:38:22 +02:00
Georgi GerganovandGitHub b541241104 metal : fix q5_k mul_mv register spill (#20399) 2026-03-11 16:25:27 +02:00
Georgi GerganovandGitHub c363256839 metal : add env var to trigger graph capture (#20398) 2026-03-11 16:25:10 +02:00
Georgi GerganovandGitHub 90b2731894 ggml : bump RPC version (#20330) 2026-03-10 21:36:57 +02:00
Georgi GerganovGitHubSigbjørn Skjæret
 		1274fbee9e models : fix assert in mamba2 (cont) (#20335) 
* models : fix assert in mamba2 (cont)

* cont : add n_group mod

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-03-10 15:00:08 +02:00
Georgi GerganovandGitHub a7b3dee7a5 server : make 2 checkpoints near the end of the prompt (#20288) 
* server : make 2 checkpoints near the end of the prompt

* cont : adjust checkpoints
 2026-03-10 14:28:23 +02:00
Georgi GerganovandGitHub 96cfc4992c server : fix checkpoints n_tokens calculation (#20287) 2026-03-09 16:47:06 +02:00
Georgi GerganovandGitHub ed0007aa32 metal : add upscale (#20284) 2026-03-09 16:45:11 +02:00
Georgi GerganovandGitHub 344ee2a38a server : warn swa-full is not supported for non-SWA models (#20291) 2026-03-09 16:44:25 +02:00
Georgi GerganovandGitHub d6e1556499 server : fix off-by-1 in server_tokens::size_up_to_pos() (#20279) 
* server : fix off-by-1 in server_tokens::size_up_to_pos()

* cont : fix typo [no ci]
 2026-03-09 16:43:38 +02:00
Georgi GerganovandGitHub 43e1cbd6c1 models : fix assert in mamba2 graph (#20270) 2026-03-09 13:15:15 +02:00
Georgi GerganovandGitHub 107d599952 server : add kill switch when server is stuck (#20277) 2026-03-09 10:33:12 +02:00
Georgi GerganovandGitHub d417bc43dd server : do not create checkpoints right after mtmd chunks (#20232) 2026-03-08 22:16:46 +02:00
Georgi GerganovandGitHub 17a4258946 kv-cache : fix M-RoPE checkpoints (#20132) 2026-03-06 08:46:51 +02:00
Georgi GerganovandGitHub 37964f44f9 mtmd : fix padding of n_tokens (#19930) 2026-02-26 18:39:49 +02:00
Georgi GerganovandGitHub 01cd448b8c server : fix ctx checkpoint restore logic (#19924) 2026-02-26 18:20:16 +02:00
Georgi GerganovandGitHub 99bd67c9b2 kv-cache : fix can_shift() check to take into account M-RoPE (#19928) 2026-02-26 18:08:54 +02:00
Georgi GerganovandGitHub 1ca3d1de15 gguf : avoid too many file size calls (#19919) 2026-02-26 12:46:32 +02:00
Georgi GerganovandGitHub f20469d919 server : enable multi-modal prompt caching (#19877) 2026-02-25 15:15:42 +02:00
Georgi GerganovGitHubtimkhronos
 		d7d826b3c1 server : support multi-modal context checkpoints (#19849) 
* Modify llama-memory-hybrid-iswa.cpp

* Modify llama-memory-recurrent.cpp

* Modify server-common.cpp

* Modify server-common.h

* Modify server-context.cpp

* Modify server-task.h

* Added comment to llama-memory-hybrid-iswa.cpp

* Remove comment from server-context.cpp

* Stylistic fix server-context.cpp

* Fix an issue when seqrm isn't called in server-context.cpp

* cont : alternative impl

* cont : cleanup

* cont : n_tokens -> int64_t

---------

Co-authored-by: timkhronos <timkhronos@gmail.com>
 2026-02-25 15:14:27 +02:00
Georgi GerganovandGitHub 244641955f models : fix graph splits (#19866) 2026-02-25 00:01:13 +02:00
Georgi GerganovGitHubSigbjørn Skjæret
 		418dea39ce ggml/gguf : prevent integer overflows (#19856) 
* gguf : prevent integer overflow for ggml_context mem size

* ggml : fix int overflows in ggml_new_object()

* gguf : prevent string exhaustion

* gguf : prevent array elements exhaustion

* ggml : fix negative tensor type oob

* py : assert that alignment is non-zero power of 2

* ggml : check int overflow in ggml_new_tensor_impl and ggml_new_object

* gguf-py : error on duplicate keys when reading

* py : restore tensor_fields

* enforce proper alignment in add_custom_alignment

* gguf : better name

* gguf : fix ctx size for no_alloc == true

* gguf : minor print fix

* ggml : print values when overflow

* ggml : remove deprecated ggml_type_sizef()

* ggml : relax ggml_type asserts to debug-only

* gguf : add mem_size overflow test

* gguf : add file size check for arrays

* ggml : relax asseerts for ggml_get_type_traits()

* flake8 fix

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-02-24 20:17:11 +02:00
Georgi GerganovandGitHub da348c9dfb models : fix qwen3.5 beta/gate shapes (#19730) 
* models : fix qwen3.5 beta/gate shapes

* cont : avoid extra reshapes
 2026-02-19 15:19:53 +02:00
Georgi GerganovandGitHub 27326bfce1 models : dedup qwen35 graphs (#19660) 
* models : dedup qwen35 graphs

* cont : add missing sigmoid
 2026-02-19 08:17:49 +02:00
Georgi GerganovGitHubOliver Simons
 		ad8207af77 cuda : enable CUDA graphs for MMID 1 <= BS <= 4 (#19645) 
* cuda : enable CUDA graphs for MMID BS <= 4

* cont : add stream capture check

Co-authored-by: Oliver Simons <osimons@nvidia.com>

* cont : add MMVQ_MMID_MAX_BATCH_SIZE

---------

Co-authored-by: Oliver Simons <osimons@nvidia.com>
 2026-02-17 12:31:49 +02:00
Georgi GerganovandGitHub cc45f2ada6 models : deduplicate delta-net graphs for Qwen family (#19597) 
* models : add llm_build_delta_net_base

* cont : keep qwen35 and qwen35moe graphs intact

* cont : add comments
 2026-02-16 14:35:04 +02:00
Georgi GerganovandGitHub d5dfc33027 graph : fix KQ mask, lora, cvec reuse checks (#19644) 
* graph : fix KQ mask reuse condition

* cont : dedup KQ mask build and can_reuse

* cont : fix build

* graph : fix adapter check for reuse
 2026-02-16 09:21:11 +02:00
Georgi Gerganov ff4affb4c1 sync : ggml 2026-02-15 22:24:29 +02:00
Georgi Gerganov 55d58599c8 ggml : bump version to 0.9.7 (ggml/1425) 2026-02-15 22:24:29 +02:00
Georgi Gerganov 1a8c700bfd ggml : bump version to 0.9.6 (ggml/1423) 2026-02-15 22:24:29 +02:00
Georgi GerganovandGitHub 341bc7d23c context : fix output reorder with backend sampling (#19638) 2026-02-15 14:57:40 +02:00
Georgi GerganovandGitHub 08e6d914b8 ggml : avoid UB in gemm ukernel (#19642) 2026-02-15 14:56:35 +02:00
Georgi GerganovandGitHub 1725e316c1 models : optimize qwen3next graph (#19375) 
* models : optimizing qwen3next graph

* cont

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* cont : remove redundant q, g chunking

* minor

* minor

* avoid passing masks around

* avoid concats during chunking

* naming + shapes

* update names and use prefix to disable CUDA graphs
 2026-02-14 12:57:36 +02:00
Georgi GerganovandGitHub 6e473fb384 metal : fix ACC op (#19427) 2026-02-14 09:54:03 +02:00
Georgi GerganovandGitHub bb96bfd361 memory : fix kv cache size for hybrid models (#19559) 2026-02-13 07:36:24 +02:00
Georgi GerganovandGitHub 0644baefde metal : improve concurrency (#19555) 2026-02-13 07:35:57 +02:00
Georgi GerganovandGitHub 490eb96b88 metal : support GGML_OP_SET (#19548) 2026-02-13 07:34:52 +02:00
Georgi GerganovandGitHub 338085c69e args : add -kvu to llama-parallel (#19577) 2026-02-12 21:52:41 +02:00
Georgi GerganovandGitHub 3b3a948134 metal : update sum_rows kernel to support float4 (#19524) 2026-02-12 11:35:28 +02:00
Georgi GerganovandGitHub 914dde72ba ggml : unary ops support non-cont src0 + metal F16 unary ops (#19511) 
* ggml : unary ops support non-cont src0

* metal : support F16 unary ops + fix ELU
 2026-02-11 18:58:43 +02:00
Georgi GerganovandGitHub 9ab072ebbe metal : extend l2_norm support for non-cont src0 (#19502) 2026-02-11 14:53:19 +02:00
Georgi GerganovandGitHub 6d95707827 model : fix wavtokenizer embedding notions (#19479) 2026-02-11 07:52:20 +02:00
Georgi GerganovandGitHub 89181c0b6d ggml : extend bin bcast for permuted src1 (#19484) 
* tests : extend bin bcast for permuted src1

* cont : extend bin support

* cont : s0 is always 1

* tests : simplify
 2026-02-11 07:52:00 +02:00
Georgi GerganovandGitHub ceaa89b786 metal : consolidate unary ops (#19490) 2026-02-11 07:51:12 +02:00
Georgi GerganovandGitHub a0d585537c cuda : extend GGML_OP_PAD to work with non-cont src0 (#19429) 
* cuda : extend GGML_OP_PAD to work with non-cont src0

* tests : add permuted pad
 2026-02-10 08:07:16 +02:00
Georgi GerganovGitHubSigbjørn Skjæret
 		81ddc60cb3 ci : add metal server workflows (#19293) 
* ci : add metal server workflows

* cont : try fix python init

* cont : move to a separate workflow that runs only on master

* cont : fix num jobs

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-02-09 15:09:30 +02:00
Georgi GerganovandGitHub 972f323e73 revert : "[Model] Qwen3.5 dense and MoE support (no vision) (#19435)" (#19453) 
This reverts commit 39bf692af1.
 2026-02-09 14:57:51 +02:00
Georgi GerganovandGitHub eb449cdfa4 server : improve context checkpoint logic (#19408) 2026-02-08 09:40:04 +02:00
Georgi GerganovandGitHub 96441c955e ci : use -j param correctly when building with sanitizers (#19411) 
* ci : use less jobs when building with sanitizers

* cont : fix nproc

* cont : fix the fix

* cont : simplify
 2026-02-07 23:50:47 +01:00
Georgi GerganovandGitHub 8872ad2125 metal : consolidate bin kernels (#19390) 
* metal : refactor bin kernels

* cont

* cont : fix cv
 2026-02-07 10:35:56 +02:00
Georgi GerganovandGitHub 34ba7b5a2f metal : fix event synchronization in cpy_tensor_async (#19402) 2026-02-07 07:37:15 +02:00
Georgi GerganovandGitHub dfde5993ea common : add common_speculative_is_compat() (#19270) 
* llama : add llama_memory_can_rm_suffix()

* Revert "llama : add llama_memory_can_rm_suffix()"

This reverts commit d30e59b62a15ef4266a6503e3f4eba770aec001b.

* spec : check if the target context is compatible for spec decoding
 2026-02-06 16:47:22 +02:00
Georgi GerganovandGitHub 7fcf1ef45d metal : skip loading all-zero mask (#19337) 
* metal : skip loading all-zero mask

* cont : minor
 2026-02-06 09:25:11 +02:00
Georgi GerganovandGitHub 3e21647666 cuda : cuda graphs now compare all node params (#19383) 2026-02-06 07:55:06 +02:00
Georgi GerganovandGitHub 22cae83218 metal : adaptive CPU/GPU interleave based on number of nodes (#19369) 2026-02-05 19:07:22 +02:00
Georgi GerganovandGitHub 3795cc1e89 benches : update models + numbers (#19359) 
* bench : update script

* benches : update numbers
 2026-02-05 14:34:07 +02:00
Georgi GerganovandGitHub 7a4f97d196 metal : add diag (#19330) 2026-02-05 10:08:45 +02:00
Georgi GerganovandGitHub 423bee462b ci : fix sanitize workflow to enable ggml sanitizers too (#19323) 2026-02-04 15:12:03 +02:00
Georgi GerganovGitHubJeff Bolz
 		eaba92c3dc tests : add non-cont, inplace rope tests (#19296) 
* tests : add non-cont, inplace rope tests

* cont : exercise dim 3

Co-authored-by: Jeff Bolz <jbolz@nvidia.com>

* cont : more dim3 exercises

---------

Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
 2026-02-04 12:45:21 +02:00
Georgi GerganovandGitHub d838c22bb3 spec : fix the check-rate logic of ngram-simple (#19261) 
* spec : fix the check-rate logic of ngram-simple

* cont : refactor + fix checks
 2026-02-04 10:39:53 +02:00
Georgi GerganovandGitHub 44008ce8f9 metal : add solve_tri (#19302) 2026-02-03 23:43:14 +02:00
Georgi GerganovandGitHub 6a9bf2f788 ci : add sanitizer runs for server (#19291) 2026-02-03 22:41:20 +02:00
Georgi GerganovandGitHub faa1bc26ee sampling : delegate input allocation to the scheduler (#19266) 
* sampling : delegate input allocation to the scheduler

* graph : compute backend samplers only if needed
 2026-02-03 22:16:16 +02:00
Georgi GerganovandGitHub c55bce4159 metal : minor cleanup (#19251) 2026-02-03 13:43:29 +02:00
Georgi GerganovandGitHub aeb827a3cc spec : simplify time measurement using common_time_meas (#19262) 2026-02-03 08:20:15 +02:00
Georgi GerganovandGitHub 6fdddb4987 metal : support virtual devices (#18919) 
* metal : support virtual devices

* cont : manage buffer type context memory

* metal : add events

* cont : implement cpy_tensor_async
 2026-02-02 14:29:44 +02:00
Georgi GerganovandGitHub 1239267cc4 authors : update (#19263) 
[no ci]
 2026-02-02 08:51:25 +02:00
Georgi GerganovandGitHub 4927795810 ngram-mod : fix build [no ci] (#19216) 2026-01-30 21:27:27 +02:00
Georgi Gerganov d9a2a4bcaa sync : ggml 2026-01-30 20:09:21 +02:00
Georgi Gerganov dfd6106c84 cuda : fix compile warnings (whisper/0) 2026-01-30 20:09:21 +02:00
Georgi GerganovandGitHub bbada8bfb9 server : wrap around the "id_slot" parameter (#19207) 
* server : wrap around the "id_slot" parameter

* cont : minor
 2026-01-30 19:46:10 +02:00
Georgi GerganovandGitHub dabaa2e77a spec : add ngram-mod (#19164) 
* spec : add ngram-mod

* cont : simplify + keep track of occupancy

* cont : cleanup

* cont : move initialization to common/speculative

* cont : cleanup

* cont : cleanup

* cont : fix
 2026-01-30 18:21:48 +02:00
Georgi GerganovandGitHub c3b87cebff tests : add GQA=20 FA test (#19095) 2026-01-30 13:52:57 +02:00
Georgi GerganovandGitHub 4fdbc1e4db cuda : fix nkvo, offload and cuda graph node properties matching (#19165) 
* cuda : fix nkvo

* cont : more robust cuda graph node property matching

* cont : restore pre-leafs implementation

* cont : comments + static_assert
 2026-01-29 18:45:30 +02:00
Georgi GerganovandGitHub eed25bc6b0 arg : add -kvu to llama-batched-bench (#19172) 2026-01-29 08:50:47 +02:00
Georgi GerganovandGitHub 631cbfcc7a cuda : fix "V is K view" check for non-unified KV cache (#19145) 2026-01-28 09:15:27 +02:00
Georgi GerganovandGitHub 2eee6c866c CUDA: tune GLM 4.7 Flash FA kernel selection logic (DGX Spark) (#19142) 2026-01-28 09:15:11 +02:00
Georgi GerganovandGitHub b931f81b5a server : adjust spec tests to generate up to 16 tokens (#19093) 2026-01-28 09:11:40 +02:00
Georgi GerganovandGitHub c5c64f72ac llama : disable Direct IO by default (#19109) 
* llama : disable Direct IO by default

* cont : override mmap if supported
 2026-01-28 09:11:13 +02:00
Georgi GerganovandGitHub 8f80d1b254 graph : fix nkvo offload with FA (#19105) 2026-01-26 20:18:34 +02:00
Georgi GerganovandGitHub 56f3ebf38e model : add correct type for GLM 4.7 Flash (#19106) 2026-01-26 11:24:30 +02:00
Georgi GerganovandGitHub d9c6ce46f7 kv-cache : support V-less cache (#19067) 
* kv-cache : support V-less cache

* cuda : better check for V_is_K_view

* cuda : improve V_is_K_view check

* graph : add comments

* hparams : refactor
 2026-01-25 15:48:56 +02:00
Georgi GerganovandGitHub 080b161995 completion : fix prompt cache for recurrent models (#19045) 2026-01-25 09:12:50 +02:00
Georgi GerganovandGitHub 557515be1e graph : utilize ggml_build_forward_select() to avoid reallocations (#18898) 
* graph : avoid branches between embedding and token inputs

* models : make deepstack graphs (e.g. Qwen3 VL) have constant topology

* ci : enable -DGGML_SCHED_NO_REALLOC=ON for server CI

* cont : pad token embeddings to n_embd_inp
 2026-01-23 18:22:34 +02:00