* vulkan : check src0 type in GGML_OP_SET_ROWS to avoid failures due to unimplemented f16 support
* chore : get rid of else
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* opencl: vec flash-attention decode kernels for f16/q8_0/q4_0 KV
* opencl: improve non FA KQ mv kernels
* opencl: tweaks for multiquery FA
* opencl: some tweaks for FA q1 kernels
* opencl: FA with DK=DV=512 for gemma-4
* opencl: various fixes
* opencl: cleanup
* opencl: fix FA decode crash for DK=512 (gemma-4)
The DK=512 decode-only program does not create the f32_f16 prefill
kernel, so the compiled check in ensure_fa_variant never hit and
supports_op gave inconsistent answers for the same op. block_n is also
unset for DK=512 decode; guard it to avoid an out-of-range read at
dispatch.
* opencl: run DK=512 FA decode on CPU
DK=512 decode is bandwidth-bound and faster on the CPU than the GPU,
increasingly so with depth. Decline it in supports_op; prefill stays on the GPU.
* opencl: compile MQ_GQA=8 FA kernels in a minimal program
The full program compiled with -D MQ_GQA=8 runs the Adreno compiler out
of memory at DK>=256. Only the vec_mq kernels are used from this
program, so compile it with FA_MQ_ONLY, which excludes everything else.
Also include the program name in the compile error log.
* opencl: remove stray token in flash_attn_f32_f16.cl
A stray "." broke the f32_f16 program build.
* opencl: split f16-KV FA decode finer (FD_KV_PER_SPLIT_F16)
The 2048 default under-fills the GPU on single-query f16-KV decode;
use 512 for f16 KV to get more splits. Quantized KV keeps 2048.
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* metal: add col2im_1d op (f32/f16/bf16)
Gather kernel mirroring the CPU/CUDA path: each output (t_out, oc)
reads its ceil(K/s0) source columns with an F32 accumulator, a single
write and no atomics. One thread per output element, 256 per
threadgroup.
* metal: check dst contiguity and type match in supports_op for COL2IM_1D
Align the GGML_OP_COL2IM_1D predicate with the CPU, CUDA, and Vulkan
backends: the kernel writes dst with linear indexing and assumes the
same type as src0, so supports_op must also require a contiguous dst
and op->type == op->src[0]->type.
* Update ggml/src/ggml-metal/ggml-metal.metal
Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>
---------
Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>
The matmul_tiled path uses large local stack buffers for A_pack and B_pack. On AIX this can trigger a segmentation fault, so reduce the buffer footprint there to keep the tiled path usable.
Performance Impact:
~ 2x gains in PP_Speed for FP32, Q4_0 and Q8_0 models tested with llama-bench, llama-batched-bench and llama-cli.
Models used: Llama3.2 3b Instruct F32, qwen 2.5 3b Q4_0 and Q8_0
Tensor parallelism (-sm tensor) combined with -ncmoe (CPU-offloaded MoE
experts) aborts during warm-up on MoE models with
GGML_ASSERT(ggml_is_contiguous(tensor)) in ggml-backend-meta.cpp.
The failing tensor is the MoE router output (ffn_moe_topk): it is mirrored
(GGML_BACKEND_SPLIT_AXIS_MIRRORED, replicated across backends since routing
must be identical) and happens to be a non-contiguous view.
ggml_backend_meta_buffer_{get,set}_tensor asserted contiguity before
consulting the split state, so a mirrored non-contiguous tensor tripped the
assert even though the GGML_BACKEND_SPLIT_AXIS_MIRRORED case right below
already handles it.
Move the split-state lookup above the assert and allow the mirrored case in
both get_tensor and set_tensor.
Diagnosis credit to the reporter (@nathanmp).
Fixes#24886
Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
* Update ggml-cuda.cu - Turing P2P access fix.
* Add original code as fallback behaviour when NCCL or P2P is not set/true.
* Update ggml/src/ggml-cuda/ggml-cuda.cu to add comment as per suggestion
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* cuda : concat implementation for quantized types
* chore : apply am17an clever suggestion to shorten the code
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* cuda: enable topk-moe fusion for 288 experts
The topk-moe fusion only accepted power-of-2 expert counts (or the
special-cased 576), so models with 288 experts (e.g. Step-3.7-Flash)
fell back to the unfused per-layer routing chain: softmax/sigmoid,
argsort, get_rows, sum_rows, div, clamp, scale. At batch size 1 that
is ~330 extra tiny graph nodes per token.
288 is a multiple of the warp size, so the existing kernel already
handles it; this adds the missing template instantiation and accepts
288 in the eligibility check.
Measured on gfx1151 with Step-3.7-Flash IQ4_XS (llama-bench,
-b 4096 -ub 4096 -fa 1 -dio 1 -ctk q8_0 -ctv q8_0; machine idle,
before/after paired so pp4096 stays matched as a load control):
test | before | after
----------------+----------------+----------------
pp4096 | 460.99 ± 0.45 | 462.47 ± 0.34 (unchanged)
tg128 | 19.10 ± 0.04 | 19.56 ± 0.03 (+2.4%)
tg128 @ d30000 | 12.68 ± 0.04 | 12.69 ± 0.03 (unchanged)
Prompt processing is unaffected (the fusion only touches decode
routing). The decode gain is ~+2.4% at shallow context and fades with
depth: by 30k tokens each step is attention-bound over the KV cache,
so removing the fixed routing overhead is no longer visible.
Assisted-By: Claude Fable 5 <noreply@anthropic.com>
* Update tests/test-backend-ops.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Add comment for case 288 in topk-moe.cu
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Remove redundant CUDA copies after gated_delta_net.
Currently, GDN writes recurrent state snapshots into its output tail, then the graph immediately copies those snapshots into ssm_states_all. With MTP draft length 3, target decode uses K=4, so that becomes 4 extra ggml_cuda_cpy calls.
The change detects that gated_delta_net -> view -> cpy pattern and makes the CUDA GDN kernel write the state snapshot(s) directly into the recurrent cache, skipping the intermediate tail writes and copy kernels when safe.
* Address review comments
* hex-mm: fold mm quant tasks into the main matmul threads
* hex-mm: minor formatting fixes
* hex-mm: cleanup is_quant checks in dma dispatch
* hex-mm: fix dst-spad alignment
* hex-mm: move fp kernels in the hvx-mm-kernels header
* hex-mm: fuse with ADD
* hex-fa: factor out ukernels into separate headers and unify the rest
* hex-fa: move kernel-params compute into the host
* hex-fa: refactor vtcm alloc for consistency
* hex-fa: add support for FA_SELECT
* hex-fa: update tracing insrumentation to cover all functions
* hex-fa: update hvx fallback thresholds to recover t/g regressions
* hex-fa: update tracing instrumentation
* hex-fa: improved tracing with additional events
* hex-fa: optimize mask processing (fastdiv, etc)
* hex-fa: improve mask dma caching
* hmx-fa: change loop order to maximize mask cache hits
* hex-fa: remove over instrumentation
* hex-fa: breakdown QKV prep trace events
* hmx-fa: further mask proc optimizations
* hex-fa: mask broadcast is the common case, optimize for that
* hex-fa: use aligned loads where possible
* hex-fa: update loops to use uint32_t indices
* hmx-fa: fold vtcm init into q prep task
* hex-fa: update rest of the hmx funcs to use uint32_t
* hmx-fa: fold build_d into the main softmax loop
* hmx-fa: start kv dmas earlier
* hmx-fa: start mask dma a bit earlier
* hex-fa: precompute rows per task to avoid divs
* hmx-fa: specialize fa_o_store for f16 and f32
* hmx-fa: prelim support for Sinks
* hmx-fa: keep softmax accumulators in fp32
* hex-fa: add tanh_f16 and exp2_f16 and use that in FA
* hex-fa: use fp16 math in the hvx kernel
* hex-fa: avoid expensive float -> __fp16 cast for slopes and softcap
* hex-fa: replace most vec_exp_f32 with vec_exp2_f16
* hmx-fa: vectorize sinks update
* hex-fa: minor formatting
* hmx-fa: fold softcap loop into the tile load
* hmx-fa: use vectoralias to populate sinks
* hex-fa: remove redudant check
* hex-fa: fix vtcm size compute to use fp32 for accumulators
* hex-mm: fix trailing spaces
* hmx-fa: dont use -inf to init mask to avoid conversion overflows
* hex-fa: no need to explicitly guard -inf in the f16->f32 converter now
* hmx-fa: cleanup fa sinks handling
* hex-mm: fixed src2 stride handling when mm is fused with add
* hex-fa: make lto happy
* vulkan: roll bk loop in matmul for asahi linux
* vulkan: fix inline comment
* vulkan: revert BK-loop unroll change
* vulkan: edit spirv directly for asahi roll bk loop
* vulkan: remove trailing whitespace at the end of comments
* HIP: keep MMQ for gfx900 MoE and Q8_0, use hipBLAS for dense K-quants
Assisted-by: GitHub Copilot CLI
* HIP: tighten conditional block to be explicitly for gfx900
* HIP: Further simplified gfx900 conditional block
* removed unnecessary comment
* opencl: rework FA kernel for f16 and f32
* opencl: flash-attention prefill prepass kernels
- flash_attn_kv_pad_f16 pads the tail KV tile to a BLOCK_N multiple
- flash_attn_mask_pad_f16 pads the matching mask tile
- flash_attn_blk_f16 classifies each KV tile per query block as
fully masked / mixed / fully unmasked, so
the main kernel can skip fully-masked tiles
and the mask lookup for fully-unmasked ones
* opencl: FA kernels for q4_0 and q8_0
* opencl: `set_rows` for f32 to q8_0/q4_0
* opencl: dequant kernels for q4_0 and q8_0
* opencl: add FA tile tuning table with override
* opencl: wire host side for FA
* opencl: q4_0 MoE tensors are also SOA'ed
* opencl: cosmetic fix
* opencl: refactor, also clarify some code paths in comments
* opencl: fix inifity for `-cl-finite-math-only`
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* [CUDA] Added a cudaMemcpy2DAsync fast path to ggml_cuda_cpy
Add a CUDA ggml_cpy fast path for same-type, same-shape strided copies that are just 2D pitched block copies.
When tensors are not fully contiguous but each row is contiguous, it now uses cudaMemcpy2DAsync instead of the slow element-wise scalar copy kernel.
This fixes the GDN recurrent snapshot update with -np 4, where rollback slots are separated by cache stride gaps.
* Add new tests that execute the new optimized strided copy path
* Return unsupported for strided copy in OpenVINO, as new tests are failing
* CUDA: Improve performance via less synchronizations between token (#17795)
* Adds CPU-to-CUDA copy capability to
ggml_backend_cuda_cpy_tensor_async()
* Adds function to relax sync requirements between input copies on
supported backends (CUDA for now)
* Exchanges synchronous copy with async copy function.
* Adds macro guards to allow compilation in non-CUDA builds
* Reworked backend detection in ggml-backend.cpp to avoid linking
conflicts
* Relax requirement of checks in async CUDA copies from backend and buffer type to just buffer type, to avoid linking issues
* Minor cleanup
* Makes opt-in to relax use of explicit syncs more general. Backends like
vulkan which require a synchronization between HtoD copies and graph
execution could also adopt this change now.
* Reintroduces stricter check for CPU->CUDA backend async copy via
GGML_DEVICE_TYPE_CPU.
* Corrects initialization of ggml_backend_sync_mode in
ggml_backend_sched_split initialization
* Simplifies synchronizations to adhere to `saaasg` pattern.
* Apply suggestion from @ggerganov (src->buffer to buf_src)
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Apply suggestion from @ggerganov (src->buffer to buf_src) v2
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Apply suggestions from @johannesgaessler code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Adds single-GPU synchronizations to multi-GPU settings to fix hip backend pipeline parallel bugs.
* Scheduler Hardening: Exclude hip/MUSA from copy_from_host CPU split ->
GPU split optimization
* Scheduler Hardening: Re-adding original additional synchronizations for
non-async backends
* Adds disclaimer to hip/musa exclusion of copy_from_host. Highlights that it is out of
precaution, but that no perf-impact is visible, and that it can be
revisited separately anytime.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* vulkan: add INTEL_PRE_XE2 arch enum and enable coopmat1 on Intel Xe-LPG Plus (1/3, Xe1-ARLH)
Co-authored-by: Xia, Jie <jie.xia@intel.com>
Co-authored-by: Liu, Russell <russell.liu@intel.com>
* Address comments of bf16 and trailing whitespace
* Rename INTEL_PRE_XE2 to INTEL_XE1 and remove driver workaround
* Add Windows driver check
---------
Co-authored-by: Xia, Jie <jie.xia@intel.com>
Co-authored-by: Liu, Russell <russell.liu@intel.com>
* ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32
2D convolutions with kernel size 9 produced different results on SVE
enabled ARM devices. After debugging it turned out that ggml_vec_dot_f32
was using data from inactive lanes.
Use svmla_f32_m(pg, sum1, ax1, ay1) so inactive lanes retain sum1.
* cont : clean-up
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Add failing test-case to test-backend-ops
Extracted from https://github.com/ggml-org/llama.cpp/issues/24072
* Minimize repro with help of AI
N = 8 * (65535 - 1) + 1 = 524273
* Port and adjust workaround from 0ba798341e
Fall-back should share code, also relax y-z constraint to be inclusive
* Add test-case + fallback also for y dim
* Fix x-guards which is 2^{31}-1, so inlusive of INT_MAX
* Fix overflow problems for transposed copy kernel
* Sycl tp stage1 (#1)
* SYCL: tensor parallelism (--split-mode tensor) for dual-GPU
Adds the comm_init/comm_free/comm_allreduce_tensor trio that the
meta-backend queries via get_proc_address to enable backend-specific
all-reduce, mirroring the pattern used by ggml-cuda.cu.
For N=2 (the common dual-GPU case) implements a degenerate ring
all-reduce with two size-branched paths:
* Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel
chained via depends_on(memcpy_event). 4 SYCL submissions/call.
* Large (nelem >= 32768): BF16-compressed. Each device compresses
FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's
inbox (HALF the PCIe bytes), then decompresses + adds into the
local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved
-- wins for any tensor where PCIe dominates kernel time.
Threshold and BF16 path pattern mirror the CUDA NCCL allreduce.
Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes
(matches both path layouts: FP32 nelem floats; BF16 outbox+inbox =
2 * nelem uint16_t each). Single alloc+free per device keeps the
SYCL pool's strict-LIFO invariant trivial.
Initial impl handles N=2 FP32 contiguous tensors. Other cases return
false, causing the meta-backend to use its generic butterfly fallback.
Per-call sync is intentionally omitted. SYCL in-order queue semantics
ensure that the meta-backend's next compute on the same per-device
queue waits for our final ADD, and the next allreduce's first op on
the same persistent buffer waits via the same queue. Only comm_free
does an explicit final wait.
OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process
in communicator_impl.hpp:47 (condition devices.size() == 1), which is
incompatible with llama.cpp's single-process multi-GPU model.
Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 +
DPC++ nightly):
Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16:
pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%)
tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%)
Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE):
pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%)
tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%)
Qwen3-4B Q4_K_M:
pp64 = 984.51 t/s, tg16 = 49.29 t/s
Llama-3.3-70B in SYCL TP now comfortably beats production layer mode
on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on
both — the BF16 path is what unlocks the many-medium-allreduces
prefill pattern.
Build/CMake: no changes. No new dependencies. ~210 lines added across
ggml-sycl.h and ggml-sycl.cpp.
* Fix comments
* documentation update to address PR feedback
* Bring over my device-to-device memcpy chagnes
* move the dev2dev_memcpy calls to the upstream 7-parameter variety
* Fix a typo and remove a trailing whitespace
