This introduces an optional feature to allocate large GPU buffers (≥ 1GB)
using USM system allocations if supported by the device. It allows using
buffers from the system allocator then letting the system manage memory
migrations between host and device as necessary.
This feature is disabled by default and requires the GGML_SYCL_USM_SYSTEM
environment variable to enable. If USM system allocations are not supported
by the device or the system, we fallback to regular allocations.
This feature can allow VRAM overcommit. For example, the test below fails
on B580 due to lack of memory for allocation, but it passes when enabling
USM system allocations:
./examples/sycl/test.sh -m Qwen3.5-27B-Q3_K_M.gguf -lv 4
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
* ui: add source toggle to mermaid and svg blocks
Add a toggle button next to copy and preview that switches a rendered
mermaid or svg block to its source code and back. The button is shared by
both block types and the rendered view stays the default.
The source view reuses the code block scroll container and the highlighted
code element captured at transform time, so it matches the app code blocks
without highlighting again.
Make tall diagrams scroll like text code blocks: safe centering keeps the
diagram centered when it fits and falls back to start alignment when it
overflows, so the top stays reachable instead of clipping above.
Keep the block header opaque and layered above the scrolled diagram, and
ignore header clicks in the zoom handler, so a button click never falls
through to the zoom dialog.
* ui: transparent diagram block header, address review from @allozaur
* spec: add spec metrics mean acceptance length and acceptance per pos
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* fix as suggestions
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* sycl: support reordered Q4_K and Q5_K MoE MUL_MAT_ID
Extend reordered-weight handling to fused MoE MUL_MAT_ID for Q4_K and Q5_K expert tensors and add Q5_K reordered DMMV coverage. Unsupported 3D reorder cases now fall back instead of aborting.
* sycl: extend MoE reorder to Q6_K mul_mat_id
* vulkan: add GGML_OP_COL2IM_1D, follow-up to the CPU op
* vulkan: col2im_1d bounded gather loop instead of full-K scan with modulo
* vulkan: col2im_1d address review from @jeffbolznv
* vulkan: col2im_1d return nullptr for unsupported types, address review from @0cc4m
* chat: harden peg-native tool call parsing
accept an optional leading type: function field in
build_json_tools_flat_keys so openai style tool calls parse on
templates whose serialization opens on the name field.
return a clean error and log the unparsed fragment on a final peg
parse failure instead of throwing the raw parser position and input.
keep the raw arguments string in func_args_not_string when it is not
valid json instead of aborting the prompt render.
* chat: surface peg-native parse failures
a final peg parse failure threw the raw parser position and input. log
the unparsed fragment and raise a clearer error instead, so a model
output that does not match the expected format no longer fails silently
with an empty assistant turn.
minimal change, no behavior change on successful parses.
* chat: handle openai style tool calls in peg-native
* nits
* common: scope OpenAI wrapper grammar trigger via autoparser flag
* chat: gate type:function parsing leniency on the analysis flag
Thread accept_openai_wrapper from the generator to build_json_tools_flat_keys
so the leading "type": "function" field is accepted only when openai_wrapper_trigger is set.
* [SYCL] Centralize Level Zero detection in ggml_sycl_init
* use the same wording
* get back the warning
* [SYCL] Remove per-allocation getenv() for GGML_SYCL_ENABLE_LEVEL_ZERO
* bring back the comment
* move it up to make sure devices call the shots
* move the env detection early
* replace g_ggml_sycl_enable_level_zero with a direct call to .ext_oneapi_level_zero
* update the comment
* switch back to g_ggml_sycl_enable_level_zero with a sentinel
* remove the check
* Reduce the diff
* reword, move lower
* move things aroudn
* remove forward declaration if favor of a full replace
* pre-cache results of zeDeviceGetProperties
* put ggml_sycl_get_env back
* replace get_sycl_env with ggml_sycl_get_env
* add whitespace back
* Apply suggestion from @sanmai
* chat: fix whitespace problems once and for all
* Purge trailing spaces from grammar generation
* Revert "Purge trailing spaces from grammar generation"
This reverts commit b0827ecb7d4767f37cefd751b3646f98d5303891.
* ui: add svg block visualizer based on allozaur's mermaid PR
* ui: rationalise diagram block styling and pre transforms shared by mermaid and svg
* ui: live render streaming svg blocks
* ui: also render svg authored in xml code fences
* ui: refactor svg block rendering, address review from allozaur
- Move the svg size ceiling and DOMPurify config out of sanitize-svg.ts into /constants.
- Rename the svg-diagram class to svg-block so the name no longer implies diagrams only.
- Replace the svg, xml and svg tag magic strings in the markdown pipeline with shared constants.
- Promote the data-svg-rendered marker and its sibling data attributes to constants.
* ui: render svg blocks in a shadow root for animation and live zoom
Mount each sanitized svg inside an open shadow root so author <style> and
keyframe or smil animations run while staying scoped to the host element.
Relax the sanitizer to forbid only foreignObject and script, which lets
animation, href and external resource refs through for wider compatibility.
Render the inline block and the zoom dialog from the same reactive source,
so a streaming svg keeps drawing live inside the open zoom popup.
* Add boilerplate for file types
* Add heic-to and implement conversion
* Load heic library from CDN
* Use jpg instead of png for conversion
* Move const to constants file
* ui: make mobile layout keyboard-aware via interactive-widget and dvh shell anchor
* ui: fix duplicate PWA refresh popup by scoping the storage check to non-PWA pages
* Add arch support for cohere2-MoE
* Removed redundant gating_func checks
* Changed ffn lookup to prefer prefix_dense_intermediate_size
* Renamed arch to cohere2moe
* Removed redundant lmhead check and chat template changes
* Removed lm_head.weight check from modify tensors, load output tensor not required, fallback to token_embd.weight
* Changed to (routed+shared)*0.5 for shared expert combined avg
* fixed sliding_window_pattern issue and pattern
* Fixed transformers crash 'first_k_dense_replace' error
* Remove comment
* Removed cohere2-moe as a tokenizer type and kept as tiny_aya. Renamed North-Mini-Code-1.0.
* Fixed MTP fail, changed to use iSWA
* Fixed remaining todos: cohere2moe renamed, changed swa parsing to use get_key_or_arr, removed extra get_arr use
* Force metadata usage
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Remove Cohere2 checkpoint comment
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Remove MTP comment
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Regenerate cohere2moe tokenizer hash
* Add cohere2moe to Llama Model Saver supported list
* Check for zerobios tensors and add support for Command to use LayerNorm
* Map expert_selection_fn to sigmoid in base.py instead of command.py
* use bools for foundnorm/foundnormrms
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
