Xuan-Son Nguyen and GitHub
8c60b8a2be
ci: pin external actions to exact commit SHA ( #21033 )
2026-03-26 20:44:00 +01:00
Xuan-Son Nguyen and GitHub
a73bbd5d92
mtmd: refactor image preprocessing ( #21031 )
...
* mtmd: refactor image pre-processing
* correct some places
* correct lfm2
* fix deepseek-ocr on server
* add comment to clarify about mtmd_image_preprocessor_dyn_size
2026-03-26 19:49:20 +01:00
914eb5ff0c
jinja: fix macro with kwargs ( #20960 )
...
* jinja: fix macro with kwargs
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* fix newline problem
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-03-25 12:22:48 +01:00
bd6992180b
contrib: add "Requirements" section to PR template ( #20841 )
...
* contrib: add "Requirements" section to PR template
* typo [no ci]
* use h2, add "Additional information"
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com >
2026-03-23 16:59:02 +01:00
Xuan-Son Nguyen and GitHub
31a5cf4c3f
server: use httplib dynamic threads ( #20817 )
...
* server: use httplib dynamic threads
* change to n_threads_http + 1024
2026-03-23 12:22:46 +01:00
Xuan-Son Nguyen and GitHub
49bfddeca1
server: allow router to report child instances sleep status ( #20849 )
...
* server: allow router to report child instances sleep status
* refactor
* move sleeping to state
* nits
2026-03-22 18:33:52 +01:00
Xuan-Son Nguyen and GitHub
568aec82d2
docs : explicit about banning accounts that violates policy ( #19593 )
2026-03-21 15:50:16 +01:00
fb78ad29bb
server: (doc) clarify in-scope and out-scope features ( #20794 )
...
* server: (doc) clarify in-scope and out-scope features
* Apply suggestions from code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-03-20 14:03:50 +01:00
Xuan-Son Nguyen and GitHub
1e64534570
mtmd: add clip_graph::build_mm() ( #20751 )
...
* clip: add build_mm()
* apply to all models
* add TODO for bias overload
2026-03-19 13:11:39 +01:00
d34ff7eb5b
model: mistral small 4 support ( #20649 )
...
* model: mistral small 4 support
* fix test
* fix test (2)
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* Update convert_hf_to_gguf.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* change newline
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-03-17 00:31:14 +01:00
Xuan-Son Nguyen and GitHub
94d0262277
mtmd: add llama-mtmd-debug binary ( #20508 )
...
* mtmd: add llama-mtmd-debug binary
* adapt
* fixes
* fix compile error
* fix windows compile error
* rm legacy clip_debug_encode()
* add MTMD_API to fix build
2026-03-14 15:52:29 +01:00
Xuan-Son Nguyen and GitHub
0842b9b465
model: fix step3.5 n_rot ( #20318 )
2026-03-09 23:42:24 +01:00
Xuan-Son Nguyen and GitHub
59db9a357d
llama: dynamic head_dim and n_rot for SWA ( #20301 )
...
* llama: dynamic head_dim and n_rot for SWA
* also add gguf_writer wrappers
* fix build
* build_rope_shift arg reorder
2026-03-09 22:22:39 +01:00
Xuan-Son Nguyen and GitHub
c747294b2d
scripts: update corpus of compare-logprobs ( #19326 )
...
* scripts: update corpus of compare-logprobs
* fix
2026-02-25 12:57:34 +01:00
Xuan-Son Nguyen and GitHub
5452d736f8
jinja: correct stats for tojson and string filters ( #19785 )
2026-02-22 21:08:23 +01:00
eeef3cfced
model: support GLM-OCR ( #19677 )
...
* model: support GLM-OCR
* Update convert_hf_to_gguf.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-02-18 17:51:40 +01:00
752584d5f5
model: support GLM MoE DSA arch (NOTE: indexer is not yet supported) ( #19460 )
...
* model: support GLM MoE DSA arch
* working version
* pyright
* keep indexer tensors
* add indexer gguf params
* loaded now
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* update
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* minor fix and cleanup
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-02-13 14:56:53 +01:00
Xuan-Son Nguyen and GitHub
9a96352729
test: fix IMROPE perf test case ( #19465 )
2026-02-10 14:37:50 +01:00
Xuan-Son Nguyen and GitHub
98e57ca422
chat: fix case where template accepts type content only ( #19419 )
...
* chat: fix case where template accepts type content only
* rm stray log
* reuse render_message_to_json
2026-02-09 22:14:12 +01:00
Xuan-Son Nguyen and GitHub
e0c93af2a0
debug: make common_debug_print_tensor readable ( #19331 )
...
* debug: make common_debug_print_tensor readable
* editorconfig
2026-02-04 17:55:31 +01:00
Xuan-Son Nguyen and GitHub
8abcc70a74
model: (qwen3next) correct vectorized key_gdiff calculation ( #19324 )
...
* model: (qwen3next) correct vectorized key_gdiff calculation
* move transpose to outside of loop
2026-02-04 13:09:58 +01:00
Xuan-Son Nguyen and GitHub
07a7412a3b
mtmd: add min/max pixels gguf metadata ( #19273 )
2026-02-02 20:59:06 +01:00
bb02f74c61
chat: fix language input for translategemma ( #19052 )
...
* chat: fix language input for translategemma
* Update common/chat.cpp
Co-authored-by: Aldehir Rojas <hello@alde.dev >
---------
Co-authored-by: Aldehir Rojas <hello@alde.dev >
2026-01-24 17:58:45 +01:00
Xuan-Son Nguyen and GitHub
b5b8fa1c8b
chat : fix translategemma crash on common_chat_format_example ( #19019 )
2026-01-23 12:03:42 +01:00
Xuan-Son Nguyen and GitHub
51fa458a92
server : support preserving reasoning_content in assistant message ( #18994 )
...
* support reasoning_content input
* report template caps to webui
* add docs
* rm commented code
2026-01-22 21:30:06 +01:00
Xuan-Son Nguyen and GitHub
4e595b250a
server: do not log certain endpoints (avoid log spam) ( #19028 )
2026-01-22 19:24:37 +01:00
Xuan-Son Nguyen and GitHub
9eb5bfec1a
mtmd : update docs to use llama_model_n_embd_inp ( #18999 )
2026-01-22 14:36:32 +01:00
Xuan-Son Nguyen and GitHub
2c1f199653
cli : fix reasoning responses in CLI ( #18961 )
...
* cli : fix reasoning responses in CLI
* fix build
* fix build (2)
2026-01-20 18:23:25 +01:00
Xuan-Son Nguyen and GitHub
6df686bee6
server : refactor oai_parser_opt, move it to server_chat_params ( #18937 )
...
* server_chat_params
* move chat format into CLI
* use meta whenever possible
* clean up, no more chatml fallback
2026-01-19 23:28:01 +01:00
fe44d35574
tests : add test-jinja -py option for cross-checking ( #18906 )
...
* tests : add test-jinja -py option or cross-checking
* Update tests/test-jinja.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* fix + add source
* SandboxedEnvironment
* fix array.map case
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-01-18 08:14:27 +01:00
Xuan-Son Nguyen and GitHub
f55b033ae6
jinja: correct member access rule ( #18905 )
2026-01-18 00:48:55 +01:00
c15395f73c
common : implement new jinja template engine ( #18462 )
...
* jinja vm
* lexer
* add vm types
* demo
* clean up
* parser ok
* binary_expression::execute
* shadow naming
* bin ops works!
* fix map object
* add string builtins
* add more builtins
* wip
* use mk_val
* eval with is_user_input
* render gemma tmpl ok
* track input string even after transformations
* support binded functions
* keyword arguments and slicing array
* use shared_ptr for values
* add mk_stmt
* allow print source on exception
* fix negate test
* testing more templates
* mostly works
* add filter_statement
* allow func to access ctx
* add jinja-value.cpp
* impl global_from_json
* a lot of fixes
* more tests
* more fix, more tests
* more fixes
* rm workarounds
* demo: type inferrence
* add placeholder for tojson
* improve function args handling
* rm type inference
* no more std::regex
* trailing spaces
* make testing more flexible
* make output a bit cleaner
* (wip) redirect minja calls
* test: add --output
* fix crash on macro kwargs
* add minimal caps system
* add some workarounds
* rm caps_apply_workarounds
* get rid of preprocessing
* more fixes
* fix test-chat-template
* move test-chat-jinja into test-chat-template
* rm test-chat-jinja from cmake
* test-chat-template: use common
* fix build
* fix build (2)
* rename vm --> interpreter
* improve error reporting
* correct lstrip behavior
* add tojson
* more fixes
* disable tests for COMMON_CHAT_FORMAT_GENERIC
* make sure tojson output correct order
* add object.length
* fully functional selectattr / rejectattr
* improve error reporting
* more builtins added, more fixes
* create jinja rendering tests
* fix testing.h path
* adjust whitespace rules
* more fixes
* temporary disable test for ibm-granite
* r/lstrip behavior matched with hf.js
* minimax, glm4.5 ok
* add append and pop
* kimi-k2 ok
* test-chat passed
* fix lstrip_block
* add more jinja tests
* cast to unsigned char
* allow dict key to be numeric
* nemotron: rm windows newline
* tests ok
* fix test
* rename interpreter --> runtime
* fix build
* add more checks
* bring back generic format support
* fix Apertus
* [json.exception.out_of_range.403] key 'content' not found
* rm generic test
* refactor input marking
* add docs
* fix windows build
* clarify error message
* improved tests
* split/rsplit with maxsplit
* non-inverse maxsplit
forgot to change after simplifying
* implement separators for tojson and fix indent
* i like to move it move it
* rename null -- > none
* token::eof
* some nits + comments
* add exception classes for lexer and parser
* null -> none
* rename global -> env
* rm minja
* update docs
* docs: add input marking caveats
* imlement missing jinja-tests functions
* oops
* support trim filter with args, remove bogus to_json reference
* numerous argument fixes
* updated tests
* implement optional strip chars parameter
* use new chars parameter
* float filter also has default
* always leave at least one decimal in float string
* jinja : static analysis + header cleanup + minor fixes
* add fuzz test
* add string.cpp
* fix chat_template_kwargs
* nits
* fix build
* revert
* unrevert
sorry :)
* add fuzz func_args, refactor to be safer
* fix array.map()
* loosen ensure_vals max count condition, add not impl for map(int)
* hopefully fix windows
* check if empty first
* normalize newlines
---------
Co-authored-by: Alde Rojas <hello@alde.dev >
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-01-16 11:22:06 +01:00
Xuan-Son Nguyen and GitHub
a04c2b06a3
server: improve slots scheduling for n_cmpl ( #18789 )
...
* server : make sure children tasks are scheduled to launch with parent
* fix
* add comment pointing to this PR
* fix
* clean up
* more debug messages
* add pop_deferred_task with specific ID version
* improve the logic
* simple approach
* no double move
* correct return type of launch_slots_with_parent_task
2026-01-15 17:10:28 +01:00
a7e6ddb8bd
lora: make sure model keep track of associated adapters ( #18490 )
...
* lora: make sure model keep track of associated adapters
* deprecate llama_adapter_lora_free
* minor : std::unordered_set over std::set
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-01-15 10:24:28 +01:00
Xuan-Son Nguyen and GitHub
e047f9ee9d
mtmd: fix use_non_causal being reported incorrectly ( #18793 )
...
* mtmd: fix use_non_causal being reported incorrectly
* move clip_is_mrope to mtmd_decode_use_mrope
* fix sloppy code ggml_cpy
2026-01-13 12:19:38 +01:00
Xuan-Son Nguyen and GitHub
ce3bf9b1a4
server: update docs for sleeping [no ci] ( #18777 )
2026-01-12 13:01:24 +01:00
Xuan-Son Nguyen and GitHub
0c3b7a9efe
model: fix qwen3next broken due to #18683 ( #18762 )
2026-01-11 21:00:10 +01:00
Xuan-Son Nguyen and GitHub
4b060bf240
security: make it clear about subtopics in server ( #18754 )
...
* security: make it clear about subtopics in server
* exclude DoS
2026-01-11 16:51:03 +01:00
Xuan-Son Nguyen and GitHub
506bb6e010
model: try to improve Qwen3 Next ( #18683 )
...
* qwen3next: simplify qkvz projection
* use ggml_swiglu_split
* revert swiglu_split, but remove redundant repeat()
* fix missing reshape
* rm 2 redundant transposes
* move mul_mat(k,q) to outside of chunking
* rm redundant cont
* improve g_cs_chunk
* add comments about no cont
* use std::pair instead of ggml_concat
* vectorize key_gdiff calculation
* rm unused tensor
* avoid ggml_concat inside loop
* bring back ggml_concat as it may not work on other backend
* nits
2026-01-11 12:53:33 +01:00
Xuan-Son Nguyen and GitHub
28068af789
security: narrow down the scope of what we consider a vulnerability ( #18752 )
...
* security: narrow down the scope of what we consider a vulnerability
* fix typo
2026-01-11 12:23:36 +01:00
Xuan-Son Nguyen and GitHub
23f82f2420
preset: allow named remote preset ( #18728 )
...
* preset: allow named remote preset
* nits: fix docs
* cont docs
2026-01-10 15:12:29 +01:00
9ac2693a30
server: fix n_cmpl not skipping processing prompt ( #18663 )
...
* server: fix n_cmpl not skipping processing
* fix infinite loop on empty batch
* cont : init child samplers + modify child logic
* cont : cleanup
* cont : improve n_cmpl logic
- launch the parent task first so it finds the slot with best cache
- parent task waits for child tasks to be launched
- when a child task finishes - remove its cache
* cont : remove redundant function
* cont : reduce parent checks
* fix : nullptr task dereference
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-01-10 00:00:41 +01:00
Xuan-Son Nguyen and GitHub
8ece3836b4
common: support remote preset ( #18520 )
...
* arg: support remote preset
* proof reading
* allow one HF repo to point to multiple HF repos
* docs: mention about multiple GGUF use case
* correct clean_file_name
* download: also return HTTP status code
* fix case with cache file used
* fix --offline option
2026-01-08 22:35:40 +01:00
Xuan-Son Nguyen and GitHub
07fbe19f1f
arg: use CSV escape style for multiple-value args ( #18643 )
...
* arg: use CSV escape style for multiple-value args
* add test
2026-01-06 17:51:08 +01:00
Xuan-Son Nguyen and GitHub
cd78e57c3a
lora: count lora nodes in graph_max_nodes ( #18469 )
...
* lora: count lora nodes in graph_max_nodes
* 3 nodes per weight
* 4 nodes
* keep track n_lora_nodes from llama_model
* fix assert
* rm redundant header
* common: load adapters before context creation
* use 6 nodes
2025-12-30 15:53:12 +01:00
Xuan-Son Nguyen and GitHub
51a48720b8
webui: fix prompt progress ETA calculation ( #18468 )
...
* webui: fix prompt progress ETA calculation
* handle case done === 0
2025-12-29 21:42:11 +01:00
3595ae5963
contributing: tighten AI usage policy ( #18388 )
...
* contributing: tighten AI usage policy
* refactor AGENTS.md
* proofreading
* update contributing
* add claude.md
* add trailing newline
* add note about dishonest practices
* rm point about dishonest
* rm requirement watermarking
* add .gemini/settings.json
* allow initially AI-generated content
* revise
* Update CONTRIBUTING.md
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
* improve
* trailing space
* Apply suggestions from code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
* update
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
2025-12-29 16:01:32 +01:00
Xuan-Son Nguyen and GitHub
cffa5c46ea
mtmd: clarify that we no longer accept AI-generated PRs ( #18406 )
2025-12-28 09:57:04 +01:00
Xuan-Son Nguyen and GitHub
f5acfb2ffa
server: (router) add stop-timeout option ( #18350 )
...
* server: (router) add stop-timeout option
* also allow stop while loading
* add docs
* unload_lru: also wait for unload to complete
2025-12-24 23:47:49 +01:00
4cbafad4f0
model: support MiMo-V2-Flash ( #18328 )
...
* mimov2: convert ok
* rename mimov2 --> mimo2
* fix conversion
* runnable not incorrect
* use sink
* add_sliding_window_pattern
* add swa and per-layer n_head_kv
* correct params
* somewhat working
* correct gating func
* nits
* mimo2: wire RMS eps + MoE bias + converter guards
* add co-author
Co-authored-by: Aaryan-Kapoor <Aaryan-Kapoor@users.noreply.github.com >
* use add_rope_freq_base_swa
---------
Co-authored-by: Aaryan Kapoor <aaryankapoor2006@gmail.com >
Co-authored-by: Aaryan-Kapoor <Aaryan-Kapoor@users.noreply.github.com >
2025-12-24 23:07:08 +01:00
Xuan-Son Nguyen and GitHub
5ee4e43f26
server: return_progress to also report 0% processing state ( #18305 )
2025-12-23 21:49:05 +01:00
Xuan-Son Nguyen and GitHub
849d021104
server: fix crash with model not having BOS/EOS ( #18321 )
2025-12-23 14:39:36 +01:00
Xuan-Son Nguyen and GitHub
179fd82a72
gen-docs: automatically update markdown file ( #18294 )
...
* gen-docs: automatically update markdown file
* also strip whitespace
* do not add extra newline
* update TOC
2025-12-22 19:30:19 +01:00
Xuan-Son Nguyen and GitHub
6ce863c803
server: prevent data race from HTTP threads ( #18263 )
...
* server: prevent data race from HTTP threads
* fix params
* fix default_generation_settings
* nits: make handle_completions_impl looks less strange
* stricter const
* fix GGML_ASSERT(idx < states.size())
* move index to be managed by server_response_reader
* http: make sure req & res lifecycle are tied together
* fix compile
* fix index handling buggy
* fix data race for lora endpoint
* nits: fix shadow variable
* nits: revert redundant changes
* nits: correct naming for json_webui_settings
2025-12-22 14:23:34 +01:00
Xuan-Son Nguyen and GitHub
3997c78e33
server: fix data race in to_json_anthropic ( #18283 )
2025-12-22 13:21:43 +01:00
Xuan-Son Nguyen and GitHub
86af848153
server: (docs) remove mention about extra_args ( #18262 )
2025-12-22 12:22:01 +01:00
Xuan-Son Nguyen and GitHub
5e25ddebff
move copilot instructions to AGENTS.md ( #18259 )
...
* move copilot --> agents.md
* agents: add disclose AI usage
* refine
2025-12-21 19:09:21 +01:00
Xuan-Son Nguyen and GitHub
ddcb75dd8a
server: add auto-sleep after N seconds of idle ( #18228 )
...
* implement sleeping at queue level
* implement server-context suspend
* add test
* add docs
* optimization: add fast path
* make sure to free llama_init
* nits
* fix use-after-free
* allow /models to be accessed during sleeping, fix use-after-free
* don't allow accessing /models during sleep, it is not thread-safe
* fix data race on accessing props and model_meta
* small clean up
* trailing whitespace
* rm outdated comments
2025-12-21 02:24:42 +01:00
9e39a1e6a9
server: support load model on startup, support preset-only options ( #18206 )
...
* server: support autoload model, support preset-only options
* add docs
* load-on-startup
* fix
* Update common/arg.cpp
Co-authored-by: Pascal <admin@serveurperso.com >
---------
Co-authored-by: Pascal <admin@serveurperso.com >
2025-12-20 09:25:27 +01:00
Xuan-Son Nguyen and GitHub
98c1c7a7bf
presets: refactor, allow cascade presets from different sources, add global section ( #18169 )
...
* presets: refactor, allow cascade presets from different sources
* update docs
* fix neg arg handling
* fix empty mmproj
* also filter out server-controlled args before to_ini()
* skip loading custom_models if not specified
* fix unset_reserved_args
* fix crash on windows
2025-12-19 12:08:20 +01:00
8ea958d4d9
model : add ASR support for LFM2-Audio-1.5B (conformer) ( #18106 )
...
* ASR with LFM2-Audio-1.5B
* Set rope_theta
* Fix comment
* Remove rope_theta setting
* Address PR feedback
* rename functions to conformer
* remove some redundant ggml_cont
* fix missing tensor
* add prefix "a." for conv tensors
* remove redundant reshape
* clean up
* add test model
---------
Co-authored-by: Tarek Dakhran <tarek@liquid.ai >
2025-12-19 00:18:01 +01:00
Xuan-Son Nguyen and GitHub
4d1316c440
arg: fix ASAN error on sampler_type_names empty ( #18167 )
2025-12-18 14:30:32 +01:00
Xuan-Son Nguyen and GitHub
e85e9d7637
server: (router) disable SSL on child process ( #18141 )
2025-12-17 21:39:08 +01:00
Xuan-Son Nguyen and GitHub
bde461de8c
server: (router) allow child process to report status via stdout ( #18110 )
...
* server: (router) allow child process to report status via stdout
* apply suggestions
2025-12-17 14:54:11 +01:00
Xuan-Son Nguyen and GitHub
ef83fb8601
model: fix LFM2 missing tensors ( #18105 )
2025-12-16 19:07:43 +01:00
Xuan-Son Nguyen and GitHub
7f2b2f3c77
arch: refactor LLM_TENSOR_NAMES ( #18051 )
...
* arch: refactor LLM_TENSOR_NAMES
* update docs
* typo
* fix LLM_ARCH_NEMOTRON_H_MOE
* show more meaningful error message on missing tensor
* fix and tested LLM_ARCH_NEMOTRON_H_MOE
2025-12-16 13:22:30 +01:00
Xuan-Son Nguyen and GitHub
7b1db3d3b7
arg: clarify auto kvu/np being set on server ( #17997 )
...
* arg: clarify auto kvu/np being set on server
* improve docs
* use invalid_argument
2025-12-16 12:01:27 +01:00
3d86c6c2b5
model: support GLM4V vision encoder ( #18042 )
...
* convert ok
* no deepstack
* less new tensors
* cgraph ok
* add mrope for text model
* faster patch merger
* add GGML_ROPE_TYPE_MRNORM
* add support for metal
* move glm4v do dedicated graph
* convert: add norm_embd
* clip: add debugging fn
* working correctly
* fix style
* use bicubic
* fix mrope metal
* improve cpu
* convert to neox ordering on conversion
* revert backend changes
* force stop if using old weight
* support moe variant
* fix conversion
* fix convert (2)
* Update tools/mtmd/clip-graph.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* process mrope_section on TextModel base class
* resolve conflict merge
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-16 11:25:26 +01:00
96a181a933
mtmd: refactor audio preprocessing ( #17978 )
...
* mtmd: refactor audio preprocessing
* refactor
Co-authored-by: Tarek <tdakhran@users.noreply.github.com >
* wip
* wip (2)
* improve constructor
* fix use_natural_log
* fix padding for short input
* clean up
* remove need_chunking
---------
Co-authored-by: Tarek <tdakhran@users.noreply.github.com >
2025-12-15 14:16:52 +01:00
Xuan-Son Nguyen and GitHub
52392291b2
preset: handle negated arg, reverse the meaning if needed ( #18041 )
2025-12-14 22:08:10 +01:00
Xuan-Son Nguyen and GitHub
0759b09c90
graph: add f_attn_temp_offset ( #18025 )
2025-12-14 13:05:59 +01:00
Xuan-Son Nguyen and GitHub
c00ff929dc
scripts: add script to compare logprobs of llama.cpp against other frameworks ( #17947 )
...
* scripts: add script to compare logits of llama.cpp against other frameworks
* accept custom prompt file
* fix code style
* clarify endpoint
* fix displaying
* use abs for diff
* fix vllm case
* rm output file
* rename to compare-logprobs
* add "pattern"
2025-12-13 22:33:29 +01:00
Xuan-Son Nguyen and GitHub
4d5ae24c0a
arg: fix common_params_parse not accepting negated arg ( #17991 )
2025-12-13 12:53:37 +01:00
380b4c984e
common: support negated args ( #17919 )
...
* args: support negated args
* update docs
* fix typo
* add more neg options
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* rm duplicated arg
* fix LLAMA_ARG_NO_HOST
* add test
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-12 23:58:53 +01:00
Xuan-Son Nguyen and GitHub
e39a2ce66d
clip: move model cgraphs into their own files ( #17965 )
...
* clip: move model cgraphs into their own files
* more explicit enums
* fix linux build
* fix naming
* missing headers
* nits: add comments for contributors
2025-12-12 21:14:48 +01:00
Xuan-Son Nguyen and GitHub
17158965ac
mtmd: explicitly forbidden inclusion of private header and libcommon ( #17946 )
2025-12-12 15:16:06 +01:00
Xuan-Son Nguyen and GitHub
54a0fee4b7
arg: add -mm and -mmu as short form of --mmproj and --mmproj-url ( #17958 )
...
* arg: add -mm and -mmu as short form of --mmproj and --mmproj-url
* correct order
* update docs
2025-12-12 14:06:06 +01:00
Xuan-Son Nguyen and GitHub
c6b2c9310c
mtmd: some small clean up ( #17909 )
...
* clip: add support for fused qkv in build_vit
* use bulid_ffn whenever possible
* fix internvl
* mtmd-cli: move image to beginning
* test script: support custom args
2025-12-10 22:20:06 +01:00
34a6d86982
cli: enable jinja by default ( #17911 )
...
* cli: enable jinja by default
* Update common/arg.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-10 22:19:42 +01:00
6c2131773c
cli: new CLI experience ( #17824 )
...
* wip
* wip
* fix logging, add display info
* handle commands
* add args
* wip
* move old cli to llama-completion
* rm deprecation notice
* move server to a shared library
* move ci to llama-completion
* add loading animation
* add --show-timings arg
* add /read command, improve LOG_ERR
* add args for speculative decoding, enable show timings by default
* add arg --image and --audio
* fix windows build
* support reasoning_content
* fix llama2c workflow
* color default is auto
* fix merge conflicts
* properly fix color problem
Co-authored-by: bandoti <bandoti@users.noreply.github.com >
* better loading spinner
* make sure to clean color on force-exit
* also clear input files on "/clear"
* simplify common_log_flush
* add warning in mtmd-cli
* implement console writter
* fix data race
* add attribute
* fix llama-completion and mtmd-cli
* add some notes about console::log
* fix compilation
---------
Co-authored-by: bandoti <bandoti@users.noreply.github.com >
2025-12-10 15:28:59 +01:00
9e79b0116e
convert: allow using quantized Mistral weight ( #17889 )
...
* convert: allow using quantized Mistral weight
* data_torch.ndim
* update dequant fn
Co-authored-by: compilade <compilade@users.noreply.github.com >
---------
Co-authored-by: compilade <compilade@users.noreply.github.com >
2025-12-10 10:26:22 +01:00
4e842d5120
console: allow using arrow left/right, home/end keys and history mode ( #17836 )
...
* console: allow using arrow left/right to edit the line (with UTF-8 support)
* console: fix arrow keys on Windows using private-use Unicode
* console: add Home/End key support for Windows and Linux
* console: add basic Up/Down history navigation
* fix build
* console: allow using arrow left/right to edit the line (with UTF-8 support)
* console: fix arrow keys on Windows using private-use Unicode
* console: add Home/End key support for Windows and Linux
* console: add basic Up/Down history navigation
* console: remove unreachable wc == 0 check after VK switch
* console: add Ctrl+Left/Right word navigation
- Add KEY_CTRL_ARROW_LEFT and KEY_CTRL_ARROW_RIGHT codes
- Windows: detect CTRL modifier via dwControlKeyState
- Linux: parse ANSI sequences with modifier (1;5D/C)
- Implement move_word_left/right with space-skipping logic
- Refactor escape sequence parsing to accumulate params
* console: add Delete key support
- Windows: VK_DELETE detection
- Linux: ESC[3~ sequence parsing
- Forward character deletion with UTF-8 support
* console: implement bash-style history editing
- Edit any history line during UP/DOWN navigation, edits persist
- Pressing Enter appends edited version as new history entry
- Original line stay untouched in their positions
* clean up
* better history impl
* fix decode_utf8
---------
Co-authored-by: Pascal <admin@serveurperso.com >
2025-12-09 11:53:59 +01:00
Xuan-Son Nguyen and GitHub
951520ddb0
server: delegate result_state creation to server_task ( #17835 )
...
* server: delegate result_state creation to server_task
* remove unued states
* add more docs
2025-12-08 17:04:38 +01:00
f896d2c34f
server: improve speed of speculative decoding ( #17808 )
...
* server: improve speed of speculative decoding
* fix small draft case
* add link to the PR
* server : fix generation time measurement
* server : fix draft acceptance logs (add SRV_CNT, SLT_CNT macros)
* server : add comment
* add PR to docs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-08 14:35:28 +01:00
Xuan-Son Nguyen and GitHub
37a4f63244
server : add development documentation ( #17760 )
...
* first draft
* rewrite
* update & remove duplicated sections
2025-12-08 13:54:58 +01:00
Xuan-Son Nguyen and GitHub
4d3726278b
model: add llama 4 scaling for mistral-large (deepseek arch) ( #17744 )
2025-12-07 22:29:54 +01:00
Xuan-Son Nguyen and GitHub
c42712b056
server: support multiple generations from one prompt (OAI "n" option) ( #17775 )
...
* backend support
* server: support multiple generations from one prompt (OAI "n" option)
* fix invalid batch
* format oai
* clean up
* disable ctx shift
* add test
* update comments
* fix style
* add n_cmpl to docs [no ci]
* allowing using both n_cmpl and n
2025-12-06 15:54:38 +01:00
dbc15a7967
convert: support Mistral 3 Large MoE ( #17730 )
...
* convert: support Mistral 3 Large MoE
* filter out vision tensors, add missing keys
* handle vocab
* add temperature_length
* fix mscale_all_dim
* clean up
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* fix
* Update gguf-py/gguf/tensor_mapping.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-06 10:49:33 +01:00
Xuan-Son Nguyen and GitHub
9d0229967a
server: strip content-length header on proxy ( #17734 )
2025-12-04 16:32:57 +01:00
c4c10bfb86
server: move msg diffs tracking to HTTP thread ( #17740 )
...
* server: move msg diffs tracking to HTTP thread
* wip
* tool call tests ok
* minor : style
* cont : fix
* move states to server_response_reader
* add safe-guard
* fix
* fix 2
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-04 15:46:08 +01:00
Xuan-Son Nguyen and GitHub
13628d8bdb
server: add --media-path for local media files ( #17697 )
...
* server: add --media-path for local media files
* remove unused fn
2025-12-02 22:49:20 +01:00
Xuan-Son Nguyen and GitHub
a96283adc4
mtmd: fix --no-warmup ( #17695 )
2025-12-02 22:48:08 +01:00
Xuan-Son Nguyen and GitHub
2c453c6c77
convert: add error message for mistral3 quantized weight ( #17686 )
2025-12-02 11:48:31 +01:00
Xuan-Son Nguyen and GitHub
5d6bd842ea
server: remove default "gpt-3.5-turbo" model name ( #17668 )
...
* server: remove default "gpt-3.5-turbo" model name
* do not reflect back model name from request
* fix test
2025-12-02 11:38:57 +01:00
Xuan-Son Nguyen and GitHub
682e6658bb
server: explicitly set exec path when create new instance ( #17669 )
...
* Revert "rm unused fn"
This reverts commit f2dbe9c087ac8776cd41f747a8b0ddf448c8399a.
* server: explicitly set exec path when create new instance
* put back TODO
* only call get_server_exec_path() once
* add fallback logic
2025-12-02 10:25:11 +01:00
Xuan-Son Nguyen and GitHub
ecf74a8417
mtmd: add mtmd_context_params::warmup option ( #17652 )
...
* mtmd: add mtmd_context_params::warmup option
* reuse the common_params::warmup
2025-12-01 21:32:25 +01:00
ec18edfcba
server: introduce API for serving / loading / unloading multiple models ( #17470 )
...
* server: add model management and proxy
* fix compile error
* does this fix windows?
* fix windows build
* use subprocess.h, better logging
* add test
* fix windows
* feat: Model/Router server architecture WIP
* more stable
* fix unsafe pointer
* also allow terminate loading model
* add is_active()
* refactor: Architecture improvements
* tmp apply upstream fix
* address most problems
* address thread safety issue
* address review comment
* add docs (first version)
* address review comment
* feat: Improved UX for model information, modality interactions etc
* chore: update webui build output
* refactor: Use only the message data `model` property for displaying model used info
* chore: update webui build output
* add --models-dir param
* feat: New Model Selection UX WIP
* chore: update webui build output
* feat: Add auto-mic setting
* feat: Attachments UX improvements
* implement LRU
* remove default model path
* better --models-dir
* add env for args
* address review comments
* fix compile
* refactor: Chat Form Submit component
* ad endpoint docs
* Merge remote-tracking branch 'webui/allozaur/server_model_management_v1_2' into xsn/server_model_maagement_v1_2
Co-authored-by: Aleksander <aleksander.grygier@gmail.com >
* feat: Add copy to clipboard to model name in model info dialog
* feat: Model unavailable UI state for model selector
* feat: Chat Form Actions UI logic improvements
* feat: Auto-select model from last assistant response
* chore: update webui build output
* expose args and exit_code in API
* add note
* support extra_args on loading model
* allow reusing args if auto_load
* typo docs
* oai-compat /models endpoint
* cleaner
* address review comments
* feat: Use `model` property for displaying the `repo/model-name` naming format
* refactor: Attachments data
* chore: update webui build output
* refactor: Enum imports
* feat: Improve Model Selector responsiveness
* chore: update webui build output
* refactor: Cleanup
* refactor: Cleanup
* refactor: Formatters
* chore: update webui build output
* refactor: Copy To Clipboard Icon component
* chore: update webui build output
* refactor: Cleanup
* chore: update webui build output
* refactor: UI badges
* chore: update webui build output
* refactor: Cleanup
* refactor: Cleanup
* chore: update webui build output
* add --models-allow-extra-args for security
* nits
* add stdin_file
* fix merge
* fix: Retrieve lost setting after resolving merge conflict
* refactor: DatabaseStore -> DatabaseService
* refactor: Database, Conversations & Chat services + stores architecture improvements (WIP)
* refactor: Remove redundant settings
* refactor: Multi-model business logic WIP
* chore: update webui build output
* feat: Switching models logic for ChatForm or when regenerating messges + modality detection logic
* chore: update webui build output
* fix: Add `untrack` inside chat processing info data logic to prevent infinite effect
* fix: Regenerate
* feat: Remove redundant settigns + rearrange
* fix: Audio attachments
* refactor: Icons
* chore: update webui build output
* feat: Model management and selection features WIP
* chore: update webui build output
* refactor: Improve server properties management
* refactor: Icons
* chore: update webui build output
* feat: Improve model loading/unloading status updates
* chore: update webui build output
* refactor: Improve API header management via utility functions
* remove support for extra args
* set hf_repo/docker_repo as model alias when posible
* refactor: Remove ConversationsService
* refactor: Chat requests abort handling
* refactor: Server store
* tmp webui build
* refactor: Model modality handling
* chore: update webui build output
* refactor: Processing state reactivity
* fix: UI
* refactor: Services/Stores syntax + logic improvements
Refactors components to access stores directly instead of using exported getter functions.
This change centralizes store access and logic, simplifying component code and improving maintainability by reducing the number of exported functions and promoting direct store interaction.
Removes exported getter functions from `chat.svelte.ts`, `conversations.svelte.ts`, `models.svelte.ts` and `settings.svelte.ts`.
* refactor: Architecture cleanup
* feat: Improve statistic badges
* feat: Condition available models based on modality + better model loading strategy & UX
* docs: Architecture documentation
* feat: Update logic for PDF as Image
* add TODO for http client
* refactor: Enhance model info and attachment handling
* chore: update webui build output
* refactor: Components naming
* chore: update webui build output
* refactor: Cleanup
* refactor: DRY `getAttachmentDisplayItems` function + fix UI
* chore: update webui build output
* fix: Modality detection improvement for text-based PDF attachments
* refactor: Cleanup
* docs: Add info comment
* refactor: Cleanup
* re
* refactor: Cleanup
* refactor: Cleanup
* feat: Attachment logic & UI improvements
* refactor: Constants
* feat: Improve UI sidebar background color
* chore: update webui build output
* refactor: Utils imports + move types to `app.d.ts`
* test: Fix Storybook mocks
* chore: update webui build output
* test: Update Chat Form UI tests
* refactor: Tooltip Provider from core layout
* refactor: Tests to separate location
* decouple server_models from server_routes
* test: Move demo test to tests/server
* refactor: Remove redundant method
* chore: update webui build output
* also route anthropic endpoints
* fix duplicated arg
* fix invalid ptr to shutdown_handler
* server : minor
* rm unused fn
* add ?autoload=true|false query param
* refactor: Remove redundant code
* docs: Update README documentations + architecture & data flow diagrams
* fix: Disable autoload on calling server props for the model
* chore: update webui build output
* fix ubuntu build
* fix: Model status reactivity
* fix: Modality detection for MODEL mode
* chore: update webui build output
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-01 19:41:04 +01:00
Xuan-Son Nguyen and GitHub
7733409734
common: improve verbosity level definitions ( #17630 )
...
* common: improve verbosity level definitions
* string_format
* update autogen docs
2025-12-01 14:38:13 +01:00
cd3c118908
model: support Ministral3 ( #17644 )
...
* conversion script
* support ministral 3
* maybe this is better?
* add TODO for rope_yarn_log_mul
* better ppl (tested on 14B-Instruct)
* Add Ministral3 support to Mistral format
* improve arch handling
* add sizes
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* nits
---------
Co-authored-by: Julien Denize <julien.denize@mistral.ai >
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-01 12:26:52 +01:00
Xuan-Son Nguyen and GitHub
ff90508d68
contributing: update guidelines for AI-generated code ( #17625 )
...
* contributing: update guidelines for AI-generated code
* revise
2025-11-30 22:51:34 +01:00
