Xuan-Son Nguyen and GitHub
f5acfb2ffa
server: (router) add stop-timeout option ( #18350 )
...
* server: (router) add stop-timeout option
* also allow stop while loading
* add docs
* unload_lru: also wait for unload to complete
2025-12-24 23:47:49 +01:00
4cbafad4f0
model: support MiMo-V2-Flash ( #18328 )
...
* mimov2: convert ok
* rename mimov2 --> mimo2
* fix conversion
* runnable not incorrect
* use sink
* add_sliding_window_pattern
* add swa and per-layer n_head_kv
* correct params
* somewhat working
* correct gating func
* nits
* mimo2: wire RMS eps + MoE bias + converter guards
* add co-author
Co-authored-by: Aaryan-Kapoor <Aaryan-Kapoor@users.noreply.github.com >
* use add_rope_freq_base_swa
---------
Co-authored-by: Aaryan Kapoor <aaryankapoor2006@gmail.com >
Co-authored-by: Aaryan-Kapoor <Aaryan-Kapoor@users.noreply.github.com >
2025-12-24 23:07:08 +01:00
Xuan-Son Nguyen and GitHub
5ee4e43f26
server: return_progress to also report 0% processing state ( #18305 )
2025-12-23 21:49:05 +01:00
Xuan-Son Nguyen and GitHub
849d021104
server: fix crash with model not having BOS/EOS ( #18321 )
2025-12-23 14:39:36 +01:00
Xuan-Son Nguyen and GitHub
179fd82a72
gen-docs: automatically update markdown file ( #18294 )
...
* gen-docs: automatically update markdown file
* also strip whitespace
* do not add extra newline
* update TOC
2025-12-22 19:30:19 +01:00
Xuan-Son Nguyen and GitHub
6ce863c803
server: prevent data race from HTTP threads ( #18263 )
...
* server: prevent data race from HTTP threads
* fix params
* fix default_generation_settings
* nits: make handle_completions_impl looks less strange
* stricter const
* fix GGML_ASSERT(idx < states.size())
* move index to be managed by server_response_reader
* http: make sure req & res lifecycle are tied together
* fix compile
* fix index handling buggy
* fix data race for lora endpoint
* nits: fix shadow variable
* nits: revert redundant changes
* nits: correct naming for json_webui_settings
2025-12-22 14:23:34 +01:00
Xuan-Son Nguyen and GitHub
3997c78e33
server: fix data race in to_json_anthropic ( #18283 )
2025-12-22 13:21:43 +01:00
Xuan-Son Nguyen and GitHub
86af848153
server: (docs) remove mention about extra_args ( #18262 )
2025-12-22 12:22:01 +01:00
Xuan-Son Nguyen and GitHub
5e25ddebff
move copilot instructions to AGENTS.md ( #18259 )
...
* move copilot --> agents.md
* agents: add disclose AI usage
* refine
2025-12-21 19:09:21 +01:00
Xuan-Son Nguyen and GitHub
ddcb75dd8a
server: add auto-sleep after N seconds of idle ( #18228 )
...
* implement sleeping at queue level
* implement server-context suspend
* add test
* add docs
* optimization: add fast path
* make sure to free llama_init
* nits
* fix use-after-free
* allow /models to be accessed during sleeping, fix use-after-free
* don't allow accessing /models during sleep, it is not thread-safe
* fix data race on accessing props and model_meta
* small clean up
* trailing whitespace
* rm outdated comments
2025-12-21 02:24:42 +01:00
9e39a1e6a9
server: support load model on startup, support preset-only options ( #18206 )
...
* server: support autoload model, support preset-only options
* add docs
* load-on-startup
* fix
* Update common/arg.cpp
Co-authored-by: Pascal <admin@serveurperso.com >
---------
Co-authored-by: Pascal <admin@serveurperso.com >
2025-12-20 09:25:27 +01:00
Xuan-Son Nguyen and GitHub
98c1c7a7bf
presets: refactor, allow cascade presets from different sources, add global section ( #18169 )
...
* presets: refactor, allow cascade presets from different sources
* update docs
* fix neg arg handling
* fix empty mmproj
* also filter out server-controlled args before to_ini()
* skip loading custom_models if not specified
* fix unset_reserved_args
* fix crash on windows
2025-12-19 12:08:20 +01:00
8ea958d4d9
model : add ASR support for LFM2-Audio-1.5B (conformer) ( #18106 )
...
* ASR with LFM2-Audio-1.5B
* Set rope_theta
* Fix comment
* Remove rope_theta setting
* Address PR feedback
* rename functions to conformer
* remove some redundant ggml_cont
* fix missing tensor
* add prefix "a." for conv tensors
* remove redundant reshape
* clean up
* add test model
---------
Co-authored-by: Tarek Dakhran <tarek@liquid.ai >
2025-12-19 00:18:01 +01:00
Xuan-Son Nguyen and GitHub
4d1316c440
arg: fix ASAN error on sampler_type_names empty ( #18167 )
2025-12-18 14:30:32 +01:00
Xuan-Son Nguyen and GitHub
e85e9d7637
server: (router) disable SSL on child process ( #18141 )
2025-12-17 21:39:08 +01:00
Xuan-Son Nguyen and GitHub
bde461de8c
server: (router) allow child process to report status via stdout ( #18110 )
...
* server: (router) allow child process to report status via stdout
* apply suggestions
2025-12-17 14:54:11 +01:00
Xuan-Son Nguyen and GitHub
ef83fb8601
model: fix LFM2 missing tensors ( #18105 )
2025-12-16 19:07:43 +01:00
Xuan-Son Nguyen and GitHub
7f2b2f3c77
arch: refactor LLM_TENSOR_NAMES ( #18051 )
...
* arch: refactor LLM_TENSOR_NAMES
* update docs
* typo
* fix LLM_ARCH_NEMOTRON_H_MOE
* show more meaningful error message on missing tensor
* fix and tested LLM_ARCH_NEMOTRON_H_MOE
2025-12-16 13:22:30 +01:00
Xuan-Son Nguyen and GitHub
7b1db3d3b7
arg: clarify auto kvu/np being set on server ( #17997 )
...
* arg: clarify auto kvu/np being set on server
* improve docs
* use invalid_argument
2025-12-16 12:01:27 +01:00
3d86c6c2b5
model: support GLM4V vision encoder ( #18042 )
...
* convert ok
* no deepstack
* less new tensors
* cgraph ok
* add mrope for text model
* faster patch merger
* add GGML_ROPE_TYPE_MRNORM
* add support for metal
* move glm4v do dedicated graph
* convert: add norm_embd
* clip: add debugging fn
* working correctly
* fix style
* use bicubic
* fix mrope metal
* improve cpu
* convert to neox ordering on conversion
* revert backend changes
* force stop if using old weight
* support moe variant
* fix conversion
* fix convert (2)
* Update tools/mtmd/clip-graph.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* process mrope_section on TextModel base class
* resolve conflict merge
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-16 11:25:26 +01:00
96a181a933
mtmd: refactor audio preprocessing ( #17978 )
...
* mtmd: refactor audio preprocessing
* refactor
Co-authored-by: Tarek <tdakhran@users.noreply.github.com >
* wip
* wip (2)
* improve constructor
* fix use_natural_log
* fix padding for short input
* clean up
* remove need_chunking
---------
Co-authored-by: Tarek <tdakhran@users.noreply.github.com >
2025-12-15 14:16:52 +01:00
Xuan-Son Nguyen and GitHub
52392291b2
preset: handle negated arg, reverse the meaning if needed ( #18041 )
2025-12-14 22:08:10 +01:00
Xuan-Son Nguyen and GitHub
0759b09c90
graph: add f_attn_temp_offset ( #18025 )
2025-12-14 13:05:59 +01:00
Xuan-Son Nguyen and GitHub
c00ff929dc
scripts: add script to compare logprobs of llama.cpp against other frameworks ( #17947 )
...
* scripts: add script to compare logits of llama.cpp against other frameworks
* accept custom prompt file
* fix code style
* clarify endpoint
* fix displaying
* use abs for diff
* fix vllm case
* rm output file
* rename to compare-logprobs
* add "pattern"
2025-12-13 22:33:29 +01:00
Xuan-Son Nguyen and GitHub
4d5ae24c0a
arg: fix common_params_parse not accepting negated arg ( #17991 )
2025-12-13 12:53:37 +01:00
380b4c984e
common: support negated args ( #17919 )
...
* args: support negated args
* update docs
* fix typo
* add more neg options
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* rm duplicated arg
* fix LLAMA_ARG_NO_HOST
* add test
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-12 23:58:53 +01:00
Xuan-Son Nguyen and GitHub
e39a2ce66d
clip: move model cgraphs into their own files ( #17965 )
...
* clip: move model cgraphs into their own files
* more explicit enums
* fix linux build
* fix naming
* missing headers
* nits: add comments for contributors
2025-12-12 21:14:48 +01:00
Xuan-Son Nguyen and GitHub
17158965ac
mtmd: explicitly forbidden inclusion of private header and libcommon ( #17946 )
2025-12-12 15:16:06 +01:00
Xuan-Son Nguyen and GitHub
54a0fee4b7
arg: add -mm and -mmu as short form of --mmproj and --mmproj-url ( #17958 )
...
* arg: add -mm and -mmu as short form of --mmproj and --mmproj-url
* correct order
* update docs
2025-12-12 14:06:06 +01:00
Xuan-Son Nguyen and GitHub
c6b2c9310c
mtmd: some small clean up ( #17909 )
...
* clip: add support for fused qkv in build_vit
* use bulid_ffn whenever possible
* fix internvl
* mtmd-cli: move image to beginning
* test script: support custom args
2025-12-10 22:20:06 +01:00
34a6d86982
cli: enable jinja by default ( #17911 )
...
* cli: enable jinja by default
* Update common/arg.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-10 22:19:42 +01:00
6c2131773c
cli: new CLI experience ( #17824 )
...
* wip
* wip
* fix logging, add display info
* handle commands
* add args
* wip
* move old cli to llama-completion
* rm deprecation notice
* move server to a shared library
* move ci to llama-completion
* add loading animation
* add --show-timings arg
* add /read command, improve LOG_ERR
* add args for speculative decoding, enable show timings by default
* add arg --image and --audio
* fix windows build
* support reasoning_content
* fix llama2c workflow
* color default is auto
* fix merge conflicts
* properly fix color problem
Co-authored-by: bandoti <bandoti@users.noreply.github.com >
* better loading spinner
* make sure to clean color on force-exit
* also clear input files on "/clear"
* simplify common_log_flush
* add warning in mtmd-cli
* implement console writter
* fix data race
* add attribute
* fix llama-completion and mtmd-cli
* add some notes about console::log
* fix compilation
---------
Co-authored-by: bandoti <bandoti@users.noreply.github.com >
2025-12-10 15:28:59 +01:00
9e79b0116e
convert: allow using quantized Mistral weight ( #17889 )
...
* convert: allow using quantized Mistral weight
* data_torch.ndim
* update dequant fn
Co-authored-by: compilade <compilade@users.noreply.github.com >
---------
Co-authored-by: compilade <compilade@users.noreply.github.com >
2025-12-10 10:26:22 +01:00
4e842d5120
console: allow using arrow left/right, home/end keys and history mode ( #17836 )
...
* console: allow using arrow left/right to edit the line (with UTF-8 support)
* console: fix arrow keys on Windows using private-use Unicode
* console: add Home/End key support for Windows and Linux
* console: add basic Up/Down history navigation
* fix build
* console: allow using arrow left/right to edit the line (with UTF-8 support)
* console: fix arrow keys on Windows using private-use Unicode
* console: add Home/End key support for Windows and Linux
* console: add basic Up/Down history navigation
* console: remove unreachable wc == 0 check after VK switch
* console: add Ctrl+Left/Right word navigation
- Add KEY_CTRL_ARROW_LEFT and KEY_CTRL_ARROW_RIGHT codes
- Windows: detect CTRL modifier via dwControlKeyState
- Linux: parse ANSI sequences with modifier (1;5D/C)
- Implement move_word_left/right with space-skipping logic
- Refactor escape sequence parsing to accumulate params
* console: add Delete key support
- Windows: VK_DELETE detection
- Linux: ESC[3~ sequence parsing
- Forward character deletion with UTF-8 support
* console: implement bash-style history editing
- Edit any history line during UP/DOWN navigation, edits persist
- Pressing Enter appends edited version as new history entry
- Original line stay untouched in their positions
* clean up
* better history impl
* fix decode_utf8
---------
Co-authored-by: Pascal <admin@serveurperso.com >
2025-12-09 11:53:59 +01:00
Xuan-Son Nguyen and GitHub
951520ddb0
server: delegate result_state creation to server_task ( #17835 )
...
* server: delegate result_state creation to server_task
* remove unued states
* add more docs
2025-12-08 17:04:38 +01:00
f896d2c34f
server: improve speed of speculative decoding ( #17808 )
...
* server: improve speed of speculative decoding
* fix small draft case
* add link to the PR
* server : fix generation time measurement
* server : fix draft acceptance logs (add SRV_CNT, SLT_CNT macros)
* server : add comment
* add PR to docs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-08 14:35:28 +01:00
Xuan-Son Nguyen and GitHub
37a4f63244
server : add development documentation ( #17760 )
...
* first draft
* rewrite
* update & remove duplicated sections
2025-12-08 13:54:58 +01:00
Xuan-Son Nguyen and GitHub
4d3726278b
model: add llama 4 scaling for mistral-large (deepseek arch) ( #17744 )
2025-12-07 22:29:54 +01:00
Xuan-Son Nguyen and GitHub
c42712b056
server: support multiple generations from one prompt (OAI "n" option) ( #17775 )
...
* backend support
* server: support multiple generations from one prompt (OAI "n" option)
* fix invalid batch
* format oai
* clean up
* disable ctx shift
* add test
* update comments
* fix style
* add n_cmpl to docs [no ci]
* allowing using both n_cmpl and n
2025-12-06 15:54:38 +01:00
dbc15a7967
convert: support Mistral 3 Large MoE ( #17730 )
...
* convert: support Mistral 3 Large MoE
* filter out vision tensors, add missing keys
* handle vocab
* add temperature_length
* fix mscale_all_dim
* clean up
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* fix
* Update gguf-py/gguf/tensor_mapping.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-06 10:49:33 +01:00
Xuan-Son Nguyen and GitHub
9d0229967a
server: strip content-length header on proxy ( #17734 )
2025-12-04 16:32:57 +01:00
c4c10bfb86
server: move msg diffs tracking to HTTP thread ( #17740 )
...
* server: move msg diffs tracking to HTTP thread
* wip
* tool call tests ok
* minor : style
* cont : fix
* move states to server_response_reader
* add safe-guard
* fix
* fix 2
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-04 15:46:08 +01:00
Xuan-Son Nguyen and GitHub
13628d8bdb
server: add --media-path for local media files ( #17697 )
...
* server: add --media-path for local media files
* remove unused fn
2025-12-02 22:49:20 +01:00
Xuan-Son Nguyen and GitHub
a96283adc4
mtmd: fix --no-warmup ( #17695 )
2025-12-02 22:48:08 +01:00
Xuan-Son Nguyen and GitHub
2c453c6c77
convert: add error message for mistral3 quantized weight ( #17686 )
2025-12-02 11:48:31 +01:00
Xuan-Son Nguyen and GitHub
5d6bd842ea
server: remove default "gpt-3.5-turbo" model name ( #17668 )
...
* server: remove default "gpt-3.5-turbo" model name
* do not reflect back model name from request
* fix test
2025-12-02 11:38:57 +01:00
Xuan-Son Nguyen and GitHub
682e6658bb
server: explicitly set exec path when create new instance ( #17669 )
...
* Revert "rm unused fn"
This reverts commit f2dbe9c087ac8776cd41f747a8b0ddf448c8399a.
* server: explicitly set exec path when create new instance
* put back TODO
* only call get_server_exec_path() once
* add fallback logic
2025-12-02 10:25:11 +01:00
Xuan-Son Nguyen and GitHub
ecf74a8417
mtmd: add mtmd_context_params::warmup option ( #17652 )
...
* mtmd: add mtmd_context_params::warmup option
* reuse the common_params::warmup
2025-12-01 21:32:25 +01:00
ec18edfcba
server: introduce API for serving / loading / unloading multiple models ( #17470 )
...
* server: add model management and proxy
* fix compile error
* does this fix windows?
* fix windows build
* use subprocess.h, better logging
* add test
* fix windows
* feat: Model/Router server architecture WIP
* more stable
* fix unsafe pointer
* also allow terminate loading model
* add is_active()
* refactor: Architecture improvements
* tmp apply upstream fix
* address most problems
* address thread safety issue
* address review comment
* add docs (first version)
* address review comment
* feat: Improved UX for model information, modality interactions etc
* chore: update webui build output
* refactor: Use only the message data `model` property for displaying model used info
* chore: update webui build output
* add --models-dir param
* feat: New Model Selection UX WIP
* chore: update webui build output
* feat: Add auto-mic setting
* feat: Attachments UX improvements
* implement LRU
* remove default model path
* better --models-dir
* add env for args
* address review comments
* fix compile
* refactor: Chat Form Submit component
* ad endpoint docs
* Merge remote-tracking branch 'webui/allozaur/server_model_management_v1_2' into xsn/server_model_maagement_v1_2
Co-authored-by: Aleksander <aleksander.grygier@gmail.com >
* feat: Add copy to clipboard to model name in model info dialog
* feat: Model unavailable UI state for model selector
* feat: Chat Form Actions UI logic improvements
* feat: Auto-select model from last assistant response
* chore: update webui build output
* expose args and exit_code in API
* add note
* support extra_args on loading model
* allow reusing args if auto_load
* typo docs
* oai-compat /models endpoint
* cleaner
* address review comments
* feat: Use `model` property for displaying the `repo/model-name` naming format
* refactor: Attachments data
* chore: update webui build output
* refactor: Enum imports
* feat: Improve Model Selector responsiveness
* chore: update webui build output
* refactor: Cleanup
* refactor: Cleanup
* refactor: Formatters
* chore: update webui build output
* refactor: Copy To Clipboard Icon component
* chore: update webui build output
* refactor: Cleanup
* chore: update webui build output
* refactor: UI badges
* chore: update webui build output
* refactor: Cleanup
* refactor: Cleanup
* chore: update webui build output
* add --models-allow-extra-args for security
* nits
* add stdin_file
* fix merge
* fix: Retrieve lost setting after resolving merge conflict
* refactor: DatabaseStore -> DatabaseService
* refactor: Database, Conversations & Chat services + stores architecture improvements (WIP)
* refactor: Remove redundant settings
* refactor: Multi-model business logic WIP
* chore: update webui build output
* feat: Switching models logic for ChatForm or when regenerating messges + modality detection logic
* chore: update webui build output
* fix: Add `untrack` inside chat processing info data logic to prevent infinite effect
* fix: Regenerate
* feat: Remove redundant settigns + rearrange
* fix: Audio attachments
* refactor: Icons
* chore: update webui build output
* feat: Model management and selection features WIP
* chore: update webui build output
* refactor: Improve server properties management
* refactor: Icons
* chore: update webui build output
* feat: Improve model loading/unloading status updates
* chore: update webui build output
* refactor: Improve API header management via utility functions
* remove support for extra args
* set hf_repo/docker_repo as model alias when posible
* refactor: Remove ConversationsService
* refactor: Chat requests abort handling
* refactor: Server store
* tmp webui build
* refactor: Model modality handling
* chore: update webui build output
* refactor: Processing state reactivity
* fix: UI
* refactor: Services/Stores syntax + logic improvements
Refactors components to access stores directly instead of using exported getter functions.
This change centralizes store access and logic, simplifying component code and improving maintainability by reducing the number of exported functions and promoting direct store interaction.
Removes exported getter functions from `chat.svelte.ts`, `conversations.svelte.ts`, `models.svelte.ts` and `settings.svelte.ts`.
* refactor: Architecture cleanup
* feat: Improve statistic badges
* feat: Condition available models based on modality + better model loading strategy & UX
* docs: Architecture documentation
* feat: Update logic for PDF as Image
* add TODO for http client
* refactor: Enhance model info and attachment handling
* chore: update webui build output
* refactor: Components naming
* chore: update webui build output
* refactor: Cleanup
* refactor: DRY `getAttachmentDisplayItems` function + fix UI
* chore: update webui build output
* fix: Modality detection improvement for text-based PDF attachments
* refactor: Cleanup
* docs: Add info comment
* refactor: Cleanup
* re
* refactor: Cleanup
* refactor: Cleanup
* feat: Attachment logic & UI improvements
* refactor: Constants
* feat: Improve UI sidebar background color
* chore: update webui build output
* refactor: Utils imports + move types to `app.d.ts`
* test: Fix Storybook mocks
* chore: update webui build output
* test: Update Chat Form UI tests
* refactor: Tooltip Provider from core layout
* refactor: Tests to separate location
* decouple server_models from server_routes
* test: Move demo test to tests/server
* refactor: Remove redundant method
* chore: update webui build output
* also route anthropic endpoints
* fix duplicated arg
* fix invalid ptr to shutdown_handler
* server : minor
* rm unused fn
* add ?autoload=true|false query param
* refactor: Remove redundant code
* docs: Update README documentations + architecture & data flow diagrams
* fix: Disable autoload on calling server props for the model
* chore: update webui build output
* fix ubuntu build
* fix: Model status reactivity
* fix: Modality detection for MODEL mode
* chore: update webui build output
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-01 19:41:04 +01:00
Xuan-Son Nguyen and GitHub
7733409734
common: improve verbosity level definitions ( #17630 )
...
* common: improve verbosity level definitions
* string_format
* update autogen docs
2025-12-01 14:38:13 +01:00
cd3c118908
model: support Ministral3 ( #17644 )
...
* conversion script
* support ministral 3
* maybe this is better?
* add TODO for rope_yarn_log_mul
* better ppl (tested on 14B-Instruct)
* Add Ministral3 support to Mistral format
* improve arch handling
* add sizes
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* nits
---------
Co-authored-by: Julien Denize <julien.denize@mistral.ai >
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-01 12:26:52 +01:00
Xuan-Son Nguyen and GitHub
ff90508d68
contributing: update guidelines for AI-generated code ( #17625 )
...
* contributing: update guidelines for AI-generated code
* revise
2025-11-30 22:51:34 +01:00
Xuan-Son Nguyen and GitHub
7f8ef50cce
clip: fix nb calculation for qwen3-vl ( #17594 )
2025-11-30 15:33:55 +01:00
Xuan-Son Nguyen and GitHub
3c136b21a3
cli: add migration warning ( #17620 )
2025-11-30 15:32:43 +01:00
ab49f094d2
server: move server-context to its own cpp|h ( #17595 )
...
* git mv
* add server-context.h
* add server-context.h
* clean up headers
* cont : cleanup
* also expose server_response_reader (to be used by CLI)
* fix windows build
* decouple server_routes and server_http
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-29 22:04:44 +01:00
Xuan-Son Nguyen and GitHub
e509411cf1
server: enable jinja by default, update docs ( #17524 )
...
* server: enable jinja by default, update docs
* fix tests
2025-11-27 01:02:50 +01:00
b8372eecd9
server: split server.cpp code into server/common/task/queue ( #17362 )
...
* add server-task, server-common
* add server-queue
* rm redundant includes
* move enum stop_type to server-task
* server : headers cleanup
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-24 14:41:53 +01:00
Xuan-Son Nguyen and GitHub
054a45c3d3
grammar: fix regression caused by #17381 ( #17412 )
...
* grammar: fix regression caused by #17381
* more readable
2025-11-20 18:35:10 +01:00
10e9780154
chat: fix int overflow, prevent size calculation in float/double ( #17357 )
...
* chat: fix int overflow, prevent size calculation in float/double
* Update common/chat.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-18 19:11:53 +01:00
0de8878c96
server: split HTTP into its own interface ( #17216 )
...
* server: split HTTP into its own interface
* move server-http and httplib to its own file
* add the remaining endpoints
* fix exception/error handling
* renaming
* missing header
* fix missing windows header
* fix error responses from http layer
* fix slot save/restore handler
* fix case where only one stream chunk is returned
* add NOMINMAX
* do not call sink.write on empty data
* use safe_json_to_str for SSE
* clean up
* add some comments
* improve usage of next()
* bring back the "server is listening on" message
* more generic handler
* add req.headers
* move the chat template print to init()
* add req.path
* cont : minor
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-17 22:05:44 +01:00
Xuan-Son Nguyen and GitHub
9b17d74ab7
mtmd: add mtmd_log_set ( #17268 )
2025-11-14 15:56:19 +01:00
Xuan-Son Nguyen and GitHub
c4abcb2457
server: fixing naming conflict res_error ( #17243 )
2025-11-13 20:53:47 +01:00
Xuan-Son Nguyen and GitHub
00c94083b3
server: (refactor) implement generator-based API for task results ( #17174 )
...
* server: (refactor) implement generator-based API for task results
* improve
* moving some code
* fix "Response ended prematurely"
* add sink.done before return false
* rm redundant check
* rm unused var
* rename generator --> reader
2025-11-12 18:50:52 +01:00
Xuan-Son Nguyen and GitHub
017eceed61
ci: add check vendor job ( #17179 )
...
* ci: add check vendor job
* use dev version of miniaudio
* move to dedicated workflow, only run on related files changed
2025-11-12 14:56:02 +01:00
Xuan-Son Nguyen and GitHub
ee8dd5c658
server: move res_error/res_ok to static function ( #17167 )
2025-11-12 14:17:24 +01:00
Xuan-Son Nguyen and GitHub
1d45b4228f
vendor: split httplib to cpp/h files ( #17150 )
...
* vendor: split httplib to cpp/h files
* move defines
* include httplib if curl is not used
* add TODO
* fix build ios
* fix build visionos instead
2025-11-11 13:32:58 +01:00
Xuan-Son Nguyen and GitHub
4b13a684c5
mtmd: fix patch_size initialized to random value in audio models ( #17128 )
...
* mtmd: fix patch_size initialized to random value in audio models
* add default hparams
2025-11-10 11:41:05 +01:00
aa3b7a90b4
arg: add --cache-list argument to list cached models ( #17073 )
...
* arg: add --cache-list argument to list cached models
* new manifest naming format
* improve naming
* Update common/arg.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-08 21:54:14 +01:00
5c9a18e674
common: move download functions to download.(cpp|h) ( #17059 )
...
* common: move download functions to download.(cpp|h)
* rm unused includes
* minor cleanup
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-07 11:23:34 +01:00
Xuan-Son Nguyen and GitHub
4882f0ff78
clip: implement minicpm-v sinusoidal embd using GGML ( #17036 )
...
* clip: implement minicpm-v sinusoidal embd using GGML
* fix repeat op
2025-11-06 11:02:54 +01:00
Xuan-Son Nguyen and GitHub
92bb84f775
mtmd: allow QwenVL to process larger image by default ( #17020 )
2025-11-05 14:26:49 +01:00
Xuan-Son Nguyen and GitHub
2f0c2db43e
mtmd: improve struct initialization ( #16981 )
2025-11-05 11:26:37 +01:00
Xuan-Son Nguyen and GitHub
070ff4d535
mtmd: add --image-min/max-tokens ( #16921 )
2025-11-03 11:11:18 +01:00
Xuan-Son Nguyen and GitHub
bf7b0c9725
mtmd: pad mask for qwen2.5vl ( #16954 )
...
* mtmd: pad mask for qwen2.5vl
* improve
2025-11-03 10:25:55 +01:00
Xuan-Son Nguyen and GitHub
cf659bbb8e
mtmd: refactor preprocessing + support max/min pixels ( #16878 )
...
* mtmd: refactor preprocessing + support max/min pixels
* fix mlp type
* implement mix/max pixels
* improve hparams
* better image preproc for qwen
* fix
* fix out of bound composite
* fix (2)
* fix token calculation
* get_merge_kernel_size()
* fix llama4 and lfm2
* gonna fix them all
* use simple resize for qwen
* qwen: increase min tokens
* no resize if dst size == src size
* restore to initial min/max tokens value for qwen
2025-11-01 15:51:36 +01:00
Xuan-Son Nguyen and GitHub
3464bdac37
llama: fix ASAN error with M-RoPE ( #16848 )
2025-10-29 20:11:39 +01:00
e3af5563bd
llama: store mrope data in KV cell ( #16825 )
...
* llama: store mrope data in KV cell
* correct x,y ordering
* address review comments
* add consistency checks
* Update src/llama-kv-cache.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* add TODO
* fix asan error
* kv-cells : improve ext handling
* cont : fix headers
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-10-29 18:09:18 +01:00
Xuan-Son Nguyen and GitHub
e1ab084803
mtmd : fix idefics3 preprocessing ( #16806 )
...
* mtmd : fix idefics3 preprocessing
* disable granite test
* fix test for granite
2025-10-27 23:12:16 +01:00
Xuan-Son Nguyen and GitHub
c55d53acec
model : add LightOnOCR-1B model ( #16764 )
...
* model : add LightOnOCR-1B model
* add test
2025-10-27 16:02:58 +01:00
Xuan-Son Nguyen and GitHub
d0660f237a
mtmd-cli : allow using --jinja ( #16718 )
...
* mtmd-cli : allow using --jinja
* support -sys
* implement chat_history
* fix clear memory
* rm -sys support, added TODO
2025-10-23 15:00:49 +02:00
Xuan-Son Nguyen and GitHub
1bb4f43380
mtmd : support home-cooked Mistral Small Omni ( #14928 )
2025-10-16 19:00:31 +02:00
3e3cb19f64
llama-quant: add support for mmproj ( #16592 )
...
* llama-quant: add support for mmproj
* Update src/llama.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* check prefix instead
* small fix
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-10-15 14:48:08 +02:00
Xuan-Son Nguyen and GitHub
05a2458121
codeowners : update ownership for @ngxson and @allozuar ( #16128 )
2025-09-22 11:10:58 +03:00
Xuan-Son Nguyen and GitHub
4b8560ab56
chat : fix build on arm64 ( #16101 )
2025-09-19 13:02:51 +07:00
Xuan-Son Nguyen and GitHub
0dd58b6877
ggml : refactor forward_dup for cpu backend ( #16062 )
...
* ggml : refactor forward_dup for cpu backend
* clean up a bit
* add quant/dequant perf test
2025-09-19 06:31:56 +02:00
Xuan-Son Nguyen and GitHub
8f8f2274ee
convert : add Llama4ForCausalLM ( #16042 )
...
* convert : add Llama4ForCausalLM
* handle swa
* half working version
* fix use_kq_norm
* fix use_kq_norm
2025-09-17 19:18:21 +02:00
Xuan-Son Nguyen and GitHub
56920f5665
server : bring back timings_per_token ( #15879 )
2025-09-08 16:50:05 +02:00
Xuan-Son Nguyen and GitHub
9fcb29f22f
ggml: allow casting between f32 and i32 ( #15783 )
...
* ggml: allow casting between f32 and i32
* fix cuda
* add vulkan
* fix CPU non-cont
* add non-cont test case
* add note
* extend test number range
* correct note
* add cont version for vulkan
2025-09-08 12:33:01 +02:00
Xuan-Son Nguyen and GitHub
3c3635d2f2
server : speed up tests ( #15836 )
...
* server : speed up tests
* clean up
* restore timeout_seconds in some places
* flake8
* explicit offline
2025-09-06 14:45:24 +02:00
61bdfd5298
server : implement prompt processing progress report in stream mode ( #15827 )
...
* server : implement `return_progress`
* add timings.cache_n
* add progress.time_ms
* add test
* fix test for chat/completions
* readme: add docs on timings
* use ggml_time_us
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-09-06 13:35:04 +02:00
Xuan-Son Nguyen and GitHub
a68d914426
server: add exceed_context_size_error type ( #15780 )
...
* server: add exceed_context_size_error type
* change error code to 400
2025-09-04 11:50:23 +02:00
Xuan-Son Nguyen and GitHub
79a546220c
mtmd : support Kimi VL model ( #15458 )
...
* convert : fix tensor naming conflict for llama 4 vision
* convert ok
* support kimi vision model
* clean up
* fix style
* fix calc number of output tokens
* refactor resize_position_embeddings
* add test case
* rename build fn
* correct a small bug
2025-08-26 12:54:19 +02:00
Xuan-Son Nguyen and GitHub
945e1f12a6
ggml : fix condition of im2col on Metal backend ( #15460 )
2025-08-21 08:32:26 +03:00
Xuan-Son Nguyen and GitHub
e9288e8869
chat : clarify the meaning of reasoning_format ( #15408 )
...
* chat : clarify the meaning of reasoning_format
* add link to this PR
2025-08-19 10:29:36 +02:00
Xuan-Son Nguyen and GitHub
f08c4c0d8d
mtmd : clean up clip_n_output_tokens ( #15391 )
2025-08-18 22:53:52 +02:00
Xuan-Son Nguyen and GitHub
fba5c0d680
chat : hotfix gpt-oss jinja raising an exception ( #15243 )
...
* chat : hotfix gpt-oss jinja raising an exception
* fix
2025-08-11 15:31:35 +02:00
Xuan-Son Nguyen and GitHub
53d0a12658
server : allow specifying reasoning_format in HTTP request ( #15238 )
2025-08-11 14:48:41 +02:00
Xuan-Son Nguyen and GitHub
50aa938901
convert : support non-mxfp4 HF model ( #15153 )
...
* convert : support non-mxfp4 HF model
* rm redundant check
* disable debug check
2025-08-07 23:26:03 +02:00
00fa15fedc
mtmd : add support for Voxtral ( #14862 )
...
* mtmd : add support for Voxtral
* clean up
* fix python requirements
* add [BEGIN_AUDIO] token
* also support Devstral conversion
* add docs and tests
* fix regression for ultravox
* minor coding style improvement
* correct project activation fn
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-07-28 15:01:48 +02:00
Xuan-Son Nguyen and GitHub
820de57d4f
chat : fix kimi-k2 chat template ( #14852 )
2025-07-24 13:59:56 +02:00
