380b4c984e
common: support negated args ( #17919 )
...
* args: support negated args
* update docs
* fix typo
* add more neg options
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* rm duplicated arg
* fix LLAMA_ARG_NO_HOST
* add test
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-12 23:58:53 +01:00
Xuan-Son Nguyen and GitHub
e39a2ce66d
clip: move model cgraphs into their own files ( #17965 )
...
* clip: move model cgraphs into their own files
* more explicit enums
* fix linux build
* fix naming
* missing headers
* nits: add comments for contributors
2025-12-12 21:14:48 +01:00
Xuan-Son Nguyen and GitHub
17158965ac
mtmd: explicitly forbidden inclusion of private header and libcommon ( #17946 )
2025-12-12 15:16:06 +01:00
Xuan-Son Nguyen and GitHub
54a0fee4b7
arg: add -mm and -mmu as short form of --mmproj and --mmproj-url ( #17958 )
...
* arg: add -mm and -mmu as short form of --mmproj and --mmproj-url
* correct order
* update docs
2025-12-12 14:06:06 +01:00
Xuan-Son Nguyen and GitHub
c6b2c9310c
mtmd: some small clean up ( #17909 )
...
* clip: add support for fused qkv in build_vit
* use bulid_ffn whenever possible
* fix internvl
* mtmd-cli: move image to beginning
* test script: support custom args
2025-12-10 22:20:06 +01:00
34a6d86982
cli: enable jinja by default ( #17911 )
...
* cli: enable jinja by default
* Update common/arg.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-10 22:19:42 +01:00
6c2131773c
cli: new CLI experience ( #17824 )
...
* wip
* wip
* fix logging, add display info
* handle commands
* add args
* wip
* move old cli to llama-completion
* rm deprecation notice
* move server to a shared library
* move ci to llama-completion
* add loading animation
* add --show-timings arg
* add /read command, improve LOG_ERR
* add args for speculative decoding, enable show timings by default
* add arg --image and --audio
* fix windows build
* support reasoning_content
* fix llama2c workflow
* color default is auto
* fix merge conflicts
* properly fix color problem
Co-authored-by: bandoti <bandoti@users.noreply.github.com >
* better loading spinner
* make sure to clean color on force-exit
* also clear input files on "/clear"
* simplify common_log_flush
* add warning in mtmd-cli
* implement console writter
* fix data race
* add attribute
* fix llama-completion and mtmd-cli
* add some notes about console::log
* fix compilation
---------
Co-authored-by: bandoti <bandoti@users.noreply.github.com >
2025-12-10 15:28:59 +01:00
9e79b0116e
convert: allow using quantized Mistral weight ( #17889 )
...
* convert: allow using quantized Mistral weight
* data_torch.ndim
* update dequant fn
Co-authored-by: compilade <compilade@users.noreply.github.com >
---------
Co-authored-by: compilade <compilade@users.noreply.github.com >
2025-12-10 10:26:22 +01:00
4e842d5120
console: allow using arrow left/right, home/end keys and history mode ( #17836 )
...
* console: allow using arrow left/right to edit the line (with UTF-8 support)
* console: fix arrow keys on Windows using private-use Unicode
* console: add Home/End key support for Windows and Linux
* console: add basic Up/Down history navigation
* fix build
* console: allow using arrow left/right to edit the line (with UTF-8 support)
* console: fix arrow keys on Windows using private-use Unicode
* console: add Home/End key support for Windows and Linux
* console: add basic Up/Down history navigation
* console: remove unreachable wc == 0 check after VK switch
* console: add Ctrl+Left/Right word navigation
- Add KEY_CTRL_ARROW_LEFT and KEY_CTRL_ARROW_RIGHT codes
- Windows: detect CTRL modifier via dwControlKeyState
- Linux: parse ANSI sequences with modifier (1;5D/C)
- Implement move_word_left/right with space-skipping logic
- Refactor escape sequence parsing to accumulate params
* console: add Delete key support
- Windows: VK_DELETE detection
- Linux: ESC[3~ sequence parsing
- Forward character deletion with UTF-8 support
* console: implement bash-style history editing
- Edit any history line during UP/DOWN navigation, edits persist
- Pressing Enter appends edited version as new history entry
- Original line stay untouched in their positions
* clean up
* better history impl
* fix decode_utf8
---------
Co-authored-by: Pascal <admin@serveurperso.com >
2025-12-09 11:53:59 +01:00
Xuan-Son Nguyen and GitHub
951520ddb0
server: delegate result_state creation to server_task ( #17835 )
...
* server: delegate result_state creation to server_task
* remove unued states
* add more docs
2025-12-08 17:04:38 +01:00
f896d2c34f
server: improve speed of speculative decoding ( #17808 )
...
* server: improve speed of speculative decoding
* fix small draft case
* add link to the PR
* server : fix generation time measurement
* server : fix draft acceptance logs (add SRV_CNT, SLT_CNT macros)
* server : add comment
* add PR to docs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-08 14:35:28 +01:00
Xuan-Son Nguyen and GitHub
37a4f63244
server : add development documentation ( #17760 )
...
* first draft
* rewrite
* update & remove duplicated sections
2025-12-08 13:54:58 +01:00
Xuan-Son Nguyen and GitHub
4d3726278b
model: add llama 4 scaling for mistral-large (deepseek arch) ( #17744 )
2025-12-07 22:29:54 +01:00
Xuan-Son Nguyen and GitHub
c42712b056
server: support multiple generations from one prompt (OAI "n" option) ( #17775 )
...
* backend support
* server: support multiple generations from one prompt (OAI "n" option)
* fix invalid batch
* format oai
* clean up
* disable ctx shift
* add test
* update comments
* fix style
* add n_cmpl to docs [no ci]
* allowing using both n_cmpl and n
2025-12-06 15:54:38 +01:00
dbc15a7967
convert: support Mistral 3 Large MoE ( #17730 )
...
* convert: support Mistral 3 Large MoE
* filter out vision tensors, add missing keys
* handle vocab
* add temperature_length
* fix mscale_all_dim
* clean up
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* fix
* Update gguf-py/gguf/tensor_mapping.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-06 10:49:33 +01:00
Xuan-Son Nguyen and GitHub
9d0229967a
server: strip content-length header on proxy ( #17734 )
2025-12-04 16:32:57 +01:00
c4c10bfb86
server: move msg diffs tracking to HTTP thread ( #17740 )
...
* server: move msg diffs tracking to HTTP thread
* wip
* tool call tests ok
* minor : style
* cont : fix
* move states to server_response_reader
* add safe-guard
* fix
* fix 2
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-04 15:46:08 +01:00
Xuan-Son Nguyen and GitHub
13628d8bdb
server: add --media-path for local media files ( #17697 )
...
* server: add --media-path for local media files
* remove unused fn
2025-12-02 22:49:20 +01:00
Xuan-Son Nguyen and GitHub
a96283adc4
mtmd: fix --no-warmup ( #17695 )
2025-12-02 22:48:08 +01:00
Xuan-Son Nguyen and GitHub
2c453c6c77
convert: add error message for mistral3 quantized weight ( #17686 )
2025-12-02 11:48:31 +01:00
Xuan-Son Nguyen and GitHub
5d6bd842ea
server: remove default "gpt-3.5-turbo" model name ( #17668 )
...
* server: remove default "gpt-3.5-turbo" model name
* do not reflect back model name from request
* fix test
2025-12-02 11:38:57 +01:00
Xuan-Son Nguyen and GitHub
682e6658bb
server: explicitly set exec path when create new instance ( #17669 )
...
* Revert "rm unused fn"
This reverts commit f2dbe9c087ac8776cd41f747a8b0ddf448c8399a.
* server: explicitly set exec path when create new instance
* put back TODO
* only call get_server_exec_path() once
* add fallback logic
2025-12-02 10:25:11 +01:00
Xuan-Son Nguyen and GitHub
ecf74a8417
mtmd: add mtmd_context_params::warmup option ( #17652 )
...
* mtmd: add mtmd_context_params::warmup option
* reuse the common_params::warmup
2025-12-01 21:32:25 +01:00
ec18edfcba
server: introduce API for serving / loading / unloading multiple models ( #17470 )
...
* server: add model management and proxy
* fix compile error
* does this fix windows?
* fix windows build
* use subprocess.h, better logging
* add test
* fix windows
* feat: Model/Router server architecture WIP
* more stable
* fix unsafe pointer
* also allow terminate loading model
* add is_active()
* refactor: Architecture improvements
* tmp apply upstream fix
* address most problems
* address thread safety issue
* address review comment
* add docs (first version)
* address review comment
* feat: Improved UX for model information, modality interactions etc
* chore: update webui build output
* refactor: Use only the message data `model` property for displaying model used info
* chore: update webui build output
* add --models-dir param
* feat: New Model Selection UX WIP
* chore: update webui build output
* feat: Add auto-mic setting
* feat: Attachments UX improvements
* implement LRU
* remove default model path
* better --models-dir
* add env for args
* address review comments
* fix compile
* refactor: Chat Form Submit component
* ad endpoint docs
* Merge remote-tracking branch 'webui/allozaur/server_model_management_v1_2' into xsn/server_model_maagement_v1_2
Co-authored-by: Aleksander <aleksander.grygier@gmail.com >
* feat: Add copy to clipboard to model name in model info dialog
* feat: Model unavailable UI state for model selector
* feat: Chat Form Actions UI logic improvements
* feat: Auto-select model from last assistant response
* chore: update webui build output
* expose args and exit_code in API
* add note
* support extra_args on loading model
* allow reusing args if auto_load
* typo docs
* oai-compat /models endpoint
* cleaner
* address review comments
* feat: Use `model` property for displaying the `repo/model-name` naming format
* refactor: Attachments data
* chore: update webui build output
* refactor: Enum imports
* feat: Improve Model Selector responsiveness
* chore: update webui build output
* refactor: Cleanup
* refactor: Cleanup
* refactor: Formatters
* chore: update webui build output
* refactor: Copy To Clipboard Icon component
* chore: update webui build output
* refactor: Cleanup
* chore: update webui build output
* refactor: UI badges
* chore: update webui build output
* refactor: Cleanup
* refactor: Cleanup
* chore: update webui build output
* add --models-allow-extra-args for security
* nits
* add stdin_file
* fix merge
* fix: Retrieve lost setting after resolving merge conflict
* refactor: DatabaseStore -> DatabaseService
* refactor: Database, Conversations & Chat services + stores architecture improvements (WIP)
* refactor: Remove redundant settings
* refactor: Multi-model business logic WIP
* chore: update webui build output
* feat: Switching models logic for ChatForm or when regenerating messges + modality detection logic
* chore: update webui build output
* fix: Add `untrack` inside chat processing info data logic to prevent infinite effect
* fix: Regenerate
* feat: Remove redundant settigns + rearrange
* fix: Audio attachments
* refactor: Icons
* chore: update webui build output
* feat: Model management and selection features WIP
* chore: update webui build output
* refactor: Improve server properties management
* refactor: Icons
* chore: update webui build output
* feat: Improve model loading/unloading status updates
* chore: update webui build output
* refactor: Improve API header management via utility functions
* remove support for extra args
* set hf_repo/docker_repo as model alias when posible
* refactor: Remove ConversationsService
* refactor: Chat requests abort handling
* refactor: Server store
* tmp webui build
* refactor: Model modality handling
* chore: update webui build output
* refactor: Processing state reactivity
* fix: UI
* refactor: Services/Stores syntax + logic improvements
Refactors components to access stores directly instead of using exported getter functions.
This change centralizes store access and logic, simplifying component code and improving maintainability by reducing the number of exported functions and promoting direct store interaction.
Removes exported getter functions from `chat.svelte.ts`, `conversations.svelte.ts`, `models.svelte.ts` and `settings.svelte.ts`.
* refactor: Architecture cleanup
* feat: Improve statistic badges
* feat: Condition available models based on modality + better model loading strategy & UX
* docs: Architecture documentation
* feat: Update logic for PDF as Image
* add TODO for http client
* refactor: Enhance model info and attachment handling
* chore: update webui build output
* refactor: Components naming
* chore: update webui build output
* refactor: Cleanup
* refactor: DRY `getAttachmentDisplayItems` function + fix UI
* chore: update webui build output
* fix: Modality detection improvement for text-based PDF attachments
* refactor: Cleanup
* docs: Add info comment
* refactor: Cleanup
* re
* refactor: Cleanup
* refactor: Cleanup
* feat: Attachment logic & UI improvements
* refactor: Constants
* feat: Improve UI sidebar background color
* chore: update webui build output
* refactor: Utils imports + move types to `app.d.ts`
* test: Fix Storybook mocks
* chore: update webui build output
* test: Update Chat Form UI tests
* refactor: Tooltip Provider from core layout
* refactor: Tests to separate location
* decouple server_models from server_routes
* test: Move demo test to tests/server
* refactor: Remove redundant method
* chore: update webui build output
* also route anthropic endpoints
* fix duplicated arg
* fix invalid ptr to shutdown_handler
* server : minor
* rm unused fn
* add ?autoload=true|false query param
* refactor: Remove redundant code
* docs: Update README documentations + architecture & data flow diagrams
* fix: Disable autoload on calling server props for the model
* chore: update webui build output
* fix ubuntu build
* fix: Model status reactivity
* fix: Modality detection for MODEL mode
* chore: update webui build output
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-01 19:41:04 +01:00
Xuan-Son Nguyen and GitHub
7733409734
common: improve verbosity level definitions ( #17630 )
...
* common: improve verbosity level definitions
* string_format
* update autogen docs
2025-12-01 14:38:13 +01:00
cd3c118908
model: support Ministral3 ( #17644 )
...
* conversion script
* support ministral 3
* maybe this is better?
* add TODO for rope_yarn_log_mul
* better ppl (tested on 14B-Instruct)
* Add Ministral3 support to Mistral format
* improve arch handling
* add sizes
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* nits
---------
Co-authored-by: Julien Denize <julien.denize@mistral.ai >
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-01 12:26:52 +01:00
Xuan-Son Nguyen and GitHub
ff90508d68
contributing: update guidelines for AI-generated code ( #17625 )
...
* contributing: update guidelines for AI-generated code
* revise
2025-11-30 22:51:34 +01:00
Xuan-Son Nguyen and GitHub
7f8ef50cce
clip: fix nb calculation for qwen3-vl ( #17594 )
2025-11-30 15:33:55 +01:00
Xuan-Son Nguyen and GitHub
3c136b21a3
cli: add migration warning ( #17620 )
2025-11-30 15:32:43 +01:00
ab49f094d2
server: move server-context to its own cpp|h ( #17595 )
...
* git mv
* add server-context.h
* add server-context.h
* clean up headers
* cont : cleanup
* also expose server_response_reader (to be used by CLI)
* fix windows build
* decouple server_routes and server_http
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-29 22:04:44 +01:00
Xuan-Son Nguyen and GitHub
e509411cf1
server: enable jinja by default, update docs ( #17524 )
...
* server: enable jinja by default, update docs
* fix tests
2025-11-27 01:02:50 +01:00
b8372eecd9
server: split server.cpp code into server/common/task/queue ( #17362 )
...
* add server-task, server-common
* add server-queue
* rm redundant includes
* move enum stop_type to server-task
* server : headers cleanup
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-24 14:41:53 +01:00
Xuan-Son Nguyen and GitHub
054a45c3d3
grammar: fix regression caused by #17381 ( #17412 )
...
* grammar: fix regression caused by #17381
* more readable
2025-11-20 18:35:10 +01:00
10e9780154
chat: fix int overflow, prevent size calculation in float/double ( #17357 )
...
* chat: fix int overflow, prevent size calculation in float/double
* Update common/chat.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-18 19:11:53 +01:00
0de8878c96
server: split HTTP into its own interface ( #17216 )
...
* server: split HTTP into its own interface
* move server-http and httplib to its own file
* add the remaining endpoints
* fix exception/error handling
* renaming
* missing header
* fix missing windows header
* fix error responses from http layer
* fix slot save/restore handler
* fix case where only one stream chunk is returned
* add NOMINMAX
* do not call sink.write on empty data
* use safe_json_to_str for SSE
* clean up
* add some comments
* improve usage of next()
* bring back the "server is listening on" message
* more generic handler
* add req.headers
* move the chat template print to init()
* add req.path
* cont : minor
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-17 22:05:44 +01:00
Xuan-Son Nguyen and GitHub
9b17d74ab7
mtmd: add mtmd_log_set ( #17268 )
2025-11-14 15:56:19 +01:00
Xuan-Son Nguyen and GitHub
c4abcb2457
server: fixing naming conflict res_error ( #17243 )
2025-11-13 20:53:47 +01:00
Xuan-Son Nguyen and GitHub
00c94083b3
server: (refactor) implement generator-based API for task results ( #17174 )
...
* server: (refactor) implement generator-based API for task results
* improve
* moving some code
* fix "Response ended prematurely"
* add sink.done before return false
* rm redundant check
* rm unused var
* rename generator --> reader
2025-11-12 18:50:52 +01:00
Xuan-Son Nguyen and GitHub
017eceed61
ci: add check vendor job ( #17179 )
...
* ci: add check vendor job
* use dev version of miniaudio
* move to dedicated workflow, only run on related files changed
2025-11-12 14:56:02 +01:00
Xuan-Son Nguyen and GitHub
ee8dd5c658
server: move res_error/res_ok to static function ( #17167 )
2025-11-12 14:17:24 +01:00
Xuan-Son Nguyen and GitHub
1d45b4228f
vendor: split httplib to cpp/h files ( #17150 )
...
* vendor: split httplib to cpp/h files
* move defines
* include httplib if curl is not used
* add TODO
* fix build ios
* fix build visionos instead
2025-11-11 13:32:58 +01:00
Xuan-Son Nguyen and GitHub
4b13a684c5
mtmd: fix patch_size initialized to random value in audio models ( #17128 )
...
* mtmd: fix patch_size initialized to random value in audio models
* add default hparams
2025-11-10 11:41:05 +01:00
aa3b7a90b4
arg: add --cache-list argument to list cached models ( #17073 )
...
* arg: add --cache-list argument to list cached models
* new manifest naming format
* improve naming
* Update common/arg.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-08 21:54:14 +01:00
5c9a18e674
common: move download functions to download.(cpp|h) ( #17059 )
...
* common: move download functions to download.(cpp|h)
* rm unused includes
* minor cleanup
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-07 11:23:34 +01:00
Xuan-Son Nguyen and GitHub
4882f0ff78
clip: implement minicpm-v sinusoidal embd using GGML ( #17036 )
...
* clip: implement minicpm-v sinusoidal embd using GGML
* fix repeat op
2025-11-06 11:02:54 +01:00
Xuan-Son Nguyen and GitHub
92bb84f775
mtmd: allow QwenVL to process larger image by default ( #17020 )
2025-11-05 14:26:49 +01:00
Xuan-Son Nguyen and GitHub
2f0c2db43e
mtmd: improve struct initialization ( #16981 )
2025-11-05 11:26:37 +01:00
Xuan-Son Nguyen and GitHub
070ff4d535
mtmd: add --image-min/max-tokens ( #16921 )
2025-11-03 11:11:18 +01:00
Xuan-Son Nguyen and GitHub
bf7b0c9725
mtmd: pad mask for qwen2.5vl ( #16954 )
...
* mtmd: pad mask for qwen2.5vl
* improve
2025-11-03 10:25:55 +01:00
Xuan-Son Nguyen and GitHub
cf659bbb8e
mtmd: refactor preprocessing + support max/min pixels ( #16878 )
...
* mtmd: refactor preprocessing + support max/min pixels
* fix mlp type
* implement mix/max pixels
* improve hparams
* better image preproc for qwen
* fix
* fix out of bound composite
* fix (2)
* fix token calculation
* get_merge_kernel_size()
* fix llama4 and lfm2
* gonna fix them all
* use simple resize for qwen
* qwen: increase min tokens
* no resize if dst size == src size
* restore to initial min/max tokens value for qwen
2025-11-01 15:51:36 +01:00
Xuan-Son Nguyen and GitHub
3464bdac37
llama: fix ASAN error with M-RoPE ( #16848 )
2025-10-29 20:11:39 +01:00
e3af5563bd
llama: store mrope data in KV cell ( #16825 )
...
* llama: store mrope data in KV cell
* correct x,y ordering
* address review comments
* add consistency checks
* Update src/llama-kv-cache.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* add TODO
* fix asan error
* kv-cells : improve ext handling
* cont : fix headers
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-10-29 18:09:18 +01:00
Xuan-Son Nguyen and GitHub
e1ab084803
mtmd : fix idefics3 preprocessing ( #16806 )
...
* mtmd : fix idefics3 preprocessing
* disable granite test
* fix test for granite
2025-10-27 23:12:16 +01:00
Xuan-Son Nguyen and GitHub
c55d53acec
model : add LightOnOCR-1B model ( #16764 )
...
* model : add LightOnOCR-1B model
* add test
2025-10-27 16:02:58 +01:00
Xuan-Son Nguyen and GitHub
d0660f237a
mtmd-cli : allow using --jinja ( #16718 )
...
* mtmd-cli : allow using --jinja
* support -sys
* implement chat_history
* fix clear memory
* rm -sys support, added TODO
2025-10-23 15:00:49 +02:00
Xuan-Son Nguyen and GitHub
1bb4f43380
mtmd : support home-cooked Mistral Small Omni ( #14928 )
2025-10-16 19:00:31 +02:00
3e3cb19f64
llama-quant: add support for mmproj ( #16592 )
...
* llama-quant: add support for mmproj
* Update src/llama.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* check prefix instead
* small fix
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-10-15 14:48:08 +02:00
Xuan-Son Nguyen and GitHub
05a2458121
codeowners : update ownership for @ngxson and @allozuar ( #16128 )
2025-09-22 11:10:58 +03:00
Xuan-Son Nguyen and GitHub
4b8560ab56
chat : fix build on arm64 ( #16101 )
2025-09-19 13:02:51 +07:00
Xuan-Son Nguyen and GitHub
0dd58b6877
ggml : refactor forward_dup for cpu backend ( #16062 )
...
* ggml : refactor forward_dup for cpu backend
* clean up a bit
* add quant/dequant perf test
2025-09-19 06:31:56 +02:00
Xuan-Son Nguyen and GitHub
8f8f2274ee
convert : add Llama4ForCausalLM ( #16042 )
...
* convert : add Llama4ForCausalLM
* handle swa
* half working version
* fix use_kq_norm
* fix use_kq_norm
2025-09-17 19:18:21 +02:00
Xuan-Son Nguyen and GitHub
56920f5665
server : bring back timings_per_token ( #15879 )
2025-09-08 16:50:05 +02:00
Xuan-Son Nguyen and GitHub
9fcb29f22f
ggml: allow casting between f32 and i32 ( #15783 )
...
* ggml: allow casting between f32 and i32
* fix cuda
* add vulkan
* fix CPU non-cont
* add non-cont test case
* add note
* extend test number range
* correct note
* add cont version for vulkan
2025-09-08 12:33:01 +02:00
Xuan-Son Nguyen and GitHub
3c3635d2f2
server : speed up tests ( #15836 )
...
* server : speed up tests
* clean up
* restore timeout_seconds in some places
* flake8
* explicit offline
2025-09-06 14:45:24 +02:00
61bdfd5298
server : implement prompt processing progress report in stream mode ( #15827 )
...
* server : implement `return_progress`
* add timings.cache_n
* add progress.time_ms
* add test
* fix test for chat/completions
* readme: add docs on timings
* use ggml_time_us
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-09-06 13:35:04 +02:00
Xuan-Son Nguyen and GitHub
a68d914426
server: add exceed_context_size_error type ( #15780 )
...
* server: add exceed_context_size_error type
* change error code to 400
2025-09-04 11:50:23 +02:00
Xuan-Son Nguyen and GitHub
79a546220c
mtmd : support Kimi VL model ( #15458 )
...
* convert : fix tensor naming conflict for llama 4 vision
* convert ok
* support kimi vision model
* clean up
* fix style
* fix calc number of output tokens
* refactor resize_position_embeddings
* add test case
* rename build fn
* correct a small bug
2025-08-26 12:54:19 +02:00
Xuan-Son Nguyen and GitHub
945e1f12a6
ggml : fix condition of im2col on Metal backend ( #15460 )
2025-08-21 08:32:26 +03:00
Xuan-Son Nguyen and GitHub
e9288e8869
chat : clarify the meaning of reasoning_format ( #15408 )
...
* chat : clarify the meaning of reasoning_format
* add link to this PR
2025-08-19 10:29:36 +02:00
Xuan-Son Nguyen and GitHub
f08c4c0d8d
mtmd : clean up clip_n_output_tokens ( #15391 )
2025-08-18 22:53:52 +02:00
Xuan-Son Nguyen and GitHub
fba5c0d680
chat : hotfix gpt-oss jinja raising an exception ( #15243 )
...
* chat : hotfix gpt-oss jinja raising an exception
* fix
2025-08-11 15:31:35 +02:00
Xuan-Son Nguyen and GitHub
53d0a12658
server : allow specifying reasoning_format in HTTP request ( #15238 )
2025-08-11 14:48:41 +02:00
Xuan-Son Nguyen and GitHub
50aa938901
convert : support non-mxfp4 HF model ( #15153 )
...
* convert : support non-mxfp4 HF model
* rm redundant check
* disable debug check
2025-08-07 23:26:03 +02:00
00fa15fedc
mtmd : add support for Voxtral ( #14862 )
...
* mtmd : add support for Voxtral
* clean up
* fix python requirements
* add [BEGIN_AUDIO] token
* also support Devstral conversion
* add docs and tests
* fix regression for ultravox
* minor coding style improvement
* correct project activation fn
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-07-28 15:01:48 +02:00
Xuan-Son Nguyen and GitHub
820de57d4f
chat : fix kimi-k2 chat template ( #14852 )
2025-07-24 13:59:56 +02:00
Xuan-Son Nguyen and GitHub
435a6d10d6
llama : minor coding style fix for smollm3 ( #14605 )
2025-07-10 10:00:20 +03:00
Xuan-Son Nguyen and GitHub
cb9178f885
llama : remove llm_graph_input_one ( #14603 )
2025-07-09 23:09:28 +02:00
Xuan-Son Nguyen and GitHub
98bab638fb
ggml : add ggml_scale_bias ( #14417 )
...
* ggml : add ggml_scale_bias
* ggml_vec_mad1_f32
* add more simd
* add CUDA
* sycl
* vulkan
* cann (placeholder)
* opencl
* will this fix cpu?
* fix cuda
* suggestions from coderabbit
* fix cann compile error
* vDSP_vsmsa
* rm __ARM_FEATURE_SVE
* use memcpy for op params
* make code looks more consistent
* use scalar for __ARM_FEATURE_SVE
* add x param to ggml_vec_mad1_f32
2025-07-09 18:16:12 +02:00
Xuan-Son Nguyen and GitHub
20b7bf8a32
convert : fix smollm3 jinja template ( #14586 )
2025-07-09 09:26:13 +03:00
08382869a2
model : add SmolLM3 ( #14581 )
...
* Init - first pass.
* Model -> ModelBase.
* fix errors in conversion.
* Update the graph.
* up.
* up.
* wip
* cgraph ok
* rm redundant code
---------
Co-authored-by: Vaibhavs10 <vaibhavs10@gmail.com >
2025-07-08 18:07:01 +02:00
8f22dc0a53
model : add hunyuan moe ( #14425 )
...
* model : add hunyuan moe
* tokenizer ok
* fix tensor name
* cgraph init
* chat template
* wip
* almost working
* skip embed, fix bos
* cleanup
* yarn scaling
* cleanup
* correct rope type
* failed token fix
* ntk alpha freq_base
* tokenization working
* cleanup and pr changes
* vocab_size sanity check
* ntk alpha generic
* Update convert_hf_to_gguf.py
* Apply suggestions from code review
* fix regression
* fix style
---------
Co-authored-by: kooshi <1934337+kooshi@users.noreply.github.com >
2025-07-08 11:24:06 +03:00
Xuan-Son Nguyen and GitHub
0c2ee38ab7
convert : correct gemma 3n conversion ( #14450 )
...
* convert : correct gemma 3n conversion
* rm redundant code
2025-07-03 10:03:06 +02:00
Xuan-Son Nguyen and GitHub
8846aace49
model : gemma3n text-only ( #14400 )
...
* gemma3n
* add llm_graph_input_one
2025-06-26 20:34:02 +03:00
Xuan-Son Nguyen and GitHub
413977de32
mtmd : refactor llava-uhd preprocessing logic ( #14247 )
...
* mtmd : refactor llava-uhd preprocessing logic
* fix editorconfig
2025-06-18 10:43:57 +02:00
Xuan-Son Nguyen and GitHub
95402553a5
llama-chat : fix multiple system message for gemma, orion ( #14246 )
2025-06-18 09:58:43 +02:00
Xuan-Son Nguyen and GitHub
3ac67535c8
llama-graph : use ggml_repeat_4d ( #13998 )
2025-06-04 10:11:26 +02:00
Xuan-Son Nguyen and GitHub
ea1431b0fa
docs : add "Quick start" section for new users ( #13862 )
...
* docs : add "Quick start" section for non-technical users
* rm flox
* Update README.md
2025-06-03 13:09:36 +02:00
bfd322796c
mtmd : fix memory leak in mtmd_helper_eval_chunk_single ( #13961 )
...
* mtmd : fix memory in mtmd_helper_eval_chunk_single
* mtmd-cli : fix mem leak
* Update tools/mtmd/mtmd-cli.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-06-02 16:29:28 +02:00
Xuan-Son Nguyen and GitHub
51fa76f172
mtmd : drop _shared from libmtmd name, merge helpers into libmtmd ( ⚠️ breaking change) ( #13917 )
...
* mtmd : fix missing public header
* no object
* apply suggestion from Georgi
* rm mtmd-helper, merge it to mtmd
* missing vendor include dir
2025-05-31 10:14:29 +02:00
Xuan-Son Nguyen and GitHub
07e4351ce6
convert : allow partial update to the chkhsh pre-tokenizer list ( #13847 )
...
* convert : allow partial update to the chkhsh pre-tokenizer list
* code style
* update tokenizer out
* rm inp/out files for models not having gguf
* fixed hash for glm
* skip nomic-bert-moe test
* Update convert_hf_to_gguf_update.py
* fix minerva-7b hash
* rm redundant import
2025-05-30 12:24:37 +02:00
Xuan-Son Nguyen and GitHub
763d06edb7
llama : fix KV shift for qwen2vl ( #13870 )
...
* llama : fix KV shift for qwen2vl
* add ref to the PR
2025-05-28 22:35:31 +02:00
Xuan-Son Nguyen and GitHub
10961339b2
mtmd : move helpers to dedicated library ( ⚠️ breaking change) ( #13866 )
...
* mtmd : move helpers to dedicated library
* fix server build
* rm leftover cmakelist code
2025-05-28 22:35:22 +02:00
Xuan-Son Nguyen and GitHub
a3938fb53d
convert : fix qwen omni conversion ( #13859 )
...
* convert : fix qwen omni conversion
* fix typo
2025-05-28 16:12:35 +02:00
Xuan-Son Nguyen and GitHub
26b79b6cb3
convert : fix tensor naming conflict for llama 4 vision ( #13836 )
...
* convert : fix tensor naming conflict for llama 4 vision
* add comment
2025-05-28 10:05:54 +02:00
Xuan-Son Nguyen and GitHub
a8ea03d8ad
ggml : add ggml_repeat_4d ( #13824 )
2025-05-27 15:53:55 +02:00
Xuan-Son Nguyen and GitHub
bc583e3c63
mtmd : support Qwen 2.5 Omni (input audio+vision, no audio output) ( #13784 )
...
* mtmd : allow multiple modalities at the same time
* refactor mtmd tokenizer
* fix compile
* ok, missing SinusoidsPositionEmbedding
* first working version
* fix style
* more strict validate of n_embd
* refactor if..else to switch
* fix regression
* add test for 3B
* update docs
* fix tokenizing with add_special
* add more tests
* fix test case "huge"
* rm redundant code
* set_position_mrope_1d rm n_tokens
2025-05-27 14:06:10 +02:00
Xuan-Son Nguyen and GitHub
2f099b510f
webui : bump max upload file size to 500MB ( #13779 )
2025-05-25 18:02:18 +01:00
Xuan-Son Nguyen and GitHub
40aaa8a403
mtmd : add support for Qwen2-Audio and SeaLLM-Audio ( #13760 )
...
* mtmd : add Qwen2-Audio support
* small clean up
* update discussion link
* clarify mtmd_get_output_embd
* clarification in multimodal.md
* fix ultravox bug
* ggml_cont
2025-05-25 14:06:32 +02:00
Xuan-Son Nguyen and GitHub
4c32832c59
ggml : add ggml_gelu_erf() CUDA kernel ( #13719 )
...
* ggml : add ggml_gelu_erf() CUDA kernel
* missing semicolon
2025-05-24 13:06:47 +02:00
Xuan-Son Nguyen and GitHub
8a2afb7520
llama : allow custom list of swa_layers ( #13726 )
2025-05-23 17:07:04 +02:00
