nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 30 Packages Projects Releases Wiki Activity
5,096 Commits 1 Branch 0 Tags
11d07e1e69138b46375e9267b31acd58e3813577
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Xuan-Son NguyenandGitHub 96e1280839 clip : bring back GPU support (#12322) 
* clip : bring back GPU support

* use n_gpu_layers param

* fix double free

* ggml_backend_init_by_type

* clean up
 2025-03-11 09:20:16 +01:00
Xuan-Son NguyenandGitHub 7b69003af7 webui : add ?m=... and ?q=... params (#12148) 
* webui : add ?m=... and ?q=... params

* also clear prefilledMessage variable

* better approach

* fix comment

* test: bump timeout on GITHUB_ACTION
 2025-03-03 11:42:45 +01:00
Xuan-Son NguyenandGitHub 06c2b1561d convert : fix Norway problem when parsing YAML (#12114) 
* convert : fix Norway problem when parsing YAML

* Update gguf-py/gguf/metadata.py

* add newline at correct place
 2025-02-28 17:44:46 +01:00
Xuan-Son NguyenGitHubNicholas Sparks
 		c43a3e7996 llama : add Phi-4-mini support (supersede #12099) (#12108) 
* Added Phi-4-mini-instruct support

* Update regex per ngxson

* Change the vocab base to Xenova/gpt-4o

* fix conversion update script

* no need to check longrope

* minor style fix

* fix python style

---------

Co-authored-by: Nicholas Sparks <nisparks@microsoft.com>
 2025-02-28 12:44:11 +01:00
Xuan-Son NguyenGitHubGeorgi Gerganov
 		63ac128563 server : add TEI API format for /rerank endpoint (#11942) 
* server : add TEI API format for /rerank endpoint

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* fix

* also gitignore examples/server/*.gz.hpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-02-18 14:21:41 +01:00
Xuan-Son NguyenandGitHub 0f2bbe6564 server : bump httplib to 0.19.0 (#11908) 2025-02-16 17:11:22 +00:00
Xuan-Son NguyenandGitHub 818a340ea8 ci : fix (again) arm64 build fails (#11895) 
* docker : attempt fixing arm64 build on ci

* qemu v7.0.0-28
 2025-02-16 10:36:39 +01:00
Xuan-Son NguyenGitHubcamel-cdr
 		be3bbd6215 ggml : x2 speed for WASM by optimizing SIMD (#11453) 
* ggml : x2 speed for WASM by optimizing SIMD

* fix bad merging

* rm trailing spaces

* rm redundant clamp

* better quantize_row_q8_K

Co-authored-by: camel-cdr <camel-cdr@protonmail.com>

* remove memset that causes buffer overflow
Co-authored-by: camel-cdr <camel-cdr@protonmail.com>

---------

Co-authored-by: camel-cdr <camel-cdr@protonmail.com>
 2025-02-13 00:33:45 +01:00
Xuan-Son NguyenandGitHub 507f9174fe server : (webui) introduce conversation branching + idb storage (#11792) 
* server : (webui) introduce conversation branching + idb storage

* mark old conv as "migrated" instead deleting them

* improve migration

* add more comments

* more clarification
 2025-02-10 21:23:17 +01:00
Xuan-Son NguyenandGitHub 0893e0114e server : correct signal handler (#11795) 2025-02-10 18:03:28 +01:00
Xuan-Son NguyenandGitHub 55ac8c7791 server : (webui) revamp Settings dialog, add Pyodide interpreter (#11759) 
* redo Settings modal UI

* add python code interpreter

* fix auto scroll

* build

* fix overflow for long output lines

* bring back sticky copy button

* adapt layout on mobile view

* fix multiple lines output and color scheme

* handle python exception

* better state management

* add webworker

* add headers

* format code

* speed up by loading pyodide on page load

* (small tweak) add small animation to make it feels like claude
 2025-02-08 21:54:50 +01:00
Xuan-Son NguyenandGitHub 0cf867160c server : (webui) fix numeric settings being saved as string (#11739) 
* server : (webui) fix numeric settings being saved as string

* add some more comments
 2025-02-08 10:42:34 +01:00
Xuan-Son NguyenandGitHub 2fb3c32a16 server : (webui) migrate project to ReactJS with typescript (#11688) 
* init version

* fix auto scroll

* bring back copy btn

* bring back thought process

* add lint and format check on CI

* remove lang from html tag

* allow multiple generations at the same time

* lint and format combined

* fix unused var

* improve MarkdownDisplay

* fix more latex

* fix code block cannot be selected while generating
 2025-02-06 17:32:29 +01:00
Xuan-Son NguyenandGitHub 3962fc1a79 server : add try..catch to places not covered by set_exception_handler (#11620) 
* server : add try..catch to places not covered by set_exception_handler

* log_server_request: rm try catch, add reminder
 2025-02-04 18:25:42 +01:00
Xuan-Son NguyenandGitHub b3451785ac server : (webui) revert hacky solution from #11626 (#11634) 2025-02-04 00:10:52 +01:00
Xuan-Son NguyenandGitHub d0c08040b6 ci : fix build CPU arm64 (#11472) 
* ci : fix build CPU arm64

* failed, trying ubuntu 22

* vulkan: ubuntu 24

* vulkan : jammy --> noble
 2025-01-29 00:02:56 +01:00
Xuan Son NguyenandGitHub caf773f249 docker : fix ARM build and Vulkan build (#11434) 
* ci : do not fail-fast for docker

* build arm64/amd64 separatedly

* fix pip

* no fast fail

* vulkan: try jammy
 2025-01-26 22:45:32 +01:00
Xuan Son NguyenandGitHub 49b0e3cec4 server : fix cleaning up stream task (#11418) 
* server : fix cleaning up stream task

* one more spot
 2025-01-25 16:36:44 +01:00
Xuan Son NguyenandGitHub 5845661640 server : add more clean up when cancel_tasks is called (#11340) 
* server : add more clean up when cancel_tasks is called

* fix recv_with_timeout

* std::remove_if

* fix std::remove_if
 2025-01-23 13:56:05 +01:00
Xuan Son NguyenandGitHub e28245f35f export-lora : fix tok_embd tensor (#11330) 2025-01-21 14:07:12 +01:00
Xuan Son NguyenandGitHub ec7f3ac9ab llama : add support for Deepseek-R1-Qwen distill model (#11310) 
* llama : add support for Deepseek-R1-Qwen distill model

* coding style
 2025-01-20 14:35:07 +01:00
Xuan Son NguyenandGitHub f30f099228 server : implement cancellable request (#11285) 
* server : implement cancellable request

* fix typo

* httplib 0.18.5

* fix i underflow
 2025-01-18 14:12:05 +01:00
Xuan Son NguyenandGitHub 681149ced2 llama : add llama_model_load_from_splits (#11255) 
* llama : add `llama_model_load_from_splits`

* update
 2025-01-16 13:54:08 +01:00
Xuan Son NguyenandGitHub b4d92a59a2 ci : add -no-cnv for tests (#11238) 2025-01-14 16:42:23 +02:00
Xuan Son NguyenandGitHub 84a44815f7 cli : auto activate conversation mode if chat template is available (#11214) 
* cli : auto activate conversation mode if chat template is detected

* add warn on bad template

* update readme (writing with the help of chatgpt)

* update readme (2)

* do not activate -cnv for non-instruct models
 2025-01-13 20:18:12 +01:00
Xuan Son NguyenandGitHub 00b4c3da62 common : support tag-based --hf-repo like on ollama (#11195) 
* common : support tag-based hf_repo like on ollama

* fix build

* various fixes

* small fixes

* fix style

* fix windows build?

* move common_get_hf_file to common.cpp

* fix complain with noreturn
 2025-01-13 13:56:23 +01:00
Xuan Son NguyenandGitHub 9a483999a6 llama : fix chat template gguf key (#11201) 2025-01-12 13:45:14 +01:00
Xuan Son NguyenandGitHub d9feae1c06 llama-chat : add phi 4 template (#11148) 2025-01-09 10:07:33 +01:00
Xuan Son NguyenandGitHub f7cd13301c ci : use actions from ggml-org (#11140) 2025-01-08 16:09:20 +01:00
Xuan Son NguyenandGitHub 4d2b3d8804 lora : improve compat with mergekit-extract-lora (#11131) 
* (wip) support mergekit-extracted lora

* support mergekit-extract-lora

* use lora->get_scale

* correct comment

* correct norm name & condition

* add some hints
 2025-01-08 15:59:53 +01:00
Xuan Son NguyenandGitHub 80ccf5d725 ci : pin dependency to specific version (#11137) 
* ci : pin dependency to specific version

* will this fix ec?
 2025-01-08 12:07:20 +01:00
Xuan Son NguyenGitHubJohannes Gäßler
 		96be8c3264 github : add cmd line field to bug report (#11090) 
* github : cmd line to bug report

* codeowners : (@ngxson) only watch dockerfile

* Apply suggestions from code review [no ci]

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* rm cmd in log output [no ci]

* rm 2 [no ci]

* no need backticks [no ci]

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
 2025-01-06 16:34:49 +01:00
Xuan Son NguyenandGitHub 09186fabbe llama : remove check flash_attn with lora (#11104) 2025-01-06 13:41:12 +01:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		0da5d86026 server : allow using LoRA adapters per-request (#10994) 
* slot.can_batch_with

* lora per request

* test: force disable cache prompt

* move can_batch_with check

* fix condition

* add slow test with llama 8b

* update docs

* move lora change task to queue

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* lora_base

* remove redundant check

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-01-02 15:05:18 +01:00
Xuan Son NguyenandGitHub 45095a61bf server : clean up built-in template detection (#11026) 
* server : clean up built-in template detection

* fix compilation

* add chat template test

* fix condition
 2024-12-31 15:22:01 +01:00
Xuan Son NguyenandGitHub 5896c65232 server : add OAI compat for /v1/completions (#10974) 
* server : add OAI compat for /v1/completions

* add test

* add docs

* better docs
 2024-12-31 12:34:13 +01:00
Xuan Son NguyenandGitHub 14b699ecde server : fix missing model id in /model endpoint (#10957) 
* server : fix missing model id in /model endpoint

* fix ci
 2024-12-23 12:52:25 +01:00
Xuan Son NguyenandGitHub 485dc01214 server : add system_fingerprint to chat/completion (#10917) 
* server : add system_fingerprint to chat/completion

* update README
 2024-12-23 12:02:44 +01:00
Xuan Son NguyenandGitHub 0ca416c91a server : (UI) fix copy to clipboard function (#10916) 2024-12-20 14:12:06 +01:00
Xuan Son NguyenandGitHub 57bb2c40cd server : fix logprobs, make it OAI-compatible (#10783) 
* server : fix logprobs, make it openai-compatible

* update docs

* add std::log

* return pre-sampling p

* sort before apply softmax

* add comment

* fix test

* set p for sampled token

* update docs

* add --multi-token-probs

* update docs

* add `post_sampling_probs` option

* update docs [no ci]

* remove --multi-token-probs

* "top_probs" with "post_sampling_probs"

* resolve review comments

* rename struct token_prob to prob_info

* correct comment placement

* fix setting prob for sampled token
 2024-12-19 15:40:08 +01:00
Xuan Son NguyenandGitHub 46828872c3 server : (embeddings) using same format for "input" and "content" (#10872) 
* server : (embeddings) using same format for "input" and "content"

* fix test case

* handle empty input case

* fix test
 2024-12-18 10:55:09 +02:00
Xuan Son NguyenandGitHub 227d7c5a7f server : (UI) fix missing async generator on safari (#10857) 
* server : (UI) fix missing async generator on safari

* fix
 2024-12-17 09:52:09 +01:00
Xuan Son NguyenandGitHub adffa6ffd5 common : improve -ctv -ctk CLI arguments (#10806) 
* common : improve ctv ctk cli argument

* regenerate docs

* even better approach

* use std::vector
 2024-12-12 22:53:05 +01:00
Xuan Son NguyenandGitHub 274ec65af6 contrib : add ngxson as codeowner (#10804) 2024-12-12 20:52:28 +01:00
Xuan Son NguyenandGitHub 9fdb124304 common : add missing env var for speculative (#10801) 2024-12-12 16:57:32 +01:00
Xuan Son NguyenandGitHub 235f6e14bf server : (UI) add tok/s, get rid of completion.js (#10786) 
* get rid of completion.js

* extract chat bubble to a component

* add tok/s info

* sync

* fix BASE_URL

* only extract timings when it's enabled

* fix auto scroll
 2024-12-11 20:52:14 +01:00
Xuan Son NguyenandGitHub 92f77a640f ci : pin nodejs to 22.11.0 (#10779) 2024-12-11 14:59:41 +01:00
Xuan Son NguyenandGitHub ce8784bdb1 server : fix format_infill (#10724) 
* server : fix format_infill

* fix

* rename

* update test

* use another model

* update test

* update test

* test_invalid_input_extra_req
 2024-12-08 23:04:29 +01:00
Xuan Son NguyenandGitHub e52522b869 server : bring back info of final chunk in stream mode (#10722) 
* server : bring back into to final chunk in stream mode

* clarify a bit

* traling space
 2024-12-08 20:38:51 +01:00
Xuan Son NguyenandGitHub 3573fa8e7b server : (refactor) no more json in server_task input (#10691) 
* server : (refactor) no more json in server_task input

* add test for slots endpoint

* add tests for /props and /slots

* remove task inf_type

* fix CI by adding safe_json_to_str

* add "model_path" to /props

* update readme
 2024-12-07 20:21:09 +01:00
Xuan Son NguyenandGitHub f162d45a21 common : bring back --no-warmup to server (#10686) 2024-12-06 13:29:05 +01:00
Xuan Son NguyenandGitHub 6c5bc0625f server : (refactoring) do not rely on JSON internally (#10643) 
* server : (refactoring) reduce usage of json internally

* move all response types to struct

* wip [no ci]

* many fixes

* add virtual function

* fix index

* minor style fix

* add std::move

* refactor handle_completions_generic

* add virtual functions

* remove server.hpp

* clarify server_sent_event RFC specs

* apply review comments

* fix model_alias and completion_probabilities

* small clean up

* remove virtual for to_json_oai_compat()

* naming oai_compat --> oaicompat

* fix unwanted recursive call

* update docs
 2024-12-06 11:14:32 +01:00
Xuan Son NguyenandGitHub 91c36c269b server : (web ui) Various improvements, now use vite as bundler (#10599) 
* hide buttons in dropdown menu

* use npm as deps manager and vite as bundler

* fix build

* fix build (2)

* fix responsive on mobile

* fix more problems on mobile

* sync build

* (test) add CI step for verifying build

* fix ci

* force rebuild .hpp files

* cmake: clean up generated files pre build
 2024-12-03 19:38:44 +01:00
Xuan Son NguyenandGitHub 3b4f2e33e2 llama : add missing LLAMA_API for llama_chat_builtin_templates (#10636) 2024-12-03 12:54:30 +01:00
Xuan Son NguyenandGitHub 642330ac7c llama : add enum for built-in chat templates (#10623) 
* llama : add enum for supported chat templates

* use "built-in" instead of "supported"

* arg: print list of built-in templates

* fix test

* update server README
 2024-12-02 22:10:19 +01:00
Xuan Son NguyenandGitHub b782e5c7d4 server : add more test cases (#10569) 
* server : add split model test

* add test speculative

* add invalid cases
 2024-11-29 21:48:56 +01:00
Xuan Son NguyenandGitHub 6c59567689 server : (tests) don't use thread for capturing stdout/stderr, bump openai client library (#10568) 
* server : (tests) don't use thread for capturing stdout/stderr

* test: bump openai to 1.55.2

* bump openai to 1.55.3
 2024-11-28 19:17:49 +01:00
Xuan Son NguyenandGitHub 9f912511bc common : fix duplicated file name with hf_repo and hf_file (#10550) 2024-11-27 22:30:52 +01:00
Xuan Son NguyenandGitHub 45abe0f74e server : replace behave with pytest (#10416) 
* server : replace behave with pytest

* fix test on windows

* misc

* add more tests

* more tests

* styling

* log less, fix embd test

* added all sequential tests

* fix coding style

* fix save slot test

* add parallel completion test

* fix parallel test

* remove feature files

* update test docs

* no cache_prompt for some tests

* add test_cache_vs_nocache_prompt
 2024-11-26 16:20:18 +01:00
Xuan Son NguyenandGitHub 9901068ac7 server : (web UI) add copy button for code block, fix api key (#10242) 
* server : (web ui) add copy btn for code blocks

* fix problem with api key

* use settings-modal-short-input component

* always show copy btn for code snippet
 2024-11-15 10:48:49 +01:00
Xuan Son NguyenandGitHub 76c6e7f105 server : minor UI fix (#10207) 2024-11-07 18:44:38 -04:00
Xuan Son NguyenandGitHub a71d81cf8c server : revamp chat UI with vuejs and daisyui (#10175) 
* server : simple chat UI with vuejs and daisyui

* move old files to legacy folder

* embed deps into binary

* basic markdown support

* add conversation history, save to localStorage

* fix bg-base classes

* save theme preferences

* fix tests

* regenerate, edit, copy buttons

* small fixes

* docs: how to use legacy ui

* better error handling

* make CORS preflight more explicit

* add GET method for CORS

* fix tests

* clean up a bit

* better auto scroll

* small fixes

* use collapse-arrow

* fix closeAndSaveConfigDialog

* small fix

* remove console.log

* fix style for <pre> element

* lighter bubble color (less distract when reading)
 2024-11-07 17:31:10 -04:00
Xuan Son NguyenandGitHub 9e0ecfb697 server : clarify /slots endpoint, add is_processing (#10162) 
* server : clarify /slots endpoint, add is_processing

* fix tests
 2024-11-04 16:33:29 +01:00
Xuan Son NguyenandGitHub 7554aa4655 convert-lora : make --base optional (#10110) 
* convert-lora : make `--base` optional

* lint

* handle case where base_model_name_or_path is invalid

* do not include metadata from base model

* clarify unspecified --base

* add small comment [no ci]

* trigger ci
 2024-11-02 12:53:17 +01:00
Xuan Son NguyenandGitHub 958367bf53 server : refactor slot input data, move tokenizer to HTTP thread (#10023) 
* server : refactor slot input data, move tokenizer to HTTP thread

* move prompt_tokens.empty() check

* fix incorrect if branch

* fix infinite generation loop

* bring back infill validation

* add infill test

* try fixing format_infill

* fix test

* remove redundant code

* rename completion to inference

* update docs

* use llama_tokens everywhere
 2024-10-24 21:51:22 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		c8c07d658a llama : fix empty batch causing llama_batch_allocr to crash (#9966) 
* llama : fix empty batch cause llama_batch_allocr to crash

* move batch_allocr inside decode/encode_internal

* fix build

* add GGML_ASSERT

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-10-22 16:59:02 +02:00
Xuan Son NguyenandGitHub c421ac072d lora : warn user if new token is added in the adapter (#9948) 2024-10-22 13:08:41 +02:00
Xuan Son NguyenandGitHub cda0e4b648 llama : remove all_pos_0, all_pos_1, all_seq_id from llama_batch (#9745) 
* refactor llama_batch_get_one

* adapt all examples

* fix simple.cpp

* fix llama_bench

* fix

* fix context shifting

* free batch before return

* use common_batch_add, reuse llama_batch in loop

* null terminated seq_id list

* fix save-load-state example

* fix perplexity

* correct token pos in llama_batch_allocr
 2024-10-18 23:18:01 +02:00
Xuan Son NguyenandGitHub 458367a906 server : better security control for public deployments (#9776) 
* server : more explicit endpoint access settings

* protect /props endpoint

* fix tests

* update server docs

* fix typo

* fix tests
 2024-10-08 13:27:04 +02:00
Xuan Son NguyenandGitHub f3fdcfaa79 ci : fine-grant permission (#9710) 2024-10-04 11:47:19 +02:00
Xuan Son NguyenandGitHub a39ab216aa llama : reduce compile time and binary size (#9712) 
* llama : speed up compile time

* fix build

* fix build (2)
 2024-10-02 15:49:55 +02:00
Xuan Son NguyenandGitHub ea9c32be71 ci : fix docker build number and tag name (#9638) 
* ci : fix docker build number and tag name

* fine-grant permissions
 2024-09-25 17:26:01 +02:00
Xuan Son NguyenandGitHub afbbfaa537 server : add more env vars, improve gen-docs (#9635) 
* server : add more env vars, improve gen-docs

* update server docs

* LLAMA_ARG_NO_CONTEXT_SHIFT
 2024-09-25 14:05:13 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		0b3bf966f4 server : add --no-context-shift option (#9607) 
* server : add --no-context-shift option

* small fix

* Update examples/server/tests/features/embeddings.feature

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* tests : minor fix

* revert usage of GGML_ASSERT

* update server documentation

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-09-23 22:23:54 +02:00
Xuan Son NguyenGitHubVJHackVinesh Janarthanan
 		feff4aa846 server : add loading html page while model is loading (#9468) 
* Adding loading page for '/' server requests

* set content when model is loading

* removed loading html file

* updated cmakelist

* updated makefile

* cleaned up whitespace

* cleanup for PR removed error

* updated server test to handle 503 HTML

* updated server test to handle 503 HTML

* ca†ch 503 before parsing json

* revert test

* account for both api and web browser requests

* precommit corrections

* eol fix

* revert changes to pre-commit

* removed print statement

* made loading message more descriptive

* also support .html files

---------

Co-authored-by: VJHack <flymyplane21@gmail.com>
Co-authored-by: Vinesh Janarthanan <36610342+VJHack@users.noreply.github.com>
 2024-09-13 14:23:11 +02:00
Xuan Son NguyenandGitHub d4c3c10fad lora : raise error if lm_head is ignored (#9103) 
* lora : raise error if lm_head is ignored

* fix style

* clarify comment
 2024-09-12 14:33:57 +03:00
Xuan Son NguyenandGitHub 0996c5597f llava : correct args for minicpmv-cli (#9429) 2024-09-11 12:59:13 +02:00
Xuan Son NguyenandGitHub 5bb2c5dbd2 files : remove accidentally added lora_test submodule (#9430) 2024-09-11 13:02:09 +03:00
Xuan Son NguyenandGitHub 6cd4e03444 arg : bring back missing ifdef (#9411) 
* arg : bring back missing ifdef

* replace with llama_supports_gpu_offload
 2024-09-10 22:41:29 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		bfe76d4a17 common : move arg parser code to arg.cpp (#9388) 
* common : move arg parser to arg.cpp

* better categorize args

* add cmake

* missing climits

* missing cstdarg

* common : more explicit includes

* fix build

* refactor gpt_params_parse

* update server readme

* fix test

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-09-09 23:36:09 +02:00
Xuan Son NguyenandGitHub 3f7ccfd649 common : bring back missing args, add env var duplication check (#9375) 
* common : bring back missing args

* move duplication check to test-arg-parser

* add check for duplicated env var

* correct default values
 2024-09-08 18:08:55 +02:00
Xuan Son NguyenandGitHub 00b02bb249 imatrix : fix arg parser for imatrix (#9366) 
* imatrix : fix arg parser

* beautify printing first arg
 2024-09-08 12:12:17 +02:00
Xuan Son NguyenGitHub
 		1b9ae5189c common : refactor arg parser (#9308) 
* (wip) argparser v3

* migrated

* add test

* handle env

* fix linux build

* add export-docs example

* fix build (2)

* skip build test-arg-parser on windows

* update server docs

* bring back missing --alias

* bring back --n-predict

* clarify test-arg-parser

* small correction

* add comments

* fix args with 2 values

* refine example-specific args

* no more lamba capture

Co-authored-by: slaren@users.noreply.github.com

* params.sparams

* optimize more

* export-docs --> gen-docs
 2024-09-07 20:43:51 +02:00
Xuan Son NguyenandGitHub 947538acb8 ggml : fix missing cpu_set_t on emscripten (#9336) 
* ggml : fix missing cpu_set_t on emscripten

* better version

* bring back android part
 2024-09-07 12:01:34 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		9b2c24c099 server : simplify state machine for slot (#9283) 
* server : simplify state machine for slot

* add SLOT_STATE_DONE_PROMPT

* pop_deferred_task

* add missing notify_one

* fix passkey test

* metrics : add n_busy_slots_per_decode

* fix test step

* add test

* maybe fix AddressSanitizer?

* fix deque ?

* missing lock

* pop_deferred_task: also notify

* Update examples/server/server.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-09-06 23:21:29 +02:00
Xuan Son NguyenandGitHub 4a1411b4f1 server : fix missing lock (#9334) 2024-09-06 14:06:04 +02:00
Xuan Son NguyenandGitHub 48baa61ecc server : test script : add timeout for all requests (#9282) 2024-09-02 22:08:38 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		6e7d133a5f server : refactor multitask handling (#9274) 
* server : remove multitask from server_task

* refactor completions handler

* fix embeddings

* use res_ok everywhere

* small change for handle_slots_action

* use unordered_set everywhere

* (try) fix test

* no more "mutable" lambda

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* use deque

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-09-02 17:11:51 +02:00
Xuan Son NguyenandGitHub a77feb5d71 server : add some missing env variables (#9116) 
* server : add some missing env variables

* add LLAMA_ARG_HOST to server dockerfile

* also add LLAMA_ARG_CONT_BATCHING
 2024-08-27 11:07:01 +02:00
Xuan Son NguyenandGitHub 3ba780e2a8 lora : fix llama conversion script with ROPE_FREQS (#9117) 2024-08-23 12:58:53 +02:00
Xuan Son NguyenandGitHub fc54ef0d1c server : support reading arguments from environment variables (#9105) 
* server : support reading arguments from environment variables

* add -fa and -dt

* readme : specify non-arg env var
 2024-08-21 11:04:34 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		8b3befc0e2 server : refactor middleware and /health endpoint (#9056) 
* server : refactor middleware and /health endpoint

* move "fail_on_no_slot" to /slots

* Update examples/server/server.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* fix server tests

* fix CI

* update server docs

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-08-16 17:19:05 +02:00
Xuan Son NguyenandGitHub 828d6ff7d7 export-lora : throw error if lora is quantized (#9002) 2024-08-13 11:41:14 +02:00
Xuan Son NguyenandGitHub 7eb23840ed llama : default n_swa for phi-3 (#8931) 
* default n_swa for phi-3

* fix

* double check swa
 2024-08-10 13:04:40 +02:00
Xuan Son NguyenandGitHub 1e6f6554aa server : add lora hotswap endpoint (WIP) (#8857) 
* server : add lora hotswap endpoint

* handle lora_no_apply

* fix build

* updae docs

* clean up struct def

* fix build

* add LoRA test

* fix style
 2024-08-06 17:33:39 +02:00
Xuan Son NguyenandGitHub 41cd47caab examples : export-lora : fix issue with quantized base models (#8687) 2024-07-25 23:49:39 +02:00
Xuan Son NguyenandGitHub be6d7c0791 examples : remove finetune and train-text-from-scratch (#8669) 
* examples : remove finetune and train-text-from-scratch

* fix build

* update help message

* fix small typo for export-lora
 2024-07-25 10:39:04 +02:00
Xuan Son NguyenandGitHub 96952e7181 llama : fix llama_chat_format_single for mistral (#8657) 
* fix `llama_chat_format_single` for mistral

* fix typo

* use printf
 2024-07-24 13:48:46 +02:00
Xuan Son NguyenandGitHub b115105f05 add llama_lora_adapter_clear (#8653) 2024-07-24 11:25:19 +02:00
Xuan Son NguyenandGitHub de280085e7 examples : Fix llama-export-lora example (#8607) 
* fix export-lora example

* add more logging

* reject merging subset

* better check

* typo
 2024-07-23 23:48:37 +02:00