nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 18 Packages Projects Releases Wiki Activity
4,613 Commits 1 Branch 0 Tags
ff227703d6d6e1888bdc7af6138514092ffcdb96
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Xuan-Son NguyenandGitHub d0c08040b6 ci : fix build CPU arm64 (#11472) 
* ci : fix build CPU arm64

* failed, trying ubuntu 22

* vulkan: ubuntu 24

* vulkan : jammy --> noble
 2025-01-29 00:02:56 +01:00
Xuan Son NguyenandGitHub caf773f249 docker : fix ARM build and Vulkan build (#11434) 
* ci : do not fail-fast for docker

* build arm64/amd64 separatedly

* fix pip

* no fast fail

* vulkan: try jammy
 2025-01-26 22:45:32 +01:00
Xuan Son NguyenandGitHub 49b0e3cec4 server : fix cleaning up stream task (#11418) 
* server : fix cleaning up stream task

* one more spot
 2025-01-25 16:36:44 +01:00
Xuan Son NguyenandGitHub 5845661640 server : add more clean up when cancel_tasks is called (#11340) 
* server : add more clean up when cancel_tasks is called

* fix recv_with_timeout

* std::remove_if

* fix std::remove_if
 2025-01-23 13:56:05 +01:00
Xuan Son NguyenandGitHub e28245f35f export-lora : fix tok_embd tensor (#11330) 2025-01-21 14:07:12 +01:00
Xuan Son NguyenandGitHub ec7f3ac9ab llama : add support for Deepseek-R1-Qwen distill model (#11310) 
* llama : add support for Deepseek-R1-Qwen distill model

* coding style
 2025-01-20 14:35:07 +01:00
Xuan Son NguyenandGitHub f30f099228 server : implement cancellable request (#11285) 
* server : implement cancellable request

* fix typo

* httplib 0.18.5

* fix i underflow
 2025-01-18 14:12:05 +01:00
Xuan Son NguyenandGitHub 681149ced2 llama : add llama_model_load_from_splits (#11255) 
* llama : add `llama_model_load_from_splits`

* update
 2025-01-16 13:54:08 +01:00
Xuan Son NguyenandGitHub b4d92a59a2 ci : add -no-cnv for tests (#11238) 2025-01-14 16:42:23 +02:00
Xuan Son NguyenandGitHub 84a44815f7 cli : auto activate conversation mode if chat template is available (#11214) 
* cli : auto activate conversation mode if chat template is detected

* add warn on bad template

* update readme (writing with the help of chatgpt)

* update readme (2)

* do not activate -cnv for non-instruct models
 2025-01-13 20:18:12 +01:00
Xuan Son NguyenandGitHub 00b4c3da62 common : support tag-based --hf-repo like on ollama (#11195) 
* common : support tag-based hf_repo like on ollama

* fix build

* various fixes

* small fixes

* fix style

* fix windows build?

* move common_get_hf_file to common.cpp

* fix complain with noreturn
 2025-01-13 13:56:23 +01:00
Xuan Son NguyenandGitHub 9a483999a6 llama : fix chat template gguf key (#11201) 2025-01-12 13:45:14 +01:00
Xuan Son NguyenandGitHub d9feae1c06 llama-chat : add phi 4 template (#11148) 2025-01-09 10:07:33 +01:00
Xuan Son NguyenandGitHub f7cd13301c ci : use actions from ggml-org (#11140) 2025-01-08 16:09:20 +01:00
Xuan Son NguyenandGitHub 4d2b3d8804 lora : improve compat with mergekit-extract-lora (#11131) 
* (wip) support mergekit-extracted lora

* support mergekit-extract-lora

* use lora->get_scale

* correct comment

* correct norm name & condition

* add some hints
 2025-01-08 15:59:53 +01:00
Xuan Son NguyenandGitHub 80ccf5d725 ci : pin dependency to specific version (#11137) 
* ci : pin dependency to specific version

* will this fix ec?
 2025-01-08 12:07:20 +01:00
Xuan Son NguyenGitHubJohannes Gäßler
 		96be8c3264 github : add cmd line field to bug report (#11090) 
* github : cmd line to bug report

* codeowners : (@ngxson) only watch dockerfile

* Apply suggestions from code review [no ci]

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* rm cmd in log output [no ci]

* rm 2 [no ci]

* no need backticks [no ci]

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
 2025-01-06 16:34:49 +01:00
Xuan Son NguyenandGitHub 09186fabbe llama : remove check flash_attn with lora (#11104) 2025-01-06 13:41:12 +01:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		0da5d86026 server : allow using LoRA adapters per-request (#10994) 
* slot.can_batch_with

* lora per request

* test: force disable cache prompt

* move can_batch_with check

* fix condition

* add slow test with llama 8b

* update docs

* move lora change task to queue

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* lora_base

* remove redundant check

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2025-01-02 15:05:18 +01:00
Xuan Son NguyenandGitHub 45095a61bf server : clean up built-in template detection (#11026) 
* server : clean up built-in template detection

* fix compilation

* add chat template test

* fix condition
 2024-12-31 15:22:01 +01:00
Xuan Son NguyenandGitHub 5896c65232 server : add OAI compat for /v1/completions (#10974) 
* server : add OAI compat for /v1/completions

* add test

* add docs

* better docs
 2024-12-31 12:34:13 +01:00
Xuan Son NguyenandGitHub 14b699ecde server : fix missing model id in /model endpoint (#10957) 
* server : fix missing model id in /model endpoint

* fix ci
 2024-12-23 12:52:25 +01:00
Xuan Son NguyenandGitHub 485dc01214 server : add system_fingerprint to chat/completion (#10917) 
* server : add system_fingerprint to chat/completion

* update README
 2024-12-23 12:02:44 +01:00
Xuan Son NguyenandGitHub 0ca416c91a server : (UI) fix copy to clipboard function (#10916) 2024-12-20 14:12:06 +01:00
Xuan Son NguyenandGitHub 57bb2c40cd server : fix logprobs, make it OAI-compatible (#10783) 
* server : fix logprobs, make it openai-compatible

* update docs

* add std::log

* return pre-sampling p

* sort before apply softmax

* add comment

* fix test

* set p for sampled token

* update docs

* add --multi-token-probs

* update docs

* add `post_sampling_probs` option

* update docs [no ci]

* remove --multi-token-probs

* "top_probs" with "post_sampling_probs"

* resolve review comments

* rename struct token_prob to prob_info

* correct comment placement

* fix setting prob for sampled token
 2024-12-19 15:40:08 +01:00
Xuan Son NguyenandGitHub 46828872c3 server : (embeddings) using same format for "input" and "content" (#10872) 
* server : (embeddings) using same format for "input" and "content"

* fix test case

* handle empty input case

* fix test
 2024-12-18 10:55:09 +02:00
Xuan Son NguyenandGitHub 227d7c5a7f server : (UI) fix missing async generator on safari (#10857) 
* server : (UI) fix missing async generator on safari

* fix
 2024-12-17 09:52:09 +01:00
Xuan Son NguyenandGitHub adffa6ffd5 common : improve -ctv -ctk CLI arguments (#10806) 
* common : improve ctv ctk cli argument

* regenerate docs

* even better approach

* use std::vector
 2024-12-12 22:53:05 +01:00
Xuan Son NguyenandGitHub 274ec65af6 contrib : add ngxson as codeowner (#10804) 2024-12-12 20:52:28 +01:00
Xuan Son NguyenandGitHub 9fdb124304 common : add missing env var for speculative (#10801) 2024-12-12 16:57:32 +01:00
Xuan Son NguyenandGitHub 235f6e14bf server : (UI) add tok/s, get rid of completion.js (#10786) 
* get rid of completion.js

* extract chat bubble to a component

* add tok/s info

* sync

* fix BASE_URL

* only extract timings when it's enabled

* fix auto scroll
 2024-12-11 20:52:14 +01:00
Xuan Son NguyenandGitHub 92f77a640f ci : pin nodejs to 22.11.0 (#10779) 2024-12-11 14:59:41 +01:00
Xuan Son NguyenandGitHub ce8784bdb1 server : fix format_infill (#10724) 
* server : fix format_infill

* fix

* rename

* update test

* use another model

* update test

* update test

* test_invalid_input_extra_req
 2024-12-08 23:04:29 +01:00
Xuan Son NguyenandGitHub e52522b869 server : bring back info of final chunk in stream mode (#10722) 
* server : bring back into to final chunk in stream mode

* clarify a bit

* traling space
 2024-12-08 20:38:51 +01:00
Xuan Son NguyenandGitHub 3573fa8e7b server : (refactor) no more json in server_task input (#10691) 
* server : (refactor) no more json in server_task input

* add test for slots endpoint

* add tests for /props and /slots

* remove task inf_type

* fix CI by adding safe_json_to_str

* add "model_path" to /props

* update readme
 2024-12-07 20:21:09 +01:00
Xuan Son NguyenandGitHub f162d45a21 common : bring back --no-warmup to server (#10686) 2024-12-06 13:29:05 +01:00
Xuan Son NguyenandGitHub 6c5bc0625f server : (refactoring) do not rely on JSON internally (#10643) 
* server : (refactoring) reduce usage of json internally

* move all response types to struct

* wip [no ci]

* many fixes

* add virtual function

* fix index

* minor style fix

* add std::move

* refactor handle_completions_generic

* add virtual functions

* remove server.hpp

* clarify server_sent_event RFC specs

* apply review comments

* fix model_alias and completion_probabilities

* small clean up

* remove virtual for to_json_oai_compat()

* naming oai_compat --> oaicompat

* fix unwanted recursive call

* update docs
 2024-12-06 11:14:32 +01:00
Xuan Son NguyenandGitHub 91c36c269b server : (web ui) Various improvements, now use vite as bundler (#10599) 
* hide buttons in dropdown menu

* use npm as deps manager and vite as bundler

* fix build

* fix build (2)

* fix responsive on mobile

* fix more problems on mobile

* sync build

* (test) add CI step for verifying build

* fix ci

* force rebuild .hpp files

* cmake: clean up generated files pre build
 2024-12-03 19:38:44 +01:00
Xuan Son NguyenandGitHub 3b4f2e33e2 llama : add missing LLAMA_API for llama_chat_builtin_templates (#10636) 2024-12-03 12:54:30 +01:00
Xuan Son NguyenandGitHub 642330ac7c llama : add enum for built-in chat templates (#10623) 
* llama : add enum for supported chat templates

* use "built-in" instead of "supported"

* arg: print list of built-in templates

* fix test

* update server README
 2024-12-02 22:10:19 +01:00
Xuan Son NguyenandGitHub b782e5c7d4 server : add more test cases (#10569) 
* server : add split model test

* add test speculative

* add invalid cases
 2024-11-29 21:48:56 +01:00
Xuan Son NguyenandGitHub 6c59567689 server : (tests) don't use thread for capturing stdout/stderr, bump openai client library (#10568) 
* server : (tests) don't use thread for capturing stdout/stderr

* test: bump openai to 1.55.2

* bump openai to 1.55.3
 2024-11-28 19:17:49 +01:00
Xuan Son NguyenandGitHub 9f912511bc common : fix duplicated file name with hf_repo and hf_file (#10550) 2024-11-27 22:30:52 +01:00
Xuan Son NguyenandGitHub 45abe0f74e server : replace behave with pytest (#10416) 
* server : replace behave with pytest

* fix test on windows

* misc

* add more tests

* more tests

* styling

* log less, fix embd test

* added all sequential tests

* fix coding style

* fix save slot test

* add parallel completion test

* fix parallel test

* remove feature files

* update test docs

* no cache_prompt for some tests

* add test_cache_vs_nocache_prompt
 2024-11-26 16:20:18 +01:00
Xuan Son NguyenandGitHub 9901068ac7 server : (web UI) add copy button for code block, fix api key (#10242) 
* server : (web ui) add copy btn for code blocks

* fix problem with api key

* use settings-modal-short-input component

* always show copy btn for code snippet
 2024-11-15 10:48:49 +01:00
Xuan Son NguyenandGitHub 76c6e7f105 server : minor UI fix (#10207) 2024-11-07 18:44:38 -04:00
Xuan Son NguyenandGitHub a71d81cf8c server : revamp chat UI with vuejs and daisyui (#10175) 
* server : simple chat UI with vuejs and daisyui

* move old files to legacy folder

* embed deps into binary

* basic markdown support

* add conversation history, save to localStorage

* fix bg-base classes

* save theme preferences

* fix tests

* regenerate, edit, copy buttons

* small fixes

* docs: how to use legacy ui

* better error handling

* make CORS preflight more explicit

* add GET method for CORS

* fix tests

* clean up a bit

* better auto scroll

* small fixes

* use collapse-arrow

* fix closeAndSaveConfigDialog

* small fix

* remove console.log

* fix style for <pre> element

* lighter bubble color (less distract when reading)
 2024-11-07 17:31:10 -04:00
Xuan Son NguyenandGitHub 9e0ecfb697 server : clarify /slots endpoint, add is_processing (#10162) 
* server : clarify /slots endpoint, add is_processing

* fix tests
 2024-11-04 16:33:29 +01:00
Xuan Son NguyenandGitHub 7554aa4655 convert-lora : make --base optional (#10110) 
* convert-lora : make `--base` optional

* lint

* handle case where base_model_name_or_path is invalid

* do not include metadata from base model

* clarify unspecified --base

* add small comment [no ci]

* trigger ci
 2024-11-02 12:53:17 +01:00
Xuan Son NguyenandGitHub 958367bf53 server : refactor slot input data, move tokenizer to HTTP thread (#10023) 
* server : refactor slot input data, move tokenizer to HTTP thread

* move prompt_tokens.empty() check

* fix incorrect if branch

* fix infinite generation loop

* bring back infill validation

* add infill test

* try fixing format_infill

* fix test

* remove redundant code

* rename completion to inference

* update docs

* use llama_tokens everywhere
 2024-10-24 21:51:22 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		c8c07d658a llama : fix empty batch causing llama_batch_allocr to crash (#9966) 
* llama : fix empty batch cause llama_batch_allocr to crash

* move batch_allocr inside decode/encode_internal

* fix build

* add GGML_ASSERT

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-10-22 16:59:02 +02:00
Xuan Son NguyenandGitHub c421ac072d lora : warn user if new token is added in the adapter (#9948) 2024-10-22 13:08:41 +02:00
Xuan Son NguyenandGitHub cda0e4b648 llama : remove all_pos_0, all_pos_1, all_seq_id from llama_batch (#9745) 
* refactor llama_batch_get_one

* adapt all examples

* fix simple.cpp

* fix llama_bench

* fix

* fix context shifting

* free batch before return

* use common_batch_add, reuse llama_batch in loop

* null terminated seq_id list

* fix save-load-state example

* fix perplexity

* correct token pos in llama_batch_allocr
 2024-10-18 23:18:01 +02:00
Xuan Son NguyenandGitHub 458367a906 server : better security control for public deployments (#9776) 
* server : more explicit endpoint access settings

* protect /props endpoint

* fix tests

* update server docs

* fix typo

* fix tests
 2024-10-08 13:27:04 +02:00
Xuan Son NguyenandGitHub f3fdcfaa79 ci : fine-grant permission (#9710) 2024-10-04 11:47:19 +02:00
Xuan Son NguyenandGitHub a39ab216aa llama : reduce compile time and binary size (#9712) 
* llama : speed up compile time

* fix build

* fix build (2)
 2024-10-02 15:49:55 +02:00
Xuan Son NguyenandGitHub ea9c32be71 ci : fix docker build number and tag name (#9638) 
* ci : fix docker build number and tag name

* fine-grant permissions
 2024-09-25 17:26:01 +02:00
Xuan Son NguyenandGitHub afbbfaa537 server : add more env vars, improve gen-docs (#9635) 
* server : add more env vars, improve gen-docs

* update server docs

* LLAMA_ARG_NO_CONTEXT_SHIFT
 2024-09-25 14:05:13 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		0b3bf966f4 server : add --no-context-shift option (#9607) 
* server : add --no-context-shift option

* small fix

* Update examples/server/tests/features/embeddings.feature

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* tests : minor fix

* revert usage of GGML_ASSERT

* update server documentation

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-09-23 22:23:54 +02:00
Xuan Son NguyenGitHubVJHackVinesh Janarthanan
 		feff4aa846 server : add loading html page while model is loading (#9468) 
* Adding loading page for '/' server requests

* set content when model is loading

* removed loading html file

* updated cmakelist

* updated makefile

* cleaned up whitespace

* cleanup for PR removed error

* updated server test to handle 503 HTML

* updated server test to handle 503 HTML

* ca†ch 503 before parsing json

* revert test

* account for both api and web browser requests

* precommit corrections

* eol fix

* revert changes to pre-commit

* removed print statement

* made loading message more descriptive

* also support .html files

---------

Co-authored-by: VJHack <flymyplane21@gmail.com>
Co-authored-by: Vinesh Janarthanan <36610342+VJHack@users.noreply.github.com>
 2024-09-13 14:23:11 +02:00
Xuan Son NguyenandGitHub d4c3c10fad lora : raise error if lm_head is ignored (#9103) 
* lora : raise error if lm_head is ignored

* fix style

* clarify comment
 2024-09-12 14:33:57 +03:00
Xuan Son NguyenandGitHub 0996c5597f llava : correct args for minicpmv-cli (#9429) 2024-09-11 12:59:13 +02:00
Xuan Son NguyenandGitHub 5bb2c5dbd2 files : remove accidentally added lora_test submodule (#9430) 2024-09-11 13:02:09 +03:00
Xuan Son NguyenandGitHub 6cd4e03444 arg : bring back missing ifdef (#9411) 
* arg : bring back missing ifdef

* replace with llama_supports_gpu_offload
 2024-09-10 22:41:29 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		bfe76d4a17 common : move arg parser code to arg.cpp (#9388) 
* common : move arg parser to arg.cpp

* better categorize args

* add cmake

* missing climits

* missing cstdarg

* common : more explicit includes

* fix build

* refactor gpt_params_parse

* update server readme

* fix test

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-09-09 23:36:09 +02:00
Xuan Son NguyenandGitHub 3f7ccfd649 common : bring back missing args, add env var duplication check (#9375) 
* common : bring back missing args

* move duplication check to test-arg-parser

* add check for duplicated env var

* correct default values
 2024-09-08 18:08:55 +02:00
Xuan Son NguyenandGitHub 00b02bb249 imatrix : fix arg parser for imatrix (#9366) 
* imatrix : fix arg parser

* beautify printing first arg
 2024-09-08 12:12:17 +02:00
Xuan Son NguyenGitHub
 		1b9ae5189c common : refactor arg parser (#9308) 
* (wip) argparser v3

* migrated

* add test

* handle env

* fix linux build

* add export-docs example

* fix build (2)

* skip build test-arg-parser on windows

* update server docs

* bring back missing --alias

* bring back --n-predict

* clarify test-arg-parser

* small correction

* add comments

* fix args with 2 values

* refine example-specific args

* no more lamba capture

Co-authored-by: slaren@users.noreply.github.com

* params.sparams

* optimize more

* export-docs --> gen-docs
 2024-09-07 20:43:51 +02:00
Xuan Son NguyenandGitHub 947538acb8 ggml : fix missing cpu_set_t on emscripten (#9336) 
* ggml : fix missing cpu_set_t on emscripten

* better version

* bring back android part
 2024-09-07 12:01:34 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		9b2c24c099 server : simplify state machine for slot (#9283) 
* server : simplify state machine for slot

* add SLOT_STATE_DONE_PROMPT

* pop_deferred_task

* add missing notify_one

* fix passkey test

* metrics : add n_busy_slots_per_decode

* fix test step

* add test

* maybe fix AddressSanitizer?

* fix deque ?

* missing lock

* pop_deferred_task: also notify

* Update examples/server/server.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-09-06 23:21:29 +02:00
Xuan Son NguyenandGitHub 4a1411b4f1 server : fix missing lock (#9334) 2024-09-06 14:06:04 +02:00
Xuan Son NguyenandGitHub 48baa61ecc server : test script : add timeout for all requests (#9282) 2024-09-02 22:08:38 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		6e7d133a5f server : refactor multitask handling (#9274) 
* server : remove multitask from server_task

* refactor completions handler

* fix embeddings

* use res_ok everywhere

* small change for handle_slots_action

* use unordered_set everywhere

* (try) fix test

* no more "mutable" lambda

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* use deque

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-09-02 17:11:51 +02:00
Xuan Son NguyenandGitHub a77feb5d71 server : add some missing env variables (#9116) 
* server : add some missing env variables

* add LLAMA_ARG_HOST to server dockerfile

* also add LLAMA_ARG_CONT_BATCHING
 2024-08-27 11:07:01 +02:00
Xuan Son NguyenandGitHub 3ba780e2a8 lora : fix llama conversion script with ROPE_FREQS (#9117) 2024-08-23 12:58:53 +02:00
Xuan Son NguyenandGitHub fc54ef0d1c server : support reading arguments from environment variables (#9105) 
* server : support reading arguments from environment variables

* add -fa and -dt

* readme : specify non-arg env var
 2024-08-21 11:04:34 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		8b3befc0e2 server : refactor middleware and /health endpoint (#9056) 
* server : refactor middleware and /health endpoint

* move "fail_on_no_slot" to /slots

* Update examples/server/server.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* fix server tests

* fix CI

* update server docs

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-08-16 17:19:05 +02:00
Xuan Son NguyenandGitHub 828d6ff7d7 export-lora : throw error if lora is quantized (#9002) 2024-08-13 11:41:14 +02:00
Xuan Son NguyenandGitHub 7eb23840ed llama : default n_swa for phi-3 (#8931) 
* default n_swa for phi-3

* fix

* double check swa
 2024-08-10 13:04:40 +02:00
Xuan Son NguyenandGitHub 1e6f6554aa server : add lora hotswap endpoint (WIP) (#8857) 
* server : add lora hotswap endpoint

* handle lora_no_apply

* fix build

* updae docs

* clean up struct def

* fix build

* add LoRA test

* fix style
 2024-08-06 17:33:39 +02:00
Xuan Son NguyenandGitHub 41cd47caab examples : export-lora : fix issue with quantized base models (#8687) 2024-07-25 23:49:39 +02:00
Xuan Son NguyenandGitHub be6d7c0791 examples : remove finetune and train-text-from-scratch (#8669) 
* examples : remove finetune and train-text-from-scratch

* fix build

* update help message

* fix small typo for export-lora
 2024-07-25 10:39:04 +02:00
Xuan Son NguyenandGitHub 96952e7181 llama : fix llama_chat_format_single for mistral (#8657) 
* fix `llama_chat_format_single` for mistral

* fix typo

* use printf
 2024-07-24 13:48:46 +02:00
Xuan Son NguyenandGitHub b115105f05 add llama_lora_adapter_clear (#8653) 2024-07-24 11:25:19 +02:00
Xuan Son NguyenandGitHub de280085e7 examples : Fix llama-export-lora example (#8607) 
* fix export-lora example

* add more logging

* reject merging subset

* better check

* typo
 2024-07-23 23:48:37 +02:00
Xuan Son NguyenGitHubslarenFrancis Couture-Harpin
 		97bdd26eee Refactor lora adapter support (#8332) 
* lora: load to devide buft

* add patch tensor function

* correct tensor patch

* llama_lora_adapter_apply

* correct ggml_backend_tensor_copy

* add llm_build_mm

* fix auto merge

* update based on review comments

* add convert script

* no more transpose A

* add f16 convert

* add metadata check

* add sanity check

* fix ftype

* add requirements

* fix requirements

* fix outfile

* conversion: only allow selected models

* fix types

* cuda : do not use dmmv if the tensor does not have enough cols

* llama : lora fixes

* do not disable mmap with lora

Co-authored-by: slaren <slarengh@gmail.com>

* llm_build_lora_mm_id

* convert_lora : MoE LoRA conversion support

* convert_lora : prefer safetensors, similarly to convert_hf

* convert_hf : simplify modify_tensors for InternLM2

* convert_lora : lazy conversion

* llama : load and use alpha from LoRA adapters

* llama : use llm_build_lora_mm in most model graphs

* auto scale

* Revert "auto scale"

This reverts commit 42415a4874e0f963e4aca6796ea5dfb97cd17464.

* remove redundant params

* Apply suggestions from code review

Co-authored-by: slaren <slarengh@gmail.com>

* change kv metadata

* move add_type to __init__

* convert_hf : move add_type to main()

* convert_lora : use the GGUFWriter from Model instead of overwriting it

---------

Co-authored-by: slaren <slarengh@gmail.com>
Co-authored-by: Francis Couture-Harpin <git@compilade.net>
 2024-07-15 20:50:47 +02:00
Xuan Son NguyenandGitHub 4db8f60fe7 fix ci (#8494) 2024-07-15 19:23:10 +02:00
Xuan Son NguyenandGitHub 60d83a0149 update main readme (#8333) 2024-07-06 19:01:23 +02:00
Xuan Son NguyenandGitHub be20e7f49d Reorganize documentation pages (#8325) 
* re-organize docs

* add link among docs

* add link to build docs

* fix style

* de-duplicate sections
 2024-07-05 18:08:32 +02:00
Xuan Son NguyenandGitHub a38b884c6c cli: add EOT when user hit Ctrl+C (#8296) 
* main: add need_insert_eot

* do not format system prompt if it is empty
 2024-07-04 20:55:03 +02:00
Xuan Son NguyenandGitHub 916248af1f fix phi 3 conversion (#8262) 2024-07-03 16:01:54 +02:00
Xuan Son NguyenandGitHub 5fac350b9c Fix gemma2 tokenizer convert (#8244) 
* fix gemma2 tokenizer convert

* remove scores

* improve code, fix new line issue
 2024-07-02 01:07:23 +02:00
Xuan Son NguyenGitHubArlo PhoenixGeorgi Gerganov
 		49122a873f gemma2: add sliding window mask (#8227) 
* gemma2: add sliding window mask

* fix data_swa uninitialized

* better naming

* add co-author

Co-authored-by: Arlo Phoenix <arlo-phoenix@users.noreply.github.com>

* replace list with single tensor

* update

* llama : minor styling

* convert : add sanity check for query_pre_attn_scalar

* fix small typo in README

---------

Co-authored-by: Arlo Phoenix <arlo-phoenix@users.noreply.github.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-07-01 18:48:34 +02:00
Xuan Son NguyenandGitHub 9ef0780062 Fix new line issue with chat template, disable template when in-prefix/suffix is set (#8203) 
* preserve new line llama_chat_format_single

* disable chat template if in-prefix/suffix is set

* remove redundant change
 2024-06-30 20:27:13 +02:00
Xuan Son NguyenandGitHub 72272b83a3 fix code typo in llama-cli (#8198) 2024-06-29 00:14:20 +02:00
Xuan Son NguyenandGitHub 26a39bbd6b Add MiniCPM, Deepseek V2 chat template + clean up llama_chat_apply_template_internal (#8172) 
* tmp_contains

* minicpm chat template

* add DeepSeek Lite template

* change deepseek-lite to deepseek2

* correct code comment

* correct code from master branch
 2024-06-28 15:11:44 +02:00
Xuan Son NguyenandGitHub a27aa50ab7 Add missing items in makefile (#8177) 2024-06-28 02:19:11 +02:00
Xuan Son NguyenandGitHub 16791b8f0b Add chatml fallback for cpp llama_chat_apply_template (#8160) 
* add chatml fallback for cpp `llama_chat_apply_template`

* remove redundant code
 2024-06-27 18:14:19 +02:00
Xuan Son NguyenandGitHub 49c03c79cd cvector: better prompt handling, add "mean vector" method (#8069) 
* remove completions file

* fix inverted vector

* add mean method

* code style

* remove inverted pca hotfix
 2024-06-25 13:59:54 +02:00
Xuan Son NguyenGitHubGeorgi Gerganov
 		48e6b92cc3 Add chat template support for llama-cli (#8068) 
* add chat template support for llama-cli

* add help message

* server: simplify format_chat

* more consistent naming

* improve

* add llama_chat_format_example

* fix server

* code style

* code style

* Update examples/main/main.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-06-25 21:56:49 +10:00