nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 28 Packages Projects Releases Wiki Activity
1,627 Commits 1 Branch 0 Tags
9494d7c4774ab745490b5a19570ff7747a194143
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Georgi GerganovGitHubslaren
 		fe680e3d10 sync : ggml (new ops, tests, backend, etc.) (#4359) 
* sync : ggml (part 1)

* sync : ggml (part 2, CUDA)

* sync : ggml (part 3, Metal)

* ggml : build fixes

ggml-ci

* cuda : restore lost changes

* cuda : restore lost changes (StableLM rope)

* cmake : enable separable compilation for CUDA

ggml-ci

* ggml-cuda : remove device side dequantize

* Revert "cmake : enable separable compilation for CUDA"

This reverts commit 09e35d04b1c4ca67f9685690160b35bc885a89ac.

* cuda : remove assert for rope

* tests : add test-backend-ops

* ggml : fix bug in ggml_concat

* ggml : restore `ggml_get_n_tasks()` logic in `ggml_graph_plan()`

* ci : try to fix macOS

* ggml-backend : remove backend self-registration

* ci : disable Metal for macOS cmake build

ggml-ci

* metal : fix "supports family" call

* metal : fix assert

* metal : print resource path

ggml-ci

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-12-07 22:26:54 +02:00
Georgi GerganovGitHubslaren
 		bcc0eb4591 llama : per-layer KV cache + quantum K cache (#4309) 
* per-layer KV

* remove unnecessary copies

* less code duplication, offload k and v separately

* llama : offload KV cache per-layer

* llama : offload K shift tensors

* llama : offload for rest of the model arches

* llama : enable offload debug temporarily

* llama : keep the KV related layers on the device

* llama : remove mirrors, perform Device -> Host when partial offload

* common : add command-line arg to disable KV cache offloading

* llama : update session save/load

* llama : support quantum K cache (#4312)

* llama : support quantum K cache (wip)

* metal : add F32 -> Q8_0 copy kernel

* cuda : add F32 -> Q8_0 copy kernel

ggml-ci

* cuda : use mmv kernel for quantum cache ops

* llama : pass KV cache type through API

* llama : fix build

ggml-ci

* metal : add F32 -> Q4_0 copy kernel

* metal : add F32 -> Q4_1 copy kernel

* cuda : wip

* cuda : add F32 -> Q4_0 and F32 -> Q4_1 copy kernels

* llama-bench : support type_k/type_v

* metal : use mm kernel only for quantum KV cache

* cuda : add comment

* llama : remove memory_f16 and kv_f16 flags

---------

Co-authored-by: slaren <slarengh@gmail.com>

* readme : add API change notice

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-12-07 13:03:17 +02:00
Georgi GerganovandGitHub 05cd6e5036 server : recognize cache_prompt parameter in OAI API (#4347) 2023-12-06 20:21:59 +02:00
Georgi Gerganov caa9249217 common : fix compile warning 2023-12-06 10:41:03 +02:00
Georgi GerganovandGitHub fbbc42827b ggml : reuse ggml_get_n_tasks() in ggml_graph_plan() (#4308) 
* ggml : fix soft max out-of-bounds access

ggml-ci

* ggml : reuse ggml_get_n_tasks() in ggml_graph_plan()

ggml-ci
 2023-12-03 15:56:35 +02:00
Georgi GerganovandGitHub adf3de4f69 ggml : fix soft max out-of-bounds access (#4307) 
ggml-ci
 2023-12-03 15:56:22 +02:00
Georgi GerganovandGitHub d7b800b8bc llama : pad KV cache size (#4280) 
* llama : pad KV cache size to 32

* metal : try to improve batched decoding
 2023-12-03 10:58:16 +02:00
Georgi Gerganov 5a7d3125e7 llama : avoid using "optional" keyword (#4283) 2023-12-01 20:39:12 +02:00
Georgi Gerganov d5a1cbde60 llama : support optional tensors (#4283) 2023-12-01 20:35:47 +02:00
Georgi GerganovandGitHub 880f57973b llama : fix integer overflow during quantization (#4284) 
happens with multi-threaded quantization of Qwen-72B

ggml-ci
 2023-12-01 18:42:11 +02:00
Georgi GerganovandGitHub ef47ec18da ggml : add ggml_soft_max_ext (#4256) 
* metal : implement soft_max_ext

* cuda : implement soft_max_ext

* ggml : implement soft_max_ext (CPU)

* batched-bench : print threads

ggml-ci

* metal : simplify soft_max encoding

ggml-ci

* cuda : use 512 threads for soft_max instead of 32

* ggml : update soft max cpu

* cuda : do warp-based block reduce

* cuda : increase max block size to 1024

* cuda : fix warp reduction initialization of shared mem

* metal : warp-based reduction for soft max kernel

* metal : warp-based reduce for rms_norm

* metal : simplify soft max kernel

ggml-ci

* alloc : fix build with debug
 2023-12-01 10:51:24 +02:00
Georgi Gerganov 1f5cd83275 examples : add readme files 2023-11-29 11:00:17 +02:00
Georgi GerganovandGitHub 8406b0924b ggml : re-enable BLAS for CPU when src0 != F32 + remove redundant full offload checks in llama.cpp (#4240) 
* ggml : use blas even if src0 is not F32

* llama : use n_threads_batch only when n_tokens >= 32

ggml-ci

* llama : revert n_threads_batch logic

ggml-ci
 2023-11-28 10:32:03 +02:00
Georgi Gerganov 3e73d31d9c lookahead : support -n -1 infinite generation 2023-11-26 21:52:23 +02:00
Georgi GerganovandGitHub 9656026b53 readme : update hot topics 2023-11-26 20:42:51 +02:00
Georgi GerganovGitHubslaren
 		922754a8d6 lookahead : add example for lookahead decoding (#4207) 
* lookahead : init

* lookahead : generate and store n-grams

* lookahead : use loop instead recursion to generate n-grams

* lookahead : initial working implementation

* lookahead : filter repeating n-grams

* lookahead : use deterministic init

* lookahead : add to Makefile

* lookahead : fix a bug in the seq_id of the lookahead tokens

* lookahead : add comments

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-11-26 20:33:07 +02:00
Georgi GerganovandGitHub 04814e718e readme : update hot topics 2023-11-25 12:02:13 +02:00
Georgi GerganovGitHubkir-gadjelloTobi Lütke
 		af19d35734 server : OAI API compatibility (#4198) 
* Add openai-compatible POST /v1/chat/completions API endpoint to server example

* fix code style

* Update server README.md

* Improve server README.md

* Fix server.cpp code style according to review

* server : some style changes

* server : indentation

* server : enable special tokens during tokenization by default

* server : minor code style

* server : change random string generator

* straightforward /v1/models endpoint

---------

Co-authored-by: kir-gadjello <111190790+kir-gadjello@users.noreply.github.com>
Co-authored-by: Tobi Lütke <tobi@Tobis-MacBook-Pro.local>
 2023-11-25 11:29:06 +02:00
Georgi GerganovGitHubKerfuffle
 		6b0a7420d0 llama : KV cache view API + better KV cache management (#4170) 
* llama : keep track of used KV cells + better KV cache management

* llama : zero KV cache used upon clear

ggml-ci

* llama : allow exporting a view of the KV cache (#4180)

* Allow exporting a view of the KV cache

* Allow dumping the sequences per cell in common

* Track max contiguous cells value and position as well

* Fix max contiguous empty cells index calculation

Make dump functions deal with lengths or sequences counts > 10 better

* Fix off by one error in dump_kv_cache_view

* Add doc comments for KV cache view functions

Eliminate cell sequence struct; use llama_seq_id directly

Minor cleanups

* common : add -dkvc arg for enabling kv cache dumps

---------

Co-authored-by: Kerfuffle <44031344+KerfuffleV2@users.noreply.github.com>
 2023-11-23 19:07:56 +02:00
Georgi GerganovandGitHub d103d935c0 readme : update hot topics 2023-11-23 13:51:22 +02:00
Georgi Gerganov ff8238f71d docs : add llama-star arch idea 2023-11-23 11:35:04 +02:00
Georgi Gerganov dae06c06e5 Revert "finetune : add --n-gpu-layers flag info to --help (#4128)" 
This reverts commit 05e8301e45.
 2023-11-19 19:16:07 +02:00
Georgi Gerganov 35985acffa gitignore : tokenize 2023-11-19 18:50:49 +02:00
Georgi Gerganov 5ad387e994 tokenize : fix trailing whitespace 2023-11-17 18:01:38 +02:00
Georgi GerganovandGitHub 4f447a4833 llama : fix data units (#4101) 
* llama : fix data units

ggml-ci

* Revert "llama : fix data units"

This reverts commit f5feac831fe225ed7f3db938d115732a49dccfc4.

* llama : disambiguate data units

ggml-ci
 2023-11-17 10:00:15 +02:00
Georgi GerganovandGitHub 3d68f364f1 ggml : sync (im2col, GPU conv, 32-bit arm compat) (#4060) 
ggml-ci
 2023-11-13 16:55:52 +02:00
Georgi GerganovandGitHub c049b37d7b readme : update hot topics 2023-11-13 14:18:08 +02:00
Georgi GerganovandGitHub 4760e7cc0b sync : ggml (backend v2) (#3912) 
* sync : ggml (backend v2) (wip)

* sync : migrate examples and llama.cpp to dynamic graphs (wip)

* sync : update tests + fix max op params to 64

ggml-ci

* sync : ggml-cuda

ggml-ci

* llama : fix save/load state context size

ggml-ci

* sync : try to fix build on tvOS

* sync : pass custom graph sizes in training examples

* sync : update graph copies to new ggml API

* sync : update sync-ggml.sh with new files

* scripts : fix header in sync script

* train : fix context size calculations

* llama : increase inference graph size up to 4096 nodes

* train : allocate grads for backward graphs

* train : allocate grads for gb_tmp
 2023-11-13 14:16:23 +02:00
Georgi GerganovandGitHub 413503d4b9 make : do not add linker flags when compiling static llava lib (#3977) 2023-11-07 20:25:32 +03:00
Georgi Gerganov 8f961abdc4 speculative : change default p_accept to 0.5 + CLI args (#3919) 
ggml-ci
 2023-11-03 09:41:56 +02:00
Georgi GerganovandGitHub 05816027d6 common : YAYF (yet another YARN fix) (#3925) 
ggml-ci
 2023-11-03 09:24:00 +02:00
Georgi GerganovandGitHub 224e7d5b14 readme : add notice about #3912 2023-11-02 20:44:12 +02:00
Georgi GerganovandGitHub c7743fe1c1 cuda : fix const ptrs warning causing ROCm build issues (#3913) 2023-11-02 20:32:11 +02:00
Georgi GerganovandGitHub 4ff1046d75 gguf : print error for GGUFv1 files (#3908) 2023-11-02 16:22:30 +02:00
Georgi GerganovandGitHub 2756c4fbff gguf : remove special-case code for GGUFv1 (#3901) 
ggml-ci
 2023-11-02 11:20:21 +02:00
Georgi Gerganov 1efae9b7dc llm : prevent from 1-D tensors being GPU split (#3697) 2023-11-02 09:54:44 +02:00
Georgi GerganovandGitHub 4d719a6d4e cuda : check if this fixes Pascal card regression (#3882) 2023-11-02 08:35:10 +02:00
Georgi GerganovandGitHub 183b3fac6c metal : fix build errors and kernel sig after #2268 (#3898) 2023-11-02 08:33:37 +02:00
Georgi Gerganov c43c2da8af llm : fix llm_build_kqv taking unused tensor (benign, #3837) 2023-11-01 23:08:30 +02:00
Georgi Gerganov 523e49b111 llm : fix falcon norm after refactoring (#3837) 2023-11-01 23:00:50 +02:00
Georgi GerganovandGitHub e16b9fa4ba metal : multi-simd softmax (#3710) 
ggml-ci
 2023-11-01 21:25:00 +02:00
Georgi Gerganov ff8f9a88da common : minor (#3715) 2023-11-01 21:15:55 +02:00
Georgi GerganovandGitHub 50337961a6 llm : add llm_build_context (#3881) 
* llm : add llm_build_context

* llm : deduce norm eps based on type + explict max_alibi_bias, clamp_kqv

* llm : restore the non-graph llm_build_ functional API

ggml-ci

* llm : cleanup + comments
 2023-11-01 20:11:02 +02:00
Georgi Gerganov 9a3b4f6c86 ggml : fix UNUSED macro (#3762) 2023-11-01 13:50:45 +02:00
Georgi GerganovandGitHub f0e209324a scripts : add server-llm.sh (#3868) 
* scripts : add deploy-server.sh

* scripts : rename to server-llm.sh

* scripts : working curl pipe
 2023-11-01 11:29:07 +02:00
Georgi GerganovandGitHub 71e3718abd llama : refactor graph build code (#3837) 
* llama : factor out ggml-alloc from graph graph build functions

ggml-ci

* metal : disable kernel load log

* llama : factor out tensor offloading outside the build call (wip)

ggml-ci

* llama : offload rest of the models

ggml-ci

* llama : update offload log messages to print node index

* llama : comments

* llama : support offloading result_norm + comments

* llama : factor graph input into a function

* llama : do tensor offload only with CUDA

* llama : fix res_norm offloading

* llama : try to optimize offloading code

* llama : fix non-CUDA build

* llama : try to fix build

* llama : move refact in correct place + optimize graph input

* llama : refactor tensor offloading as callback

* llama : add layer index to all tensor names

* llama : add functional header

* llama : comment

ggml-ci

* llama : remove obsolete map for layer counting

* llama : add llm_build helper functions (#3848)

* llama : add llm_build_norm helper function

ggml-ci

* llama : add llm_build_ffn helper function (#3849)

ggml-ci

* llama : add llm_build_k_shift helper

ggml-ci

* llama : fix offloading after recent changes

* llama : add llm_build_kv_store helper

ggml-ci

* llama : remove obsolete offload names

* llama : fix llm_build_k_shift to use n_head_kv instead of n_head

* llama : simplify falcon Q, K, V computation

* llama : remove obsolete comments in build graphs

* llama : add llm_build_kqv helper

ggml-ci

* llama : minor

* llama : add LLAMA_OFFLOAD_DEBUG + fix starcoder offloading

* llama : fix input allocation logic

* llama : update offload functions for KQ tensors

* llama : normalize tensor names

ggml-ci

* llama : enable warning about not offloaded tensors

* llama : remove extra ; + deduplicate gate_b logic

* llama : add llm_build_inp_embd helper
 2023-11-01 08:04:02 +02:00
Georgi GerganovandGitHub 207b51900e ggml : move FP16 <-> FP32 code to ggml-impl.h (#3861) 
* ggml : move FP16 <-> FP32 stuff to ggml-impl.h

ggml-ci

* tests : fix ARM build

* ggml : explicitly initialize deprecated type traits

* ggml : add math.h to ggml-impl.h

* ggml : remove duplicate static assert macros

* ggml : prefix lookup tables with ggml_

ggml-ci

* ggml-impl : move extern "C" to start of file
 2023-10-30 19:19:15 +02:00
Georgi GerganovandGitHub 71a09da301 llama : fix kv shift bug (#3835) 
ggml-ci
 2023-10-29 18:32:51 +02:00
Georgi GerganovGitHubcebtenzzre
 		d69d777c02 ggml : quantization refactoring (#3833) 
* ggml : factor all quantization code in ggml-quants

ggml-ci

* ggml-quants : fix Zig and Swift builds + quantize tool

ggml-ci

* quantize : --pure option for disabling k-quant mixtures

---------

Co-authored-by: cebtenzzre <cebtenzzre@gmail.com>
 2023-10-29 18:32:28 +02:00
Georgi Gerganov ba231e8a6d issues : change label from bug to bug-unconfirmed (#3748) 2023-10-28 15:35:26 +03:00
Georgi GerganovandGitHub 8a2f2fea29 convert : ignore tokens if their IDs are within [0, vocab_size) (#3831) 2023-10-28 06:25:15 -06:00
Georgi GerganovandGitHub ee1a0ec9cb llama : add option for greedy sampling with probs (#3813) 
* llama : add option for greedy sampling with probs

* llama : add comment about llama_sample_token_greedy() missing probs

* sampling : temp == 0.0 -> no probs, temp < 0.0 -> probs
 2023-10-28 14:23:11 +03:00
Georgi GerganovandGitHub fdee152e4e starcoder : add GPU offloading (#3827) 
* starcoder : do not GPU split 1D bias tensors

* starcoder : offload layers to GPU

ggml-ci
 2023-10-28 12:06:08 +03:00
Georgi GerganovandGitHub 2f9ec7e271 cuda : improve text-generation and batched decoding performance (#3776) 
* cuda : prints wip

* cuda : new cublas gemm branch for multi-batch quantized src0

* cuda : add F32 sgemm branch

* cuda : fine-tune >= VOLTA params + use MMQ only for small batches

* cuda : remove duplicated cuBLAS GEMM code

* cuda : add CUDA_USE_TENSOR_CORES and GGML_CUDA_FORCE_MMQ macros

* build : add compile option to force use of MMQ kernels
 2023-10-27 17:01:23 +03:00
Georgi Gerganov 34b2a5e1ee server : do not release slot on image input (#3798) 2023-10-26 22:54:17 +03:00
Georgi Gerganov 6961c4bd0b batched-bench : print params at start 2023-10-25 10:26:27 +03:00
Georgi Gerganov cc44877486 log : disable pid in log filenames 2023-10-25 10:09:16 +03:00
Georgi GerganovandGitHub 1717521cdb server : do not block system prompt update (#3767) 
* server : do not block system prompt update

* server : update state machine logic to process system prompts

* server : minor
 2023-10-24 23:08:20 +03:00
Georgi GerganovandGitHub b2f7e04bd3 sync : ggml (conv ops + cuda MSVC fixes) (#3765) 
ggml-ci
 2023-10-24 21:51:20 +03:00
Georgi GerganovGitHubKerfuffle
 		2b4ea35e56 cuda : add batched cuBLAS GEMM for faster attention (#3749) 
* cmake : add helper for faster CUDA builds

* batched : add NGL arg

* ggml : skip nops in compute_forward

* cuda : minor indentation

* cuda : batched cuBLAS GEMMs for src0 F16 and src1 F32 (attention ops)

* Apply suggestions from code review

These changes plus:

```c++
#define cublasGemmBatchedEx hipblasGemmBatchedEx
```

are needed to compile with ROCM. I haven't done performance testing, but it seems to work.

I couldn't figure out how to propose a change for lines outside what the pull changed, also this is the first time trying to create a multi-part review so please forgive me if I mess something up.

* cuda : add ROCm / hipBLAS cublasGemmBatchedEx define

* cuda : add cublasGemmStridedBatchedEx for non-broadcasted cases

* cuda : reduce mallocs in cublasGemmBatchedEx branch

* cuda : add TODO for calling cublas from kernel + using mem pool

---------

Co-authored-by: Kerfuffle <44031344+KerfuffleV2@users.noreply.github.com>
 2023-10-24 16:48:37 +03:00
Georgi Gerganov 469c9addef metal : handle ggml_scale for n%4 != 0 (close #3754) 
ggml-ci
 2023-10-24 09:47:22 +03:00
Georgi Gerganov e3932593d4 Revert "make : add optional CUDA_NATIVE_ARCH (#2482)" 
This reverts commit 96981f37b1.

See:

https://github.com/ggerganov/llama.cpp/pull/2482#issuecomment-1775975866
 2023-10-23 23:46:05 +03:00
Georgi GerganovGitHubFSSRepoDamian StewartSteward GarciaJhen-Jie HongM. Yusuf Sarıgöz
 		438c2ca830 server : parallel decoding and multimodal (#3677) 
* implementing parallel decoding in server example

* crash fixed

* save dev progress

* refactored sampling function

* completion endpoint working

* multiple client support

* grammar + no stream completion

* cached prompt support

* chat.mjs support cached prompt + some fixes

* server ui now support multiple clients

* unused change reverted

* fixed timings per slot

* add context swap

* add changes to README.md

* llava multimodal integration

* fixed tokens probs

* add multimodal input - alfa

* refactor code + remove unused comments + improved README.md

* fix compilation errors with llvm

* notify the user from server ui that multimodality is unavialable

* some ci fixes

* fix ci make build undefined ref errors

* fix long prompt than ctx proposed in #3639

* fixed premature end due stop word

* context shift fixed

* fix llava implementation

* sync README.md changes

* readme change

* update api like OpenAI

* multimodal support enabled by default

* fix make bui;d errors

* fix multiple clients

* fix zig build

* new sampling API

* latest changes of sampling API

* server : coding-style normalization

* server : coding-style normalization (part 2)

* server : remove beam-search functionality

* server : bug fix in ingest_images

n_tokens is incremented internally by llama_batch_add

* server : use refs + use llama_batch_clear()

* server : snake case

* server : minor sync

* added thread safe pipeline

* server : bach has to be allocated for n_parallel sequences

* server : no need for atomic int - already using mutex

* server : logs + minor code style

* server : fix multibyte handle in partial response (#3706)

* fix image load + view image in chat

* make : silence stb warnings

* clip : link to ggml, not to llama

* server : fix switch fallthrough

* server : fix crash in Debug on macOS (I have no idea why this fixes it!?)

* server : refactor ctx_sampling init + n_ctx + names

* server : bug fix for prompt caching

* Do not save/load image_data to localStorage

* editorconfig : new line in index.html

* server : completion requests remember slot_id

* Update readme to document multimodal in server

* server : minor style

* Update readme to document multimodal in server

* server : hide ctx_sampling->prev behind API (#3696)

* server : apply fix from #3722

* server : fix slot reuse

* server : add comment about changing slot_state to bool

---------

Co-authored-by: FSSRepo <go778sgt@gmail.com>
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Steward Garcia <57494570+FSSRepo@users.noreply.github.com>
Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
Co-authored-by: M. Yusuf Sarıgöz <yusufsarigoz@gmail.com>
 2023-10-22 22:53:08 +03:00
Georgi Gerganov 22c69a2794 batched : add len CLI argument 2023-10-22 08:37:20 +03:00
Georgi GerganovandGitHub d1031cf49c sampling : refactor init to use llama_sampling_params (#3696) 
* sampling : refactor init to use llama_sampling_params

* llama : combine repetition, frequency and presence penalties in 1 call

* examples : remove embd-input and gptneox-wip

* sampling : rename penalty params + reduce size of "prev" vector

* sampling : add llama_sampling_print helper

* sampling : hide prev behind API and apply #3661

ggml-ci
 2023-10-20 21:07:23 +03:00
Georgi Gerganov a0edf73bda server : fix uninitialized sampling context (close #3685) 2023-10-20 13:06:10 +03:00
Georgi GerganovandGitHub 004797f6ac readme : update hot topics 2023-10-18 21:44:43 +03:00
Georgi Gerganov 4e82b2ea3f speculative : bug fixes 2023-10-18 18:49:40 +03:00
Georgi GerganovandGitHub 0e89203b51 speculative : add tree-based sampling example (#3624) 
* sampling : one sequence per sampling context

ggml-ci

* speculative : add tree-based sampling support

ggml-ci

* speculative : reuse the n_parallel CLI param

* speculative : refactor sampling

* examples : fix build after sampling refactoring

ggml-ci

* batched : fix n_seq_id

* sampling : fix malloc

ggml-ci

* swift : fix build

ggml-ci

* swift : try to fix build

ggml-ci

* prompts : add assistant.txt

* common : add llama_batch_add() and llama_batch_clear() helpers

* speculative : minor refactor

ggml-ci

* minor : comments + rename

ggml-ci

* speculative : fix off-by-one for n_drafted

* speculative : fix the n_drafted fix + p constants
 2023-10-18 16:21:57 +03:00
Georgi Gerganov e1675d133c llama : avoid fprintf in favor of LLAMA_LOG (#3538) 2023-10-17 22:34:26 +03:00
Georgi Gerganov e74c705e15 editorconfig : remove trailing spaces 2023-10-17 19:52:53 +03:00
Georgi GerganovandGitHub 1142013da4 save-load-state : fix example + add ci test (#3655) 
* save-load-state : fix example (close #3606)

* ci : add test for save-load-state example

ggml-ci
 2023-10-17 19:12:46 +03:00
Georgi GerganovandGitHub 281ef73c25 k-quants : fix quantization ranges (#3646) 2023-10-17 09:19:28 +03:00
Georgi GerganovGitHubM. Yusuf Sarıgöz
 		940efa95fe llava : fix tokenization to not add bos between image embeddings and user prompt (#3645) 
* llava : fix tokenization to not add bos after system prompt

* set seed

---------

Co-authored-by: M. Yusuf Sarıgöz <yusufsarigoz@gmail.com>
 2023-10-16 23:58:00 +03:00
Georgi Gerganov d28e572c02 cmake : fix add_compile_options on macOS 2023-10-12 14:31:05 +03:00
Georgi GerganovandGitHub 1a8c8795d6 ci : check if there is enough VRAM (#3596) 
ggml-ci
 2023-10-12 13:44:56 +03:00
Georgi Gerganov 6b3ae4da92 prompts : add mnemonics.txt 2023-10-12 09:35:30 +03:00
Georgi GerganovandGitHub 57dd55e2c7 server : fix kv cache management (#3588) 2023-10-12 09:29:04 +03:00
Georgi Gerganov b8fe4b5cc9 main : fix session loading bug (#3400) 2023-10-11 23:55:41 +03:00
Georgi GerganovandGitHub 8c70a5ff25 batched : add bench tool (#3545) 
* batched : add bench tool

* batched : minor fix table

* batched-bench : add readme + n_kv_max is now configurable

* batched-bench : init warm-up batch

* batched-bench : pass custom set of PP, TG and PL

* batched-bench : add mmq CLI arg
 2023-10-11 21:25:33 +03:00
Georgi GerganovandGitHub fcca0a7004 refact : fix convert script + zero out KV cache to avoid nans (#3523) 
* refact : fix convert script + zero out KV cache to avoid nans

* ggml : silu(-inf) should never happen

* metal : assert various kernel requirements
 2023-10-09 14:32:17 +03:00
Georgi GerganovandGitHub dcc09d2596 metal : do not use mul_mm kernels when ne00 < 64 (#3542) 2023-10-09 14:28:27 +03:00
Georgi GerganovandGitHub db3abcc114 sync : ggml (ggml-backend) (#3548) 
* sync : ggml (ggml-backend)

ggml-ci

* zig : add ggml-backend to the build
 2023-10-08 20:19:14 +03:00
Georgi GerganovandGitHub 94e502dfb7 ci : enable on obj-c changes + fix metal build (#3540) 2023-10-08 11:24:50 +03:00
Georgi GerganovandGitHub b0ec5218c3 metal : support MTLGPUFamily < Apple7, formatting, style (#3524) 
* metal : improve decoding speed for batches of 2-16

* metal : rename kernels mul_mat_ to mul_mv_

* metal : indentations

* minor

* metal : print more GPU info + disable mul_mm for MTLGPUFamiliy < Apple7
 2023-10-08 10:01:53 +03:00
Georgi Gerganov 0c731ca403 prompts : fix editorconfig checks after #3416 2023-10-06 16:36:32 +03:00
Georgi GerganovandGitHub beabc8cfb0 readme : add project status link 2023-10-04 16:50:44 +03:00
Georgi Gerganov 0d152b37fe ggml : fix build after #3329 2023-10-04 16:25:41 +03:00
Georgi GerganovandGitHub f93af02488 sync : ggml (conv 1d + 2d updates, UB fixes) (#3468) 
* sync : ggml (conv 1d + 2d updates)

ggml-ci

* ggml : fix UB in q5_0 and q5_1 quantize code

ggml.c:1033:39: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior

ggml.c:1081:39: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior

ggml-ci

* tests : fix UB in test-quantize-perf
 2023-10-04 15:29:58 +03:00
Georgi GerganovandGitHub ac2219fef3 llama : fix session saving/loading (#3400) 
* llama : fix session saving/loading

* llama : temp fix for clearing "future" tokens from the KV cache

* llama : fix handling of "future" tokens when loading sessions

* llama : fix comments for llama_kv_cache API
 2023-10-03 21:04:01 +03:00
Georgi GerganovGitHubxaedes
 		bc34dd4f5b train : fix KQ_pos allocation (#3392) 
* train : fix KQ_pos allocation

* make sure KQ_pos is not reallocated in finetune

---------

Co-authored-by: xaedes <xaedes@gmail.com>
 2023-09-29 19:05:18 +03:00
Georgi GerganovandGitHub 2619109ad5 ci : disable freeBSD builds due to lack of VMs (#3381) 2023-09-28 19:36:36 +03:00
Georgi GerganovGitHubslaren
 		ec893798b7 llama : custom attention mask + parallel decoding + no context swaps (#3228) 
* tests : verify that RoPE is "additive"

* llama : replace ggml_diag_mask_inf with ggml_add (custom -inf mask)

* ggml : ggml_rope now takes a vector with positions instead of n_past

* metal : add rope_f16 kernel + optimize cpy kernels

* llama : unified KV cache + batch inference API

* llama : add new llama_decode() API that works with llama_batch

* llama : add cell_max heuristic for more efficient kv_cache

* llama : extend llama_kv_cache API

* llama : more robust cell_max heuristic + wip shift

* metal : disable concurrency optimization

* llama : add llama_kv_cache_shift_seq + no more context swaps

* llama : apply K-cache roping for Falcon and Baichuan

* speculative : fix KV cache management

* parallel : example for serving multiple users in parallel

* parallel : disable hot-plug to avoid cache fragmentation

* fixes : speculative KV cache + llama worst-case graph

* llama : extend batch API to select which logits to output

* llama : fix worst case graph build

* ggml-cuda : update rope implementation for parallel decoding (#3254)

* ggml-cuda : update rope implementation for parallel decoding

* better solution for p0 computation

* fix rope

* simpler rope implementation

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* make : add parallel to build + fix static functions in llama.cpp

* simple : fix token counting

* parallel : various improvements

* llama : fix cell_max logic + rename functions

* parallel : try smaller batches when the KV cache is fragmented

* parallel : fix sequence termination criteria

* llama : silence errors KV cache errors

* parallel : remove new line from prompt

* parallel : process system prompt once + configurable paramters + llama API

* parallel : remove question with short answers

* parallel : count cache misses

* parallel : print misses on each request

* parallel : minor

* llama : fix n_kv to never become 0

* parallel : rename hot-plug to continuous-batching

* llama : improve llama_batch API + simplify parallel example

* simple : add parallel decoding support

* simple : improve comments + free batch

* ggml-cuda : add rope f16, restore performance with parallel decoding (#3272)

* ggml-cuda : add rope f16, restore performance

* offload KQ_mask with all models

* fix rope shift

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* llama : disable MPI for now

ggml-ci

* train : make KQ_pos memory buffer permanent via dummy scale op

* ggml : revert change to ggml_cpy, add ggml_cont_Nd instead (#3275)

ggml-ci

* parallel : fix bug (extra BOS) + smaller token_prev array

* parallel : fix cases where the input prompts can overflow the batch

* parallel : add disabled experimental batch chunking in powers of two

* llama : llama.h formatting + comments

* simple : add README.md

* llama : fix kv cache heuristic when context is less than 32

* parallel : fix crash when `-n -1`

* llama : simplify returns if/else branches

* metal : use mm kernels for batch size > 2

* examples : utilize new llama_get_logits_ith()

* examples : add example for batched decoding

* examples : do not eval prompt 2 times (close #3348)

* server : clear the KV cache beyond n_past before llama_decode

* server : avoid context swaps by shifting the KV cache

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-09-28 19:04:36 +03:00
Georgi GerganovandGitHub 7eb41179ed readme : update hot topics 2023-09-20 20:48:22 +03:00
Georgi GerganovandGitHub d119c04c15 examples : fix benchmark-matmult (#1554) 
The precision for Q4_0 has degraded since #1508
 2023-09-20 10:02:39 +03:00
Georgi GerganovandGitHub c6f1491da0 metal : fix bug in soft_max kernels (out-of-bounds access) (#3194) 2023-09-15 20:17:24 +03:00
Georgi GerganovandGitHub 8c00b7a6ff sync : ggml (Metal F32 support + reduce ggml-alloc size) (#3192) 
* sync : ggml (Metal F32 support + reduce ggml-alloc size)

ggml-ci

* llama-bench : fix ggml_cpu_has_metal() duplicate function

ggml-ci
 2023-09-15 19:06:03 +03:00
Georgi GerganovandGitHub a51b687657 metal : relax conditions on fast matrix multiplication kernel (#3168) 
* metal : relax conditions on fast matrix multiplication kernel

* metal : revert the concurrnecy change because it was wrong

* llama : remove experimental stuff
 2023-09-15 11:09:24 +03:00
Georgi GerganovandGitHub 94f10b91ed readme : update hot tpoics 2023-09-08 18:18:04 +03:00
Georgi GerganovandGitHub b3e9852e47 sync : ggml (CUDA GLM RoPE + POSIX) (#3082) 
ggml-ci
 2023-09-08 17:58:07 +03:00