nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 23 Packages Projects Releases Wiki Activity
1,847 Commits 1 Branch 0 Tags
f238461236f4e0e18cac1a554af23c7deadc9b01
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Georgi Gerganov f238461236 ggml : fix 32-bit ARM compat for IQ2_XS (whisper/1758) 
* ggml : fix 32-bit ARM compat

* ggml : fix fix

* ggml : fix fix fix
 2024-01-12 22:02:11 +02:00
Georgi Gerganov 584d674be6 llama : remove redundant assert for StableLM (#4901) 2024-01-12 20:54:12 +02:00
Georgi Gerganov 5537d9d36b gitignore : imatrix 2024-01-12 14:33:21 +02:00
Georgi Gerganov 3cabe80630 llama : fix typo "imp_embd" -> "inp_embd" 2024-01-12 13:11:15 +02:00
Georgi Gerganov 2d00741e12 py : fix lint (#4889) 2024-01-12 13:03:38 +02:00
Georgi GerganovandGitHub f445c0e68c llama : fix llm_build_k_shift to use correct n_rot (#4889) 
* llama : fix llm_build_k_shift to use correct n_rot

ggml-ci

* llama : always use hparams.n_rot for ggml_rope_custom

ggml-ci

* convert : fix persimmon conversion to write correct n_rot
 2024-01-12 13:01:56 +02:00
Georgi GerganovandGitHub 1d118386fe server : fix infill when prompt is empty (#4833) 2024-01-11 23:23:49 +02:00
Georgi Gerganov 7edefbd79c main : better name for variable n_print (#4874) 2024-01-11 22:46:26 +02:00
Georgi Gerganov 3ca63b4538 main : disable token count by default (#4874) 2024-01-11 22:43:05 +02:00
Georgi Gerganov b037787548 swift : track ggml release branch (#4867) 2024-01-11 21:58:28 +02:00
Georgi GerganovandGitHub 3ba5b8ca8e swift : pin ggml commit + remove ggml.h from spm-headers (#4878) 
ggml-ci
 2024-01-11 21:31:31 +02:00
Georgi Gerganov 64802ec00d sync : ggml 2024-01-11 09:39:08 +02:00
Georgi Gerganov 3267c2abc7 metal : fix deprecation warning (ggml/690) 2024-01-11 09:39:05 +02:00
Georgi GerganovandGitHub 5c1980d8d4 server : fix build + rename enums (#4870) 2024-01-11 09:10:34 +02:00
Georgi GerganovandGitHub 4dccb38d9a metal : improve dequantize precision to match CPU (#4836) 
ggml-ci
 2024-01-09 19:37:08 +02:00
Georgi Gerganov 9a818f7c42 scripts : improve get-pg.sh (#4838) 2024-01-09 19:21:13 +02:00
Georgi GerganovandGitHub d9653894df scripts : script to get Paul Graham essays in txt format (#4838) 2024-01-09 16:23:05 +02:00
Georgi GerganovandGitHub 18c2e1752c ggml : fix vld1q_s8_x4 32-bit compat (#4828) 
* ggml : fix vld1q_s8_x4 32-bit compat

ggml-ci

* ggml : fix 32-bit ARM compat (cont)

ggml-ci
 2024-01-09 10:42:06 +02:00
Georgi GerganovandGitHub a9a8c5de3d readme : add link to SOTA models 2024-01-08 20:25:17 +02:00
Georgi GerganovandGitHub 668b31fc7d swift : exclude ggml-metal.metal from the package (#4822) 2024-01-08 16:40:51 +02:00
Georgi GerganovandGitHub 42ea63c5a3 llama.swiftui : update readme 2024-01-08 15:57:36 +02:00
Georgi GerganovandGitHub 52531fdff8 main : add self-extend support (#4815) 
* examples : add passkey test

* passkey : better prints

* passkey : select pass key pos from CLI

* passkey : simplify n_past logic

* llama : "self-extend"-like context extension

* passkey : add comment

* main : add Self-Extend support

* llama : add comment about llama_kv_cache_seq_div
 2024-01-08 11:18:32 +02:00
Georgi GerganovandGitHub b0034d93ce examples : add passkey test (#3856) 
* examples : add passkey test

* passkey : better prints

* passkey : select pass key pos from CLI

* passkey : simplify n_past logic

* make : add passkey target

* passkey : add "self-extend"-like context extension (#4810)

* llama : "self-extend"-like context extension

* passkey : add comment

* passkey : add readme
 2024-01-08 11:14:04 +02:00
Georgi Gerganov 9dede37d81 llama : remove unused vars (#4796) 2024-01-07 14:29:36 +02:00
Georgi GerganovandGitHub 3c36213df8 llama : remove redundant GQA check (#4796) 2024-01-07 11:21:53 +02:00
Georgi Gerganov d117d4dc5d llama : print tensor meta for debugging 2024-01-07 09:51:12 +02:00
Georgi GerganovandGitHub 67984921a7 server : fix n_predict check (#4798) 2024-01-07 08:45:26 +02:00
Georgi Gerganov 96e80dabc6 examples : improve base-translate.sh script (#4783) 2024-01-06 11:40:24 +02:00
Georgi Gerganov 91d38876df metal : switch back to default.metallib (ggml/681) 
ggml-ci
 2024-01-05 18:02:06 +02:00
Georgi Gerganov d061bf9405 ggml : fix q2_k bpw in comments (ggml/680) 2024-01-05 18:02:06 +02:00
Georgi GerganovandGitHub c1d7cb28d3 ggml : do not sched_yield when calling BLAS (#4761) 
* ggml : do not sched_yield when calling BLAS

ggml-ci

* ggml : fix do_yield logic

ggml-ci

* ggml : simplify do_yield logic

ggml-ci
 2024-01-05 15:18:21 +02:00
Georgi GerganovandGitHub 3681f22443 examples : add few-shot translation example (#4783) 2024-01-05 15:11:10 +02:00
Georgi GerganovandGitHub 012cf349ae server : send token probs for "stream == false" (#4714) 2024-01-04 19:56:33 +02:00
Georgi GerganovandGitHub dc891b7f7a ggml : include stdlib.h before intrin.h (#4736) 2024-01-04 10:12:26 +02:00
Georgi Gerganovandslaren 7bed7eba35 cuda : simplify expression 
Co-authored-by: slaren <slarengh@gmail.com>
 2024-01-03 14:38:38 +02:00
Georgi Gerganov d55356d3ba cuda : mark I16 and I32 ops as unsupported 
ggml-ci
 2024-01-03 14:38:38 +02:00
Georgi Gerganov 75e3fd8581 sync : ggml 
ggml-ci
 2024-01-03 14:38:38 +02:00
Georgi Gerganov 289313716f metal : add kernel_get_rows_i32 
ggml-ci
 2024-01-03 14:38:38 +02:00
Georgi Gerganov ab62fc3e55 scripts : fix sync order + metal sed 2024-01-03 14:38:38 +02:00
Georgi GerganovandGitHub f3f62f0d83 metal : optimize ggml_mul_mat_id (faster Mixtral PP) (#4725) 
* ggml : disable fast-math for Metal (cmake build only)

ggml-ci

* metal : fix Metal API debug warnings

* cmake : add -fno-inline for Metal build (#4545)

* metal : fix API debug warnings

* metal : fix compile warnings

* metal : use uint64_t for strides

* cmake : rename option to LLAMA_METAL_SHADER_DEBUG

* metal : fix mat-vec Q8_0 kernel for BS > 1

* metal : normalize mat-vec kernel signatures

* cmake : respect LLAMA_QKK_64 option

* metal : fix mat-vec Q4_K kernel for QK_K == 64

* metal : optimizing ggml_mul_mat_id (wip)

* metal : minor fix

* metal : opt mul_mm_id
 2024-01-02 21:07:47 +02:00
Georgi Gerganov 540938f890 llama : llama_model_desc print number of experts 2024-01-02 16:26:45 +02:00
Georgi Gerganov 32866c5edd editorconfig : fix whitespace and indentation #4710 2024-01-02 13:28:15 +02:00
Georgi GerganovandGitHub 58ba655af0 metal : enable shader debugging (cmake option) (#4705) 
* ggml : disable fast-math for Metal (cmake build only)

ggml-ci

* metal : fix Metal API debug warnings

* cmake : add -fno-inline for Metal build (#4545)

* metal : fix API debug warnings

* metal : fix compile warnings

* metal : use uint64_t for strides

* cmake : rename option to LLAMA_METAL_SHADER_DEBUG

* metal : fix mat-vec Q8_0 kernel for BS > 1

* metal : normalize mat-vec kernel signatures

* cmake : respect LLAMA_QKK_64 option

* metal : fix mat-vec Q4_K kernel for QK_K == 64

ggml-ci
 2024-01-02 10:57:44 +02:00
Georgi GerganovandGitHub e39106c055 ggml : add ggml_vdotq_s32 alias (#4715) 
ggml-ci
 2023-12-31 11:43:31 +02:00
Georgi GerganovandGitHub 9fbda719de clip : refactor + bug fixes (#4696) 
* clip : refactor + bug fixes

ggml-ci

* server : add log message
 2023-12-30 23:24:42 +02:00
Georgi Gerganov 0235b9b571 clip : use ggml_backend_buffer_is_host (#4205) 2023-12-29 18:53:34 +02:00
Georgi Gerganov c8255f8a6b scripts : print list of sync commits 2023-12-29 15:12:35 +02:00
Georgi Gerganov 38b3de4658 sync : ggml 2023-12-29 14:56:41 +02:00
Georgi Gerganov ca38b8d334 scripts : do not sync commits from this repo 2023-12-29 14:54:05 +02:00
Georgi Gerganov b47879b0dd scripts : add sync-ggml-am.sh 2023-12-27 11:44:22 +02:00
Georgi GerganovandGitHub 951010fa53 ggml : fix dot product for ARM (#4630) 
ggml-ci
 2023-12-27 11:02:13 +02:00
Georgi GerganovGitHubleejet
 		ba66175132 sync : ggml (fix im2col) (#4591) 
* cuda : fix im2col_f32_f16 (ggml/#658)

ggml-ci

* ggml-alloc : fix ggml_tallocr_is_own

---------

Co-authored-by: leejet <leejet714@gmail.com>
 2023-12-22 17:53:43 +02:00
Georgi GerganovandGitHub afefa319f1 ggml : change ggml_scale to take a float instead of tensor (#4573) 
* ggml : change ggml_scale to take a float instead of tensor

* ggml : fix CPU implementation

* tests : fix test-grad0

ggml-ci
 2023-12-21 23:20:49 +02:00
Georgi GerganovandGitHub 769a7bc85e gguf-py : fix broken link 2023-12-21 23:20:36 +02:00
Georgi Gerganov 32259b2dad gguf : simplify example dependencies 2023-12-21 23:08:14 +02:00
Georgi GerganovandGitHub c083718c89 readme : update coding guidelines 2023-12-21 19:27:14 +02:00
Georgi Gerganov 0e18b2e7d0 llama.swiftui : add tinyllama 1.1B F16 2023-12-18 20:17:43 +02:00
Georgi Gerganov 6ff39b129d llama.swiftui : add more models 2023-12-18 20:05:12 +02:00
Georgi GerganovandGitHub b1306c4394 readme : update hot topics 2023-12-17 20:16:23 +02:00
Georgi GerganovGitHubjhen
 		800a489e4a llama.swiftui : add bench functionality (#4483) 
* llama.swiftui : add bench button

* llama.swiftui : initial bench functionality

* force to use n_gpu_layers on simulator

* add download buttons & expose llamaState.loadModel

* update project.pbxproj

* comment #Preview & fix editorconfig check

* gitignore : xcode stuff

* llama.swiftui : UX improvements

* llama.swiftui : avoid data copy via "downloadTask"

* llama.swiftui : remove model from project

* llama : remove "mostly" from model infos

* llama.swiftui : improve bench

---------

Co-authored-by: jhen <developer@jhen.me>
 2023-12-17 19:38:41 +02:00
Georgi Gerganov 55e87c3749 ggml : fix OpenCL broadcast requirement for ggml_mul (close #4453) 2023-12-14 10:35:29 +02:00
Georgi GerganovGitHubslaren
 		4d98d9a656 sync : ggml (SD ops, tests, kernels) (#4444) 
* sync : ggml (SD ops, tests, kernels)

ggml-ci

* cuda : restore im2col

ggml-ci

* metal : fix accuracy of dequantization kernels

ggml-ci

* cuda : restore correct im2col

ggml-ci

* metal : try to fix moe test by reducing expert size

ggml-ci

* cuda : fix bin bcast when src1 and dst have different types

ggml-ci

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-12-13 21:54:54 +02:00
Georgi GerganovandGitHub 113f9942fc readme : update hot topics 2023-12-13 14:05:38 +02:00
Georgi GerganovGitHubslaren
 		fe680e3d10 sync : ggml (new ops, tests, backend, etc.) (#4359) 
* sync : ggml (part 1)

* sync : ggml (part 2, CUDA)

* sync : ggml (part 3, Metal)

* ggml : build fixes

ggml-ci

* cuda : restore lost changes

* cuda : restore lost changes (StableLM rope)

* cmake : enable separable compilation for CUDA

ggml-ci

* ggml-cuda : remove device side dequantize

* Revert "cmake : enable separable compilation for CUDA"

This reverts commit 09e35d04b1c4ca67f9685690160b35bc885a89ac.

* cuda : remove assert for rope

* tests : add test-backend-ops

* ggml : fix bug in ggml_concat

* ggml : restore `ggml_get_n_tasks()` logic in `ggml_graph_plan()`

* ci : try to fix macOS

* ggml-backend : remove backend self-registration

* ci : disable Metal for macOS cmake build

ggml-ci

* metal : fix "supports family" call

* metal : fix assert

* metal : print resource path

ggml-ci

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-12-07 22:26:54 +02:00
Georgi GerganovGitHubslaren
 		bcc0eb4591 llama : per-layer KV cache + quantum K cache (#4309) 
* per-layer KV

* remove unnecessary copies

* less code duplication, offload k and v separately

* llama : offload KV cache per-layer

* llama : offload K shift tensors

* llama : offload for rest of the model arches

* llama : enable offload debug temporarily

* llama : keep the KV related layers on the device

* llama : remove mirrors, perform Device -> Host when partial offload

* common : add command-line arg to disable KV cache offloading

* llama : update session save/load

* llama : support quantum K cache (#4312)

* llama : support quantum K cache (wip)

* metal : add F32 -> Q8_0 copy kernel

* cuda : add F32 -> Q8_0 copy kernel

ggml-ci

* cuda : use mmv kernel for quantum cache ops

* llama : pass KV cache type through API

* llama : fix build

ggml-ci

* metal : add F32 -> Q4_0 copy kernel

* metal : add F32 -> Q4_1 copy kernel

* cuda : wip

* cuda : add F32 -> Q4_0 and F32 -> Q4_1 copy kernels

* llama-bench : support type_k/type_v

* metal : use mm kernel only for quantum KV cache

* cuda : add comment

* llama : remove memory_f16 and kv_f16 flags

---------

Co-authored-by: slaren <slarengh@gmail.com>

* readme : add API change notice

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-12-07 13:03:17 +02:00
Georgi GerganovandGitHub 05cd6e5036 server : recognize cache_prompt parameter in OAI API (#4347) 2023-12-06 20:21:59 +02:00
Georgi Gerganov caa9249217 common : fix compile warning 2023-12-06 10:41:03 +02:00
Georgi GerganovandGitHub fbbc42827b ggml : reuse ggml_get_n_tasks() in ggml_graph_plan() (#4308) 
* ggml : fix soft max out-of-bounds access

ggml-ci

* ggml : reuse ggml_get_n_tasks() in ggml_graph_plan()

ggml-ci
 2023-12-03 15:56:35 +02:00
Georgi GerganovandGitHub adf3de4f69 ggml : fix soft max out-of-bounds access (#4307) 
ggml-ci
 2023-12-03 15:56:22 +02:00
Georgi GerganovandGitHub d7b800b8bc llama : pad KV cache size (#4280) 
* llama : pad KV cache size to 32

* metal : try to improve batched decoding
 2023-12-03 10:58:16 +02:00
Georgi Gerganov 5a7d3125e7 llama : avoid using "optional" keyword (#4283) 2023-12-01 20:39:12 +02:00
Georgi Gerganov d5a1cbde60 llama : support optional tensors (#4283) 2023-12-01 20:35:47 +02:00
Georgi GerganovandGitHub 880f57973b llama : fix integer overflow during quantization (#4284) 
happens with multi-threaded quantization of Qwen-72B

ggml-ci
 2023-12-01 18:42:11 +02:00
Georgi GerganovandGitHub ef47ec18da ggml : add ggml_soft_max_ext (#4256) 
* metal : implement soft_max_ext

* cuda : implement soft_max_ext

* ggml : implement soft_max_ext (CPU)

* batched-bench : print threads

ggml-ci

* metal : simplify soft_max encoding

ggml-ci

* cuda : use 512 threads for soft_max instead of 32

* ggml : update soft max cpu

* cuda : do warp-based block reduce

* cuda : increase max block size to 1024

* cuda : fix warp reduction initialization of shared mem

* metal : warp-based reduction for soft max kernel

* metal : warp-based reduce for rms_norm

* metal : simplify soft max kernel

ggml-ci

* alloc : fix build with debug
 2023-12-01 10:51:24 +02:00
Georgi Gerganov 1f5cd83275 examples : add readme files 2023-11-29 11:00:17 +02:00
Georgi GerganovandGitHub 8406b0924b ggml : re-enable BLAS for CPU when src0 != F32 + remove redundant full offload checks in llama.cpp (#4240) 
* ggml : use blas even if src0 is not F32

* llama : use n_threads_batch only when n_tokens >= 32

ggml-ci

* llama : revert n_threads_batch logic

ggml-ci
 2023-11-28 10:32:03 +02:00
Georgi Gerganov 3e73d31d9c lookahead : support -n -1 infinite generation 2023-11-26 21:52:23 +02:00
Georgi GerganovandGitHub 9656026b53 readme : update hot topics 2023-11-26 20:42:51 +02:00
Georgi GerganovGitHubslaren
 		922754a8d6 lookahead : add example for lookahead decoding (#4207) 
* lookahead : init

* lookahead : generate and store n-grams

* lookahead : use loop instead recursion to generate n-grams

* lookahead : initial working implementation

* lookahead : filter repeating n-grams

* lookahead : use deterministic init

* lookahead : add to Makefile

* lookahead : fix a bug in the seq_id of the lookahead tokens

* lookahead : add comments

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-11-26 20:33:07 +02:00
Georgi GerganovandGitHub 04814e718e readme : update hot topics 2023-11-25 12:02:13 +02:00
Georgi GerganovGitHubkir-gadjelloTobi Lütke
 		af19d35734 server : OAI API compatibility (#4198) 
* Add openai-compatible POST /v1/chat/completions API endpoint to server example

* fix code style

* Update server README.md

* Improve server README.md

* Fix server.cpp code style according to review

* server : some style changes

* server : indentation

* server : enable special tokens during tokenization by default

* server : minor code style

* server : change random string generator

* straightforward /v1/models endpoint

---------

Co-authored-by: kir-gadjello <111190790+kir-gadjello@users.noreply.github.com>
Co-authored-by: Tobi Lütke <tobi@Tobis-MacBook-Pro.local>
 2023-11-25 11:29:06 +02:00
Georgi GerganovGitHubKerfuffle
 		6b0a7420d0 llama : KV cache view API + better KV cache management (#4170) 
* llama : keep track of used KV cells + better KV cache management

* llama : zero KV cache used upon clear

ggml-ci

* llama : allow exporting a view of the KV cache (#4180)

* Allow exporting a view of the KV cache

* Allow dumping the sequences per cell in common

* Track max contiguous cells value and position as well

* Fix max contiguous empty cells index calculation

Make dump functions deal with lengths or sequences counts > 10 better

* Fix off by one error in dump_kv_cache_view

* Add doc comments for KV cache view functions

Eliminate cell sequence struct; use llama_seq_id directly

Minor cleanups

* common : add -dkvc arg for enabling kv cache dumps

---------

Co-authored-by: Kerfuffle <44031344+KerfuffleV2@users.noreply.github.com>
 2023-11-23 19:07:56 +02:00
Georgi GerganovandGitHub d103d935c0 readme : update hot topics 2023-11-23 13:51:22 +02:00
Georgi Gerganov ff8238f71d docs : add llama-star arch idea 2023-11-23 11:35:04 +02:00
Georgi Gerganov dae06c06e5 Revert "finetune : add --n-gpu-layers flag info to --help (#4128)" 
This reverts commit 05e8301e45.
 2023-11-19 19:16:07 +02:00
Georgi Gerganov 35985acffa gitignore : tokenize 2023-11-19 18:50:49 +02:00
Georgi Gerganov 5ad387e994 tokenize : fix trailing whitespace 2023-11-17 18:01:38 +02:00
Georgi GerganovandGitHub 4f447a4833 llama : fix data units (#4101) 
* llama : fix data units

ggml-ci

* Revert "llama : fix data units"

This reverts commit f5feac831fe225ed7f3db938d115732a49dccfc4.

* llama : disambiguate data units

ggml-ci
 2023-11-17 10:00:15 +02:00
Georgi GerganovandGitHub 3d68f364f1 ggml : sync (im2col, GPU conv, 32-bit arm compat) (#4060) 
ggml-ci
 2023-11-13 16:55:52 +02:00
Georgi GerganovandGitHub c049b37d7b readme : update hot topics 2023-11-13 14:18:08 +02:00
Georgi GerganovandGitHub 4760e7cc0b sync : ggml (backend v2) (#3912) 
* sync : ggml (backend v2) (wip)

* sync : migrate examples and llama.cpp to dynamic graphs (wip)

* sync : update tests + fix max op params to 64

ggml-ci

* sync : ggml-cuda

ggml-ci

* llama : fix save/load state context size

ggml-ci

* sync : try to fix build on tvOS

* sync : pass custom graph sizes in training examples

* sync : update graph copies to new ggml API

* sync : update sync-ggml.sh with new files

* scripts : fix header in sync script

* train : fix context size calculations

* llama : increase inference graph size up to 4096 nodes

* train : allocate grads for backward graphs

* train : allocate grads for gb_tmp
 2023-11-13 14:16:23 +02:00
Georgi GerganovandGitHub 413503d4b9 make : do not add linker flags when compiling static llava lib (#3977) 2023-11-07 20:25:32 +03:00
Georgi Gerganov 8f961abdc4 speculative : change default p_accept to 0.5 + CLI args (#3919) 
ggml-ci
 2023-11-03 09:41:56 +02:00
Georgi GerganovandGitHub 05816027d6 common : YAYF (yet another YARN fix) (#3925) 
ggml-ci
 2023-11-03 09:24:00 +02:00
Georgi GerganovandGitHub 224e7d5b14 readme : add notice about #3912 2023-11-02 20:44:12 +02:00
Georgi GerganovandGitHub c7743fe1c1 cuda : fix const ptrs warning causing ROCm build issues (#3913) 2023-11-02 20:32:11 +02:00
Georgi GerganovandGitHub 4ff1046d75 gguf : print error for GGUFv1 files (#3908) 2023-11-02 16:22:30 +02:00
Georgi GerganovandGitHub 2756c4fbff gguf : remove special-case code for GGUFv1 (#3901) 
ggml-ci
 2023-11-02 11:20:21 +02:00
Georgi Gerganov 1efae9b7dc llm : prevent from 1-D tensors being GPU split (#3697) 2023-11-02 09:54:44 +02:00
Georgi GerganovandGitHub 4d719a6d4e cuda : check if this fixes Pascal card regression (#3882) 2023-11-02 08:35:10 +02:00