nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 29 Packages Projects Releases Wiki Activity
1,930 Commits 1 Branch 0 Tags
97c1549808d2742d37584a3c9df28154bdf34417
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Georgi GerganovandGitHub a5cacb22b2 imatrix : add README.md 2024-01-19 15:24:47 +02:00
Georgi Gerganov de9a147df1 py : fix flake8 lint 2024-01-19 13:52:22 +02:00
Georgi GerganovandGitHub 8b20858e5e perplexity : faster Winogrande via batching (#5024) 
* perplexity : faster Winogrande via batching

ggml-ci

* perplexity : remove unused function

* perplexity : only tokenize selected tasks for Winogrande
 2024-01-19 10:45:06 +02:00
Georgi Gerganov 9b6ea4263a cmake : add ggml public headers (#5011) 2024-01-18 23:36:07 +02:00
Georgi Gerganov 2d5419d08a imatrix : fix assert for src0 non-cont check 2024-01-18 21:45:51 +02:00
Georgi Gerganov d391ae9b49 perplexity : fix winogrande N tasks option 2024-01-18 20:49:00 +02:00
Georgi Gerganov e9240cdfa0 scripts : add get-winogrande.sh 2024-01-18 20:45:39 +02:00
Georgi GerganovandGitHub ad19812cda perplexity : faster HellaSwag via batching (#5017) 
* perplexity : faster HellaSwag

ggml-ci

* perplexity : clean-up

ggml-ci

* perplexity : no need for decode_helper

ggml-ci

* perplexity : add comments

* perplexity : option to specify max batched tasks via `n_parallel`

* perplexity : remove HellaSwag restruction for n_batch
 2024-01-18 15:33:01 +02:00
Georgi Gerganov dcad445d0c scritps : add helper script to get hellaswag data in txt format 2024-01-18 11:44:49 +02:00
Georgi Gerganov 6b6916b215 sync : ggml 2024-01-17 20:54:50 +02:00
Georgi GerganovandGitHub 38566680cd ggml : add IQ2 to test-backend-ops + refactoring (#4990) 
* ggml : add IQ2 to test-backend-ops + refactoring

ggml-ci

* cuda : update supports_op for IQ2

ggml-ci

* ci : enable LLAMA_CUBLAS=1 for CUDA nodes

ggml-ci

* cuda : fix out-of-bounds-access in `mul_mat_vec_q`

ggml-ci

* tests : avoid creating RNGs for each Q tensor

ggml-ci

* tests : avoid creating RNGs for each tensor

ggml-ci
 2024-01-17 18:54:56 +02:00
Georgi GerganovandGitHub ba69bbc84c imatrix : offload to GPU support (#4957) 
* backend : add eval callback

ggml-ci

* backend : group nodes in a single compute when user don't need them

* backend : clean-up the implementation

ggml-ci

* simple : do not perform tensor data copy if not needed

* simple : fix

* imatrix : offload to GPU support

* imatrix : fix ggml_mul_mat_id hanlding

ggml-ci

* ci : add imatrix test

ggml-ci

* ci : rearrange output

ggml-ci
 2024-01-17 18:46:30 +02:00
Georgi GerganovandGitHub 44a1a4a41a backend : add eval callback (#4935) 
* backend : add eval callback

ggml-ci

* backend : group nodes in a single compute when user don't need them

* backend : clean-up the implementation

ggml-ci

* simple : do not perform tensor data copy if not needed

* simple : fix

* simple : no need for ggml_is_contiguous + fix bool parse

* llama : fix callback placement in llama_context_params

* backend : avoid double-ask callback calls

* simple : restore examples, imatrix will serve as a demo
 2024-01-17 18:39:41 +02:00
Georgi GerganovandGitHub c918fe8dca metal : create autorelease pool during library build (#4970) 
* metal : create autorelease pool during library build

ggml-ci

* test : simplify

ggml-ci
 2024-01-17 18:38:39 +02:00
Georgi Gerganov 0f83e727af py : fix whitespace 2024-01-17 18:37:36 +02:00
Georgi GerganovandGitHub 4f4bf35f46 py : fix missing added_tokens_dict for SPM and BPE vocabs (#4971) 
* py : fix missing added_tokens_dict for SPM vocab

* py : pad with unknown tokens when data is missing

ggml-ci

* py : fix BPE vocab conversion

ggml-ci

* py : fix padded dummy tokens (I hope)
 2024-01-17 15:45:03 +02:00
Georgi Gerganov 5c99960901 py : remove unnecessary hasattr (#4903) 2024-01-16 20:59:31 +02:00
Georgi GerganovandGitHub 959ef0c0df perplexity : fix kv cache handling for hellaswag (#4981) 
ggml-ci
 2024-01-16 19:34:54 +02:00
Georgi GerganovGitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
 		c37b3474e6 flake.lock: update flake-parts, flake-parts/nixpkgs-lib, and nixpkgs (#4920) 
Flake lock file updates:

• Updated input 'flake-parts':
    'github:hercules-ci/flake-parts/34fed993f1674c8d06d58b37ce1e0fe5eebcb9f5' (2023-12-01)
  → 'github:hercules-ci/flake-parts/07f6395285469419cf9d078f59b5b49993198c00' (2024-01-11)
• Updated input 'flake-parts/nixpkgs-lib':
    'github:NixOS/nixpkgs/e92039b55bcd58469325ded85d4f58dd5a4eaf58?dir=lib' (2023-11-29)
  → 'github:NixOS/nixpkgs/b0d36bd0a420ecee3bc916c91886caca87c894e9?dir=lib' (2023-12-30)
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/cfc3698c31b1fb9cdcf10f36c9643460264d0ca8' (2023-12-27)
  → 'github:NixOS/nixpkgs/317484b1ead87b9c1b8ac5261a8d2dd748a0492d' (2024-01-08)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-01-16 09:13:54 -08:00
Georgi Gerganov ddb008d845 cuda : fix dequantize kernel names (#4938) 2024-01-15 13:27:00 +02:00
Georgi GerganovandGitHub bb0c139247 llama : check LLAMA_TRACE env for extra logging (#4929) 
* llama : minor fix indent

* llama : check LLAMA_TRACE env for extra logging

ggml-ci
 2024-01-14 13:26:53 +02:00
Georgi Gerganov 9408cfdad6 scripts : sync-ggml-am.sh option to skip commits 2024-01-14 11:08:41 +02:00
Georgi Gerganov 03c5267490 llama : use LLAMA_LOG_ macros for logging 2024-01-14 11:03:19 +02:00
Georgi Gerganov 76484fbfd3 sync : ggml 2024-01-14 00:14:46 +02:00
Georgi GerganovandGitHub 4be5ef556d metal : remove old API (#4919) 
ggml-ci
 2024-01-13 20:45:45 +02:00
Georgi GerganovandGitHub 0ea069b87b server : fix prompt caching with system prompt (#4914) 2024-01-13 19:31:26 +02:00
Georgi GerganovGitHubgoerch
 		f172de03f1 llama : fix detokenization of non-special added-tokens (#4916) 
Co-authored-by: goerch <jhr.walter@t-online.de>
 2024-01-13 18:47:38 +02:00
Georgi Gerganov 2d57de5255 metal : disable log for loaded kernels (#4794) 2024-01-13 18:46:37 +02:00
Georgi GerganovandGitHub b38b5e93ae metal : refactor kernel loading code (#4794) 
* metal : detect more GPU families

* metal : refactor kernel loading

* metal : set kernel family requirements

* metal : fix kernel init + fix compile options

* metal : take into account simdgroup reduction support

* metal : print only skipped kernels

* metal : fix check for simdgroup reduction support

* metal : check for Metal 3

* metal : free allocations

* metal : normalize encoder:setComputePipelineStatus calls

ggml-ci

* metal : fix Metal3 family check

ggml-ci

* metal : check for simdgroup matrix mul. feature

ggml-ci
 2024-01-13 18:03:45 +02:00
Georgi GerganovandGitHub 15ebe59210 convert : update phi-2 to latest HF repo (#4903) 
* convert : update phi-2 to latest HF repo

ggml-ci

* py : try to fix flake stuff
 2024-01-13 13:44:37 +02:00
Georgi Gerganov de473f5f8e sync : ggml 2024-01-12 22:02:43 +02:00
Georgi Gerganov f238461236 ggml : fix 32-bit ARM compat for IQ2_XS (whisper/1758) 
* ggml : fix 32-bit ARM compat

* ggml : fix fix

* ggml : fix fix fix
 2024-01-12 22:02:11 +02:00
Georgi Gerganov 584d674be6 llama : remove redundant assert for StableLM (#4901) 2024-01-12 20:54:12 +02:00
Georgi Gerganov 5537d9d36b gitignore : imatrix 2024-01-12 14:33:21 +02:00
Georgi Gerganov 3cabe80630 llama : fix typo "imp_embd" -> "inp_embd" 2024-01-12 13:11:15 +02:00
Georgi Gerganov 2d00741e12 py : fix lint (#4889) 2024-01-12 13:03:38 +02:00
Georgi GerganovandGitHub f445c0e68c llama : fix llm_build_k_shift to use correct n_rot (#4889) 
* llama : fix llm_build_k_shift to use correct n_rot

ggml-ci

* llama : always use hparams.n_rot for ggml_rope_custom

ggml-ci

* convert : fix persimmon conversion to write correct n_rot
 2024-01-12 13:01:56 +02:00
Georgi GerganovandGitHub 1d118386fe server : fix infill when prompt is empty (#4833) 2024-01-11 23:23:49 +02:00
Georgi Gerganov 7edefbd79c main : better name for variable n_print (#4874) 2024-01-11 22:46:26 +02:00
Georgi Gerganov 3ca63b4538 main : disable token count by default (#4874) 2024-01-11 22:43:05 +02:00
Georgi Gerganov b037787548 swift : track ggml release branch (#4867) 2024-01-11 21:58:28 +02:00
Georgi GerganovandGitHub 3ba5b8ca8e swift : pin ggml commit + remove ggml.h from spm-headers (#4878) 
ggml-ci
 2024-01-11 21:31:31 +02:00
Georgi Gerganov 64802ec00d sync : ggml 2024-01-11 09:39:08 +02:00
Georgi Gerganov 3267c2abc7 metal : fix deprecation warning (ggml/690) 2024-01-11 09:39:05 +02:00
Georgi GerganovandGitHub 5c1980d8d4 server : fix build + rename enums (#4870) 2024-01-11 09:10:34 +02:00
Georgi GerganovandGitHub 4dccb38d9a metal : improve dequantize precision to match CPU (#4836) 
ggml-ci
 2024-01-09 19:37:08 +02:00
Georgi Gerganov 9a818f7c42 scripts : improve get-pg.sh (#4838) 2024-01-09 19:21:13 +02:00
Georgi GerganovandGitHub d9653894df scripts : script to get Paul Graham essays in txt format (#4838) 2024-01-09 16:23:05 +02:00
Georgi GerganovandGitHub 18c2e1752c ggml : fix vld1q_s8_x4 32-bit compat (#4828) 
* ggml : fix vld1q_s8_x4 32-bit compat

ggml-ci

* ggml : fix 32-bit ARM compat (cont)

ggml-ci
 2024-01-09 10:42:06 +02:00
Georgi GerganovandGitHub a9a8c5de3d readme : add link to SOTA models 2024-01-08 20:25:17 +02:00
Georgi GerganovandGitHub 668b31fc7d swift : exclude ggml-metal.metal from the package (#4822) 2024-01-08 16:40:51 +02:00
Georgi GerganovandGitHub 42ea63c5a3 llama.swiftui : update readme 2024-01-08 15:57:36 +02:00
Georgi GerganovandGitHub 52531fdff8 main : add self-extend support (#4815) 
* examples : add passkey test

* passkey : better prints

* passkey : select pass key pos from CLI

* passkey : simplify n_past logic

* llama : "self-extend"-like context extension

* passkey : add comment

* main : add Self-Extend support

* llama : add comment about llama_kv_cache_seq_div
 2024-01-08 11:18:32 +02:00
Georgi GerganovandGitHub b0034d93ce examples : add passkey test (#3856) 
* examples : add passkey test

* passkey : better prints

* passkey : select pass key pos from CLI

* passkey : simplify n_past logic

* make : add passkey target

* passkey : add "self-extend"-like context extension (#4810)

* llama : "self-extend"-like context extension

* passkey : add comment

* passkey : add readme
 2024-01-08 11:14:04 +02:00
Georgi Gerganov 9dede37d81 llama : remove unused vars (#4796) 2024-01-07 14:29:36 +02:00
Georgi GerganovandGitHub 3c36213df8 llama : remove redundant GQA check (#4796) 2024-01-07 11:21:53 +02:00
Georgi Gerganov d117d4dc5d llama : print tensor meta for debugging 2024-01-07 09:51:12 +02:00
Georgi GerganovandGitHub 67984921a7 server : fix n_predict check (#4798) 2024-01-07 08:45:26 +02:00
Georgi Gerganov 96e80dabc6 examples : improve base-translate.sh script (#4783) 2024-01-06 11:40:24 +02:00
Georgi Gerganov 91d38876df metal : switch back to default.metallib (ggml/681) 
ggml-ci
 2024-01-05 18:02:06 +02:00
Georgi Gerganov d061bf9405 ggml : fix q2_k bpw in comments (ggml/680) 2024-01-05 18:02:06 +02:00
Georgi GerganovandGitHub c1d7cb28d3 ggml : do not sched_yield when calling BLAS (#4761) 
* ggml : do not sched_yield when calling BLAS

ggml-ci

* ggml : fix do_yield logic

ggml-ci

* ggml : simplify do_yield logic

ggml-ci
 2024-01-05 15:18:21 +02:00
Georgi GerganovandGitHub 3681f22443 examples : add few-shot translation example (#4783) 2024-01-05 15:11:10 +02:00
Georgi GerganovandGitHub 012cf349ae server : send token probs for "stream == false" (#4714) 2024-01-04 19:56:33 +02:00
Georgi GerganovandGitHub dc891b7f7a ggml : include stdlib.h before intrin.h (#4736) 2024-01-04 10:12:26 +02:00
Georgi Gerganovandslaren 7bed7eba35 cuda : simplify expression 
Co-authored-by: slaren <slarengh@gmail.com>
 2024-01-03 14:38:38 +02:00
Georgi Gerganov d55356d3ba cuda : mark I16 and I32 ops as unsupported 
ggml-ci
 2024-01-03 14:38:38 +02:00
Georgi Gerganov 75e3fd8581 sync : ggml 
ggml-ci
 2024-01-03 14:38:38 +02:00
Georgi Gerganov 289313716f metal : add kernel_get_rows_i32 
ggml-ci
 2024-01-03 14:38:38 +02:00
Georgi Gerganov ab62fc3e55 scripts : fix sync order + metal sed 2024-01-03 14:38:38 +02:00
Georgi GerganovandGitHub f3f62f0d83 metal : optimize ggml_mul_mat_id (faster Mixtral PP) (#4725) 
* ggml : disable fast-math for Metal (cmake build only)

ggml-ci

* metal : fix Metal API debug warnings

* cmake : add -fno-inline for Metal build (#4545)

* metal : fix API debug warnings

* metal : fix compile warnings

* metal : use uint64_t for strides

* cmake : rename option to LLAMA_METAL_SHADER_DEBUG

* metal : fix mat-vec Q8_0 kernel for BS > 1

* metal : normalize mat-vec kernel signatures

* cmake : respect LLAMA_QKK_64 option

* metal : fix mat-vec Q4_K kernel for QK_K == 64

* metal : optimizing ggml_mul_mat_id (wip)

* metal : minor fix

* metal : opt mul_mm_id
 2024-01-02 21:07:47 +02:00
Georgi Gerganov 540938f890 llama : llama_model_desc print number of experts 2024-01-02 16:26:45 +02:00
Georgi Gerganov 32866c5edd editorconfig : fix whitespace and indentation #4710 2024-01-02 13:28:15 +02:00
Georgi GerganovandGitHub 58ba655af0 metal : enable shader debugging (cmake option) (#4705) 
* ggml : disable fast-math for Metal (cmake build only)

ggml-ci

* metal : fix Metal API debug warnings

* cmake : add -fno-inline for Metal build (#4545)

* metal : fix API debug warnings

* metal : fix compile warnings

* metal : use uint64_t for strides

* cmake : rename option to LLAMA_METAL_SHADER_DEBUG

* metal : fix mat-vec Q8_0 kernel for BS > 1

* metal : normalize mat-vec kernel signatures

* cmake : respect LLAMA_QKK_64 option

* metal : fix mat-vec Q4_K kernel for QK_K == 64

ggml-ci
 2024-01-02 10:57:44 +02:00
Georgi GerganovandGitHub e39106c055 ggml : add ggml_vdotq_s32 alias (#4715) 
ggml-ci
 2023-12-31 11:43:31 +02:00
Georgi GerganovandGitHub 9fbda719de clip : refactor + bug fixes (#4696) 
* clip : refactor + bug fixes

ggml-ci

* server : add log message
 2023-12-30 23:24:42 +02:00
Georgi Gerganov 0235b9b571 clip : use ggml_backend_buffer_is_host (#4205) 2023-12-29 18:53:34 +02:00
Georgi Gerganov c8255f8a6b scripts : print list of sync commits 2023-12-29 15:12:35 +02:00
Georgi Gerganov 38b3de4658 sync : ggml 2023-12-29 14:56:41 +02:00
Georgi Gerganov ca38b8d334 scripts : do not sync commits from this repo 2023-12-29 14:54:05 +02:00
Georgi Gerganov b47879b0dd scripts : add sync-ggml-am.sh 2023-12-27 11:44:22 +02:00
Georgi GerganovandGitHub 951010fa53 ggml : fix dot product for ARM (#4630) 
ggml-ci
 2023-12-27 11:02:13 +02:00
Georgi GerganovGitHubleejet
 		ba66175132 sync : ggml (fix im2col) (#4591) 
* cuda : fix im2col_f32_f16 (ggml/#658)

ggml-ci

* ggml-alloc : fix ggml_tallocr_is_own

---------

Co-authored-by: leejet <leejet714@gmail.com>
 2023-12-22 17:53:43 +02:00
Georgi GerganovandGitHub afefa319f1 ggml : change ggml_scale to take a float instead of tensor (#4573) 
* ggml : change ggml_scale to take a float instead of tensor

* ggml : fix CPU implementation

* tests : fix test-grad0

ggml-ci
 2023-12-21 23:20:49 +02:00
Georgi GerganovandGitHub 769a7bc85e gguf-py : fix broken link 2023-12-21 23:20:36 +02:00
Georgi Gerganov 32259b2dad gguf : simplify example dependencies 2023-12-21 23:08:14 +02:00
Georgi GerganovandGitHub c083718c89 readme : update coding guidelines 2023-12-21 19:27:14 +02:00
Georgi Gerganov 0e18b2e7d0 llama.swiftui : add tinyllama 1.1B F16 2023-12-18 20:17:43 +02:00
Georgi Gerganov 6ff39b129d llama.swiftui : add more models 2023-12-18 20:05:12 +02:00
Georgi GerganovandGitHub b1306c4394 readme : update hot topics 2023-12-17 20:16:23 +02:00
Georgi GerganovGitHubjhen
 		800a489e4a llama.swiftui : add bench functionality (#4483) 
* llama.swiftui : add bench button

* llama.swiftui : initial bench functionality

* force to use n_gpu_layers on simulator

* add download buttons & expose llamaState.loadModel

* update project.pbxproj

* comment #Preview & fix editorconfig check

* gitignore : xcode stuff

* llama.swiftui : UX improvements

* llama.swiftui : avoid data copy via "downloadTask"

* llama.swiftui : remove model from project

* llama : remove "mostly" from model infos

* llama.swiftui : improve bench

---------

Co-authored-by: jhen <developer@jhen.me>
 2023-12-17 19:38:41 +02:00
Georgi Gerganov 55e87c3749 ggml : fix OpenCL broadcast requirement for ggml_mul (close #4453) 2023-12-14 10:35:29 +02:00
Georgi GerganovGitHubslaren
 		4d98d9a656 sync : ggml (SD ops, tests, kernels) (#4444) 
* sync : ggml (SD ops, tests, kernels)

ggml-ci

* cuda : restore im2col

ggml-ci

* metal : fix accuracy of dequantization kernels

ggml-ci

* cuda : restore correct im2col

ggml-ci

* metal : try to fix moe test by reducing expert size

ggml-ci

* cuda : fix bin bcast when src1 and dst have different types

ggml-ci

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-12-13 21:54:54 +02:00
Georgi GerganovandGitHub 113f9942fc readme : update hot topics 2023-12-13 14:05:38 +02:00
Georgi GerganovGitHubslaren
 		fe680e3d10 sync : ggml (new ops, tests, backend, etc.) (#4359) 
* sync : ggml (part 1)

* sync : ggml (part 2, CUDA)

* sync : ggml (part 3, Metal)

* ggml : build fixes

ggml-ci

* cuda : restore lost changes

* cuda : restore lost changes (StableLM rope)

* cmake : enable separable compilation for CUDA

ggml-ci

* ggml-cuda : remove device side dequantize

* Revert "cmake : enable separable compilation for CUDA"

This reverts commit 09e35d04b1c4ca67f9685690160b35bc885a89ac.

* cuda : remove assert for rope

* tests : add test-backend-ops

* ggml : fix bug in ggml_concat

* ggml : restore `ggml_get_n_tasks()` logic in `ggml_graph_plan()`

* ci : try to fix macOS

* ggml-backend : remove backend self-registration

* ci : disable Metal for macOS cmake build

ggml-ci

* metal : fix "supports family" call

* metal : fix assert

* metal : print resource path

ggml-ci

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-12-07 22:26:54 +02:00
Georgi GerganovGitHubslaren
 		bcc0eb4591 llama : per-layer KV cache + quantum K cache (#4309) 
* per-layer KV

* remove unnecessary copies

* less code duplication, offload k and v separately

* llama : offload KV cache per-layer

* llama : offload K shift tensors

* llama : offload for rest of the model arches

* llama : enable offload debug temporarily

* llama : keep the KV related layers on the device

* llama : remove mirrors, perform Device -> Host when partial offload

* common : add command-line arg to disable KV cache offloading

* llama : update session save/load

* llama : support quantum K cache (#4312)

* llama : support quantum K cache (wip)

* metal : add F32 -> Q8_0 copy kernel

* cuda : add F32 -> Q8_0 copy kernel

ggml-ci

* cuda : use mmv kernel for quantum cache ops

* llama : pass KV cache type through API

* llama : fix build

ggml-ci

* metal : add F32 -> Q4_0 copy kernel

* metal : add F32 -> Q4_1 copy kernel

* cuda : wip

* cuda : add F32 -> Q4_0 and F32 -> Q4_1 copy kernels

* llama-bench : support type_k/type_v

* metal : use mm kernel only for quantum KV cache

* cuda : add comment

* llama : remove memory_f16 and kv_f16 flags

---------

Co-authored-by: slaren <slarengh@gmail.com>

* readme : add API change notice

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-12-07 13:03:17 +02:00
Georgi GerganovandGitHub 05cd6e5036 server : recognize cache_prompt parameter in OAI API (#4347) 2023-12-06 20:21:59 +02:00
Georgi Gerganov caa9249217 common : fix compile warning 2023-12-06 10:41:03 +02:00
Georgi GerganovandGitHub fbbc42827b ggml : reuse ggml_get_n_tasks() in ggml_graph_plan() (#4308) 
* ggml : fix soft max out-of-bounds access

ggml-ci

* ggml : reuse ggml_get_n_tasks() in ggml_graph_plan()

ggml-ci
 2023-12-03 15:56:35 +02:00
Georgi GerganovandGitHub adf3de4f69 ggml : fix soft max out-of-bounds access (#4307) 
ggml-ci
 2023-12-03 15:56:22 +02:00