nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity
734 Commits 1 Branch 0 Tags
65bdd52a867539691007f85c5508146d507f72c1
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Georgi Gerganov 65bdd52a86 tests : sync test-grad0 from ggml 2023-06-24 19:40:18 +03:00
Georgi Gerganov 11da1a85cd readme : fix whitespaces 2023-06-24 13:38:18 +03:00
Georgi GerganovandGitHub 049aa16b8c readme : add link to p1 2023-06-20 19:05:54 +03:00
Georgi Gerganov 18b35625c3 ggml : fix bug in LBFGS optimizer (found by ggml tests) 2023-06-19 20:43:30 +03:00
Georgi Gerganov 23fc5c219a cmake : fix trailing whitespaces 2023-06-19 18:18:34 +03:00
Georgi GerganovandGitHub b97ca431db ggml : sync latest ggml repo (#1924) 
* ggml : sync latest ggml repo

* ggml : remove unused comments

* ggml : asserts
 2023-06-19 18:12:33 +03:00
Georgi GerganovandGitHub ce2c7d72e2 metal : handle buffers larger than device's maxBufferLength (#1826) 
* metal : handle buffers larger than device's maxBufferLength

* metal : print more verbose device info + handle errors

* metal : fix prints for overlapping views

* metal : minimize view overlap to try to utilize device memory better
 2023-06-18 09:09:47 +03:00
Georgi Gerganov b2416493ab make : do not print help for simple example 2023-06-17 20:55:03 +03:00
Georgi Gerganov 4f9c43e3bd minor : warning fixes 2023-06-17 20:24:11 +03:00
Georgi Gerganov 051e1b0e6a llama : fix kv_cache n init (close #1903) 2023-06-17 19:31:20 +03:00
Georgi Gerganov bed9275617 cmake : remove whitespaces 2023-06-15 21:56:50 +03:00
Georgi GerganovandGitHub 4bfcc855ab metal : parallel command buffer encoding (#1860) 
* metal : parallel command buffer encoding

* metal : determine number of command buffers based on gf->n_threads
 2023-06-15 20:29:48 +03:00
Georgi GerganovandGitHub 2347e45e7b llama : do a warm-up eval at start for better timings (#1824) 2023-06-13 20:20:07 +03:00
Georgi Gerganov 4de0334f5c cmake : fix Metal build (close #1791) 2023-06-10 22:56:53 +03:00
Georgi Gerganov 17c10acfb4 ggml : force no_alloc == false when creating opt tensors (close #1699) 
This is needed to make operators like ggml_view() be able to store their
parameters in the ggml context's memory and not get discarded when
no_alloc is true
 2023-06-10 12:08:15 +03:00
Georgi Gerganov b33dee282f metal : fix build "tanhf" -> "tanh" 2023-06-09 11:11:04 +03:00
Georgi Gerganov 0bf7cf1b29 Revert "ggml : load data into int8x16x4_t using vld4q_s8 on arm64 (#1738)" 
This reverts commit 8432d4d9f7.
 2023-06-08 20:48:14 +03:00
Georgi Gerganov 53aba3f393 clang-tidy : restore dot file from accidental deletion 2023-06-08 10:09:08 +03:00
Georgi GerganovandGitHub 5c64a0952e k-quants : allow to optionally disable at compile time (#1734) 
* k-quants : put behind optional compile flag LLAMA_K_QUANTS

* build : enable k-quants by default
 2023-06-07 10:59:52 +03:00
Georgi GerganovandGitHub 4dc62c545d readme : add June roadmap 2023-06-07 07:15:08 +03:00
Georgi Gerganov 2d7bf110ed llama : fix vram_scratch var 2023-06-06 22:54:39 +03:00
Georgi Gerganov 2a4e41a086 llama : fix compile warnings 2023-06-06 22:41:53 +03:00
Georgi Gerganov 44f906e853 metal : add f16 support 2023-06-06 20:21:56 +03:00
Georgi Gerganov 2d43387daf ggml : fix builds, add ggml-quants-k.o (close #1712, close #1710) 2023-06-06 10:18:03 +03:00
Georgi Gerganov 7ad7750c5c gitignore : add .clang-tidy 2023-06-06 09:55:25 +03:00
Georgi Gerganov 7a74dee6b4 llama : temporary disable Q6_K output quantization (#1711) 2023-06-06 09:39:38 +03:00
Georgi GerganovandGitHub e7fe66e670 ci : disable auto tidy (#1705) 2023-06-05 23:05:05 +03:00
Georgi Gerganov d1f563a743 llama : fix Metal KV cache sync (close #1695) 2023-06-05 10:19:03 +03:00
Georgi GerganovandGitHub 827f5eda91 readme : update hot topics 2023-06-04 23:38:19 +03:00
Georgi GerganovandGitHub ecb217db4f llama : Metal inference (#1642) 
* mtl : export the LLaMA computation graph

* ci : disable temporary

* mtl : adapt the MNIST example as starter

* mtl : no need for mtl-export tool, add cli arg for main instead

* mtl : export just a small part of the graph for now to make it easier

* mtl : move MSL code into separate file for easy editing

* mtl : initial get_rows_q4_0 kernel

* mtl : confirmed get_rows_q4_0 is working correctly

* mtl : add rms_norm kernel + confirm working

* mtl : add mul kernel + confirm working

* mtl : initial mul_mat Q4 kernel (wrong results)

* mtl : mul_mat fixes (still wrong)

* mtl : another mul_mat Q4 (still does not work)

* mtl : working mul_mat q4

* ggml : fix handling of "view" ops in ggml_graph_import()

* mtl : add rope kernel

* mtl : add reshape and transpose handling

* ggml : store offset as opt arg for ggml_view_xd() operators

* mtl : add cpy kernel + handle view ops

* mtl : confirm f16 x f32 attention mul mat

* mtl : add scale kernel

* mtl : add diag_mask_inf kernel

* mtl : fix soft_max kernel

* ggml : update ggml_nbytes() to handle non-contiguous tensors

* mtl : verify V tensor contents

* mtl : add f32 -> f32 cpy kernel

* mtl : add silu kernel

* mtl : add non-broadcast mul kernel

* mtl : full GPU inference of the computation graph

* mtl : optimize rms_norm and soft_max kernels

* mtl : add f16 mat x f32 vec multiplication kernel

* mtl : fix bug in f16 x f32 mul mat + speed-up computation

* mtl : faster mul_mat_q4_0_f32 kernel

* mtl : fix kernel signature + roll inner loop

* mtl : more threads for rms_norm + better timing

* mtl : remove printfs from inner loop

* mtl : simplify implementation

* mtl : add save/load vocab to ggml file

* mtl : plug Metal inference into llama.cpp (very quick-n-dirty)

* mtl : make it work with main example

Lots of hacks but at least now it generates text

* mtl : preparing for merge

* mtl : clean-up ggml mtl interface + suport scratch / inplace

* mtl : remove temp / debug code

* metal : final refactoring and simplification

* Revert "ci : disable temporary"

This reverts commit 98c267fc77fe811082f672538fc91bcfc9072d63.

* metal : add comments

* metal : clean-up stuff, fix typos

* readme : add Metal instructions

* readme : add example for main
 2023-06-04 23:34:30 +03:00
Georgi Gerganov 7552ac5863 ggml : sync cgraph import / export API 2023-05-29 19:31:44 +03:00
Georgi Gerganov 5d1830b99d ggml : fix bug in ggml_alibi 2023-05-29 19:30:49 +03:00
Georgi Gerganov 93618031c7 ggml : add ggml_tensor_overhead() 2023-05-27 16:19:56 +03:00
Georgi Gerganov bdbda1b17a ggml : sync ggml core (minor additions, e.g. ggml_get_tensor_by_name()) 2023-05-27 12:23:16 +03:00
Georgi Gerganov 265db9834e ggml : output 3d sizes in ggml_graph_dump_dot() 2023-05-21 11:56:23 +03:00
Georgi Gerganov fab49c685e ggml : update WASM SIMD 2023-05-20 20:00:41 +03:00
Georgi GerganovandGitHub 3de84b2606 ggml : add ggml_clamp() (#1539) 
* ggml : add ggml_clamp()

* ggml : indentation
 2023-05-20 15:34:45 +03:00
Georgi Gerganov ea600071cb Revert "feature : add blis and other BLAS implementation support (#1502)" 
This reverts commit 07e9ace0f9.
 2023-05-20 12:03:48 +03:00
Georgi Gerganov ec2e10c444 llama : add llama_init_backend() API (close #1527) 2023-05-20 11:06:37 +03:00
Georgi Gerganov 8a203f9fa1 llama : fix compile warnings in llama_set_state_data() 2023-05-20 10:14:43 +03:00
Georgi Gerganov 4fd3e29297 ggml : fix scalar implementation of Q4_1 dot 2023-05-20 10:13:19 +03:00
Georgi GerganovandGitHub 2d5db48371 ggml : use F16 instead of F32 in Q4_0, Q4_1, Q8_0 (#1508) 
* ggml : use F16 instead of F32 in Q4_0, Q4_1 and Q8_0

* llama : bump LLAMA_FILE_VERSION to 3

* cuda : update Q4 and Q8 dequantize kernels

* ggml : fix AVX dot products

* readme : update performance table + hot topics
 2023-05-19 22:17:18 +03:00
Georgi Gerganov 6986c7835a tests : add missing header 2023-05-19 21:17:28 +03:00
Georgi Gerganov 4b7e245adf minor : fix compile warnings 2023-05-19 20:14:51 +03:00
Georgi GerganovandGitHub 13c351ad72 ggml : various fixes (#1450) 
- `ggml_rope()`
- `ggml_diag_mask_inf()` multi-threaded
- compatibility with scratch buffers
 2023-05-14 18:22:50 +03:00
Georgi Gerganov 601a033475 ggml : add GGML_QNT_VERSION to track quantization format changes 
https://github.com/ggerganov/ggml/issues/150#issuecomment-1546625668
 2023-05-14 10:20:19 +03:00
Georgi Gerganov 08737ef720 cuda : fix convert function (#1412) 2023-05-13 17:40:58 +03:00
Georgi Gerganov bda4d7c215 make : fix PERF build with cuBLAS 2023-05-13 17:25:09 +03:00
Georgi Gerganov 5a5aeb1e91 llama : fix unused warning 2023-05-13 16:55:14 +03:00
Georgi GerganovandGitHub 66841fdb0e ggml : multi-thread mul and diag_mask ops (#1428) 2023-05-13 16:48:03 +03:00
Georgi Gerganov f048af0230 ggml : sync alibi fix from ggml repo 2023-05-13 11:54:33 +03:00
Georgi Gerganov 0cd22e190a llama : fix various warnings 2023-05-13 11:23:15 +03:00
Georgi GerganovandGitHub cdd5350892 readme : update Q4_0 perplexities 
I think these were affected by the removal of the `round` during quantization
 2023-05-13 09:12:44 +03:00
Georgi Gerganov 738ace394a llama : free ggml context in set / copy state data (close #1425) 2023-05-13 09:08:52 +03:00
Georgi Gerganov fb62f92433 llama : fix --mtest option (close #1414) 2023-05-12 21:44:20 +03:00
Georgi GerganovGitHubStephan Walter
 		b9fd7eee57 ggml : remove bit shuffling (#1405) 
* ggml : remove Q4_0 bit shufling (ARM NEON)

* ggml : remove Q4_1 bit shuffling (ARM NEON + reference)

* ggml : nibbles_from_floats() + bytes_from_nibbles() (ARM NEON)

* ggml : remove Q4_2 bit shuffling (WIP, BROKEN)

* ggml : remove Q5_0 bit shuffling (ARM NEON)

* ggml : 2x faster scalar implementations

* ggml : remove Q5_1 bit shuffling (ARM NEON + scalar)

* ggml : simplify scalar dot

* ggml : remove WASM SIMD bit shuffling + remove vzip for ARM 32-bit

* ggml : fix Q4_1 quantization

* ggml : update cuBLAS + normalize variable names

* ggml : remove Q4_2 mode

* ggml : minor formatting

* ggml : fix Q5_0 quantization

* scripts : add script for measuring the time per token

* AVX implementations (#1370)

* ggml : uniform 5th bit extraction

* llama : produce error upon loading old model files

* llama : fix model magic/version write

* ggml : speed-up Q5_0 + Q5_1 at 4 threads

* ggml : preserve old Q4 and Q5 formats

* ggml : simplify Q8_1 - no need for low / high sums anymore

* ggml : fix Q8_0 and Q8_1 rounding

* Revert "AVX implementations (#1370)"

This reverts commit 948d124837f9d287d8490f41338e0e4cceb0814f.

* ggml : fix AVX2 implementation

* sha : update hashes for 7B and 13B

* readme : update timings + remove warning banner

* llama : update v2 PR number to 1405

* ggml : fix WASM comments

* ggml : back to original bit order

* readme : add note that Q4 and Q5 have been changed

* llama : fix return for unknown version

---------

Co-authored-by: Stephan Walter <stephan@walter.name>
 2023-05-12 00:23:08 +03:00
Georgi GerganovandGitHub 56551bc11f readme : add notice about upcoming breaking change 2023-05-08 22:52:18 +03:00
Georgi GerganovandGitHub f9a6364912 llama : require first token to be BOS (#1303) 
* llama : require first token to be BOS

* scripts : add ppl-run-all.sh

* perplexity : add BOS for each chunk

* readme : update perplexity values after BOS fix

* perplexity : add clarifying comments
 2023-05-08 17:41:54 +03:00
Georgi Gerganov 799fdc1b5d ggml : vectorize Q8_0 quantization 
https://github.com/ggerganov/ggml/pull/127#issuecomment-1533648531
 2023-05-03 23:24:20 +03:00
Georgi GerganovandGitHub bca9ad938a minor : fix whitespaces (#1302) 2023-05-03 20:09:42 +03:00
Georgi Gerganov e2a937ca6a minor : fix trailing whitespaces 2023-05-03 18:43:23 +03:00
Georgi Gerganov 0e6cbff1b7 llama : fix compile warnings 2023-05-02 23:09:08 +03:00
Georgi Gerganov 5d5817ca60 ggml : fix 32-bit ARM 2023-05-02 22:14:50 +03:00
Georgi GerganovandGitHub 70269cae37 llama : fix session load / save (#1263) 2023-05-01 14:54:59 +03:00
Georgi Gerganov 7ff0dcd320 ggml : fix UB (int << 31) 2023-04-30 22:28:51 +03:00
Georgi Gerganov 6bc4400e67 ggml : add Q5 WASM SIMD + GGML_FTYPE 2023-04-30 19:07:43 +03:00
Georgi Gerganov 3e5aa8a1c4 ggml : fix labels for GGML_OP_ALIBI 2023-04-30 10:25:46 +03:00
Georgi Gerganov c3ca7a5f05 ggml : fix 32-bit ARM NEON 2023-04-29 21:34:23 +03:00
Georgi Gerganov e8c051611a ggml : use vzip instead of vuzp for consistency 2023-04-29 21:12:56 +03:00
Georgi Gerganov 0b5a935099 ggml : fix visibility and unused warnings 2023-04-29 19:28:36 +03:00
Georgi GerganovandGitHub ec728e44d7 ggml : fix #if for f32_f32 mul_mat (CLBlast) (#1229) 2023-04-29 18:43:42 +03:00
Georgi GerganovandGitHub 214b6a3570 ggml : adjust mul_mat_f16 work memory (#1226) 
* llama : minor - remove explicity int64_t cast

* ggml : reduce memory buffer for F16 mul_mat when not using cuBLAS

* ggml : add asserts to guard for incorrect wsize
 2023-04-29 18:43:28 +03:00
Georgi Gerganov 305eb5afd5 build : fix reference to old llama_util.h 2023-04-29 13:53:12 +03:00
Georgi Gerganov 84ca9c2ecf examples : fix save-load-state + rename llama-util.h 2023-04-29 13:48:11 +03:00
Georgi GerganovandGitHub 334637e43e common : change default parameters to pre-#1126 (#1223) 2023-04-29 09:51:06 +03:00
Georgi GerganovandGitHub 7f15c5c477 readme : update hot topics 2023-04-28 21:32:52 +03:00
Georgi Gerganov 55390bcaf2 ggml : sync ggml (ggml_alibi) 2023-04-28 20:51:05 +03:00
Georgi Gerganov 11d902364b ggml : add helper debug printf in soft_max 2023-04-28 17:59:08 +03:00
Georgi GerganovandGitHub f9be42add0 readme : add quantization info 2023-04-26 23:24:42 +03:00
Georgi GerganovGitHubStephan Walter
 		574406dc7e ggml : add Q5_0 and Q5_1 quantization (#1187) 
* ggml : add Q5_0 quantization (cuBLAS only)

* ggml : fix Q5_0 qh -> uint32_t

* ggml : fix q5_0 histogram stats

* ggml : q5_0 scalar dot product

* ggml : q5_0 ARM NEON dot

* ggml : q5_0 more efficient ARM NEON using uint64_t masks

* ggml : rename Q5_0 -> Q5_1

* ggml : adding Q5_0 mode

* quantize : add Q5_0 and Q5_1 to map

* ggml : AVX2 optimizations for Q5_0, Q5_1 (#1195)

---------

Co-authored-by: Stephan Walter <stephan@walter.name>
 2023-04-26 23:14:13 +03:00
Georgi GerganovandGitHub 7a32fcb3b2 ggml : add Q8_0 quantization format (rename the old one to Q8_1) (ARM NEON) (#1179) 
* ggml : add Q8_0 quantization format (rename the old one to Q8_1)

* tests : fix test-quantize-fns

* ggml : finalize Q8_0 implementation

* ggml : use q4_0_q8_0 and q4_2_q8_0

* ggml : fix Q8_0 dot product bug (ARM)

* ggml : Q8_0 unroll x2

* ggml : fix bug - using wrong block type

* ggml : extend quantize_fns_t with "vec_dot_type"

* ggml : fix Q8_0 to use 255 values out of 256

* ggml : fix assert using wrong QK4_2 instead of QK4_3
 2023-04-25 23:40:51 +03:00
Georgi GerganovandGitHub 8a0f8673ba ggml : export symbols (#1155) 2023-04-24 22:18:25 +03:00
Georgi Gerganov 957c8ae21d llama : increase scratch buffer size for 65B (ref #1152) 
Temporary solution
 2023-04-24 18:47:30 +03:00
Georgi GerganovandGitHub c4fe84fb0d llama : refactor get / set state + remove redundant kv cache API (#1143) 2023-04-24 07:40:02 +03:00
Georgi Gerganov 284685f169 scripts : add helper scripts to synch ggml repo 2023-04-23 19:57:09 +03:00
Georgi Gerganov ec9cdb6752 ggml : do not print perf ops that have not been used at all 2023-04-23 18:32:52 +03:00
Georgi Gerganov e4422e299c ggml : better PERF prints + support "LLAMA_PERF=1 make" 2023-04-23 18:15:39 +03:00
Georgi Gerganov 0e018fe008 ggml : fix Q4_3 cuBLAS 2023-04-22 16:32:07 +03:00
Georgi Gerganov 872c365a91 ggml : fix AVX build + update to new Q8_0 format 2023-04-22 11:08:12 +03:00
Georgi GerganovandGitHub 955ef9a5d5 ggml : alternative Q4_3 implementation using modified Q8_0 (#1109) 
* ggml : prefer vzip to vuzp

This way we always use the same type of instruction across all quantizations

* ggml : alternative Q4_3 implementation using modified Q8_0

* ggml : fix Q4_3 scalar imlpementation

* ggml : slight improvement of Q4_3 - no need for loop unrolling

* ggml : fix AVX paths for Q8_0 quantization
 2023-04-22 10:55:35 +03:00
Georgi Gerganov d40fded93e llama : fix comment for "output.weight" tensor 2023-04-21 10:24:02 +03:00
Georgi Gerganov 12b5900dbc ggml : sync ggml (add GPT-NeoX RoPE implementation) 2023-04-20 23:32:59 +03:00
Georgi Gerganov 9ff334f3c9 ggml : fix bug in ggml_compute_forward_dup_f32() 2023-04-20 21:58:38 +03:00
Georgi Gerganov 8a1756abdf ggml : do not break cuBLAS build (Q4_3 is not yet implemented) 2023-04-20 21:43:50 +03:00
Georgi Gerganov 66aab46079 ggml : fix Q4_3 quantization 
Broke it during conflict resolution in last PR
 2023-04-20 20:44:05 +03:00
Georgi GerganovandGitHub e0305ead3a ggml : add Q4_3 quantization (#1082) 2023-04-20 20:35:53 +03:00
Georgi GerganovGitHubslaren
 		884e7d7a2b ggml : use 8-bit precision for Q4_1 intermediate results (#1047) 
* ggml : use 8-bit precision for Q4_1 intermediate results (ARM)

* ggml : optimize ggml_vec_dot_q4_1_q8_0() via vmalq_n_f32

56 ms/token with Q4_1 !

* ggml : AVX2 implementation of ggml_vec_dot_q4_1_q8_0 (#1051)

* gitignore : ignore ppl-*.txt files

---------

Co-authored-by: slaren <2141330+slaren@users.noreply.github.com>
 2023-04-19 20:10:08 +03:00
Georgi GerganovandGitHub 7cd5c4a3e9 readme : add warning about Q4_2 and Q4_3 2023-04-19 19:07:54 +03:00
Georgi GerganovandGitHub 77a73403ca ggml : add new Q4_2 quantization (ARM only) (#1046) 
* ggml : Q4_2 ARM

* ggml : add ggml_is_quantized()

* llama : update llama_type_name() with Q4_2 entry

* ggml : speed-up q4_2

- 4 threads: ~100ms -> ~90ms
- 8 threads:  ~55ms -> ~50ms

* ggml : optimize q4_2 using vmlaq_n_f32 + vmulq_n_f32
 2023-04-18 23:54:57 +03:00
Georgi Gerganov 50a8a2af97 ggml : scratch that - vmlaq_n_f32 is always better 
Had a background process that was messing with the timings
 2023-04-18 23:11:23 +03:00