nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 23 Packages Projects Releases Wiki Activity
372 Commits 1 Branch 0 Tags
42747220b4cac548b6e3059b66b3e960b517cfa4
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Georgi Gerganov eb17a026fd quantize-stats : fix bug in --type argument 2023-04-17 17:31:06 +03:00
Georgi Gerganov 69b740289f ggml : avoid using ggml_fp16_to_fp32() and ggml_fp32_to_fp16() in ggml.c 2023-04-17 16:16:23 +03:00
Georgi Gerganov 3173a62eb9 stdout : vertical align outputs for better readibility 2023-04-16 13:59:27 +03:00
Georgi GerganovGitHubStephan Walter
 		e95b6554b4 ggml : add Q8_0 quantization for intermediate results (#951) 
* ggml : add Q8_0 quantization for intermediate results

* quantize-stats : fix test + add it to Makefile default

* Q8: use int8_t, AVX/AVX2 optimizations

* ggml : fix quantize_row_q8_0() ARM_NEON rounding

* minor : updates after rebase to latest master

* quantize-stats : delete obsolete strings

* ggml : fix q4_1 dot func

---------

Co-authored-by: Stephan Walter <stephan@walter.name>
 2023-04-15 17:53:22 +03:00
Georgi Gerganov aa485cee33 ggml : use posix_memalign on non-Windows env 2023-04-15 14:25:45 +03:00
Georgi Gerganov 1623a6e9b4 ggml : minor 2023-04-14 13:31:29 +03:00
Georgi Gerganov c14e0d2f23 ggml : always allocate buffers with size multiple of GGML_MEM_ALIGN 2023-04-14 13:31:15 +03:00
Georgi Gerganov 0f07cacb05 ggml : fix q4_1 dot product types 2023-04-14 09:45:42 +03:00
Georgi Gerganov a3a2a0eda8 ggml : add GGML_DEFAULT_N_THREADS 2023-04-13 18:36:48 +03:00
Georgi GerganovandGitHub d990e3fffc ggml : speed-up ggml_vec_dot_q4_1() ARM_NEON + 32-bit ARM support (#900) 
* ggml : speed-up q4_1 ARM_NEON by ~5%

* ggml : implement vaddvq when missing

* ggml : implement vminvq and vmaxvq when missing

* ggml : implement vzip when missing

* ggml : fix comment

* ggml : try to use correct ifdef
 2023-04-13 18:32:36 +03:00
Georgi Gerganov 9190e8eac8 llama : merge llama_internal.h into llama.h 
Hide it behind an #ifdef
 2023-04-13 18:04:45 +03:00
Georgi Gerganov c85980acd0 gitignore : benchmark 2023-04-13 18:01:33 +03:00
Georgi GerganovandGitHub f76cb3a34d readme : change "GPU support" link to discussion 2023-04-12 14:48:57 +03:00
Georgi GerganovandGitHub 782438070f readme : update hot topics with link to "GPU support" issue 2023-04-12 14:31:12 +03:00
Georgi Gerganov 461ba9e66e ggml : fix WASM build 2023-04-10 23:20:01 +03:00
Georgi Gerganov c3ac702e5e ggml : add ggml_cont() + optimize ggml_cpy() for contiguous dst 2023-04-10 22:42:28 +03:00
Georgi Gerganov 9d634ef452 ggml : remove trailing whitespaces 2023-04-10 22:42:28 +03:00
Georgi Gerganov 684da25926 ggml : fix quantize_row_q4_1() ARM_NEON (close #876) 2023-04-10 19:29:48 +03:00
Georgi GerganovandGitHub eeaa7b0492 ggml : multi-thread ggml_rope() (~3-4 times faster on M1) (#781) 2023-04-05 22:11:03 +03:00
Georgi GerganovandGitHub 986b6ce9f9 ggml, llama : avoid heavy V transpose + improvements (#775) 
ggml :

- added ggml_view_3d()
- ggml_view_tensor() now inherits the stride too
- reimplement ggml_cpy() to account for dst stride
- no longer require tensor->data to be memory aligned

llama :

- compute RoPE on 32-bit tensors (should be more accurate)
- store RoPE-ed K in the KV cache
- store transposed V in the KV cache (significant speed-up)
- avoid unnecessary Q copy
 2023-04-05 22:07:33 +03:00
Georgi GerganovandGitHub 3416298929 Update README.md 2023-04-05 19:54:30 +03:00
Georgi Gerganov 62b3e81aae media : add logos and banners 2023-04-05 18:58:31 +03:00
Georgi GerganovandGitHub 8d10406d6e readme : change logo + add bindings + add uis + add wiki 2023-04-05 18:56:20 +03:00
Georgi GerganovandGitHub 3df890aef4 readme : update supported models 2023-03-30 22:31:54 +03:00
Georgi Gerganov 77efdf5a50 ggml : fix NEON signs (close #620, #622) 2023-03-30 20:27:32 +03:00
Georgi Gerganov b51c717d5c ggml : init time on first ggml_init() call 2023-03-29 22:15:34 +03:00
Georgi Gerganov 0ba76c1e73 llama : fix compile warnings when reading the vocab 2023-03-29 22:13:12 +03:00
Georgi Gerganov cea1c85948 ggml : add ARM_NEON dequantize_row_q4_1() 2023-03-29 22:10:01 +03:00
Georgi Gerganov f202ada131 ggml : add ARM_NEON quantize_row_q4_1() 2023-03-29 22:03:07 +03:00
Georgi Gerganov 3b44d30d9b ggml : add ARM_NEON ggml_vec_dot_q4_1() 2023-03-29 22:03:07 +03:00
Georgi GerganovandGitHub b467702b87 readme : fix typos 2023-03-29 19:38:31 +03:00
Georgi GerganovandGitHub 516d88e75c readme : add GPT4All instructions (close #588) 2023-03-29 19:37:20 +03:00
Georgi Gerganov 53635c081c py : add GPT4All conversion script 
For now: copy-paste
Too much time for me to deduplicate the python code
 2023-03-29 19:29:52 +03:00
Georgi Gerganov 96f9c0506f ci : make ctest verbose, hopefully we see what is wrong with the sanitizer 2023-03-28 20:01:09 +03:00
Georgi Gerganov d502bc7c9d tests : free llama context at the end of the test 2023-03-28 19:51:55 +03:00
Georgi Gerganov e0670260fb gitignore : add "embedding" 2023-03-28 18:34:35 +03:00
Georgi GerganovandGitHub 348d6926ee Add logo to README.md 2023-03-26 10:20:49 +03:00
Georgi Gerganov c2b25b6912 Fix colors enabling on WIN32 2023-03-25 21:53:39 +02:00
Georgi Gerganov 79b2b266db If n_predict == -1, generate forever 2023-03-25 21:51:41 +02:00
Georgi Gerganov e2d490dafd Inifinite generation via context swapping (#71) 2023-03-25 21:36:22 +02:00
Georgi Gerganov 03f7e33560 Cleanup STL headers + fix embedding examples + minor stuff 2023-03-25 20:51:14 +02:00
Georgi Gerganov 55ad42af84 Move chat scripts into "./examples" 2023-03-25 20:37:09 +02:00
Georgi Gerganov a316a425d0 Overhaul the examples structure 
- main -> examples
- utils -> examples (renamed to "common")
- quantize -> examples
- separate tools for "perplexity" and "embedding"

Hope I didn't break something !
 2023-03-25 20:26:40 +02:00
Georgi GerganovandGitHub ecbe466a36 Retire the ggml_mul_mat() branch for transposed src0 (#500) 
* Retire the ggml_mul_mat() for transposed src0

- It can always be made contiguous with ggml_cpy()
- The code is now simplified
- The results are deterministic in respect to num threads

* SIMD-ify dequantize_row_q4_0() for ARM_NEON (#502)

* Attempt to SIMD-ify dequantize_row_q4_0() for ARM_NEON

* Fix dequantization - forgot to interleave the quants
 2023-03-25 19:47:21 +02:00
Georgi Gerganov 502a400192 Disable prompt verbosity by default and add option to enable (#480) 2023-03-25 17:17:16 +02:00
Georgi Gerganov 4640eff23d Don't interefe with BLAS for large prompts by running only 1 thread 2023-03-25 17:03:10 +02:00
Georgi Gerganov ab77d76312 Add longer DAN prompt for testing big batch numbers 2023-03-25 16:49:09 +02:00
Georgi GerganovandGitHub 4a7129acd2 Remove obsolete information from README 2023-03-25 16:30:32 +02:00
Georgi Gerganov 6b6dbc8910 Remove obsolete assert and fix compiler warning 2023-03-25 16:22:05 +02:00
Georgi Gerganov 2a2e63ce05 Fix nasty bug in ggml_compute_forward_mul_mat_f32() and reenable BLAS 2023-03-25 16:10:14 +02:00
Georgi Gerganov 8520fc310e Disable BLAS altogether - the bug is not just for qunatized mat mul 2023-03-24 23:47:06 +02:00
Georgi Gerganov b3f460e941 Disable BLAS branch in mul_mat - seems there is a bug 2023-03-24 23:39:17 +02:00
Georgi GerganovandGitHub 04c6f5ed6f Immediately start processing the prompt before user input has been provided (#476) 2023-03-24 23:17:58 +02:00
Georgi GerganovandGitHub 7a9b6c3a8b Reduce memory usage and allocate enough memory for largest context (#473) 
* Reduce memory usage and allocate enough memory for large contexts

* Simpler scratch buffer usage

* Reenable BLAS for quantized mul_mat

* Fix number of layers in 30B and 65B

* Fix KV cache size for F32
 2023-03-24 23:17:37 +02:00
Georgi Gerganov 31572d9665 Temporary bump the memory buffer size - hopefully fix issues from 483bab2e 2023-03-24 18:23:56 +02:00
Georgi Gerganov afd220d9c6 Properly free llama_context on failure 2023-03-24 17:21:01 +02:00
Georgi GerganovandGitHub b6b268d441 Add link to Roadmap discussion 2023-03-24 09:13:35 +02:00
Georgi Gerganov 3cd8dde0d1 Revert "Fix memory allocation issues and seg faults" 
This reverts commit 4870e455b3.

Will provide the correct fix later
 2023-03-24 06:22:28 +02:00
Georgi Gerganov 4870e455b3 Fix memory allocation issues and seg faults 2023-03-24 00:11:53 +02:00
Georgi GerganovandGitHub 483bab2e3d Avoid the transposed X branch in the Z = X * Y matrix multiplication (#439) 
Should make results reproducible for different number of threads and batch sizes
 2023-03-23 23:22:01 +02:00
Georgi Gerganov 4cc053b6d5 Remove oboslete command from Docker script 2023-03-23 22:39:44 +02:00
Georgi Gerganov 0ba5a3a9a5 Obsolete 2023-03-23 22:32:21 +02:00
Georgi GerganovandGitHub 93208cfb92 Adjust repetition penalty .. 2023-03-23 10:46:58 +02:00
Georgi GerganovandGitHub 03ace14cfd Add link to recent podcast about whisper.cpp and llama.cpp 2023-03-23 09:48:51 +02:00
Georgi Gerganov ae44e23ee3 When seed <= 0 - use the clock to generate one 2023-03-22 07:47:15 +02:00
Georgi Gerganov 928480ef5b Init llama_context_params properly from CLI (#370) 2023-03-22 07:45:14 +02:00
Georgi GerganovandGitHub 56817b1f88 Remove temporary notice and update hot topics 2023-03-22 07:34:02 +02:00
Georgi GerganovandGitHub f5a77a629b Introduce C-style API (#370) 
* Major refactoring - introduce C-style API

* Clean up

* Add <cassert>

* Add <iterator>

* Add <algorithm> ....

* Fix timing reporting and accumulation

* Measure eval time only for single-token calls

* Change llama_tokenize return meaning
 2023-03-22 07:32:36 +02:00
Georgi GerganovandGitHub 3366853e41 Add notice about pending change 2023-03-21 22:57:35 +02:00
Georgi GerganovandGitHub 0f61352708 Update issue templates 2023-03-21 19:47:27 +02:00
Georgi GerganovandGitHub 1daf4dd712 Minor style changes 2023-03-21 18:10:32 +02:00
Georgi Gerganov dc6a845b85 Add chat.sh script 2023-03-21 18:09:46 +02:00
Georgi Gerganov 3bfa3b43b7 Fix convert script, warnings alpaca instructions, default params 2023-03-21 17:59:16 +02:00
Georgi Gerganov 8f644a0a85 Change default repeat_penalty to 1.0 
I feel this penalty is not really helping.
Especially for the example from the README it makes results pretty bad
 2023-03-21 17:32:14 +02:00
Georgi GerganovandGitHub eb34620aec Add tokenizer test + revert to C++11 (#355) 
* Add test-tokenizer-0 to do a few tokenizations - feel free to expand
* Added option to convert-pth-to-ggml.py script to dump just the vocabulary
* Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests)
* Added utility to load vocabulary file from previous point (temporary implementation)
* Avoid using std::string_view and drop back to C++11 (hope I didn't break something)
* Rename gpt_vocab -> llama_vocab
* All CMake binaries go into ./bin/ now
 2023-03-21 17:29:41 +02:00
Georgi Gerganov 4545539d71 Rename script 2023-03-19 21:58:51 +02:00
Georgi Gerganov edeba28366 Add temporary helper script for Alpaca chat 2023-03-19 21:57:48 +02:00
Georgi GerganovandGitHub 160bfb217d Update hot topics to mention Alpaca support 2023-03-19 19:51:55 +02:00
Georgi Gerganov c494ed5b94 Fix off-by-one bug (#115) 2023-03-19 19:46:32 +02:00
Georgi Gerganov c1c7026b47 Fix python stuff (#109) 2023-03-19 19:33:18 +02:00
Georgi Gerganov 70f01cb863 Drop trailing new line from file prompts (#80) 2023-03-19 19:05:04 +02:00
Georgi GerganovandGitHub a4e63b73df Add instruction for using Alpaca (#240) 2023-03-19 18:49:50 +02:00
Georgi Gerganov 9e1707218a Add "--instruct" argument for usage with Alpaca (#240) 
Also start adding prompts in "./prompts"
 2023-03-19 18:37:02 +02:00
Georgi Gerganov 22213a17b5 Change RMSNorm eps to 1e-6 (#173) 
I think this is what is used in the Python code
 2023-03-19 17:30:00 +02:00
Georgi Gerganov 4f54609110 Default to 4 threads (#243) 2023-03-17 21:46:46 +02:00
Georgi GerganovandGitHub e81b9c81c1 Update Contributing section 2023-03-17 20:30:04 +02:00
Georgi GerganovandGitHub 721311070e Update README.md 2023-03-16 15:00:09 +02:00
Georgi GerganovandGitHub ac15de7895 Expand "Contributing" section 2023-03-16 08:55:13 +02:00
Georgi GerganovandGitHub 273abc47ff Update hot topics - RMSnorm 2023-03-16 07:12:12 +02:00
Georgi GerganovandGitHub 97ab2b2578 Add Misc section + update hot topics + minor fixes 2023-03-14 09:43:52 +02:00
Georgi Gerganov c09a9cfb06 CMake build in Release by default (#75) 2023-03-13 21:22:15 +02:00
Georgi GerganovandGitHub 7ec903d3c1 Update contribution section, hot topics, limitations, etc. 2023-03-13 19:21:51 +02:00
Georgi Gerganov 4497ad819c Print system information 2023-03-13 19:15:08 +02:00
Georgi GerganovandGitHub 84d9015c4a Use vdotq_s32 to improve performance (#67) 
* 10% performance boost on ARM

* Back to original change
 2023-03-13 18:36:44 +02:00
Georgi GerganovandGitHub 1808ee0500 Add initial contribution guidelines 2023-03-13 09:42:26 +02:00
Georgi Gerganov c80e2a8f2a Revert "10% performance boost on ARM" 
This reverts commit 113a9e83eb.

There are some reports for illegal instruction.
Moved this stuff to vdotq_s32 branch until resolve
 2023-03-13 01:28:08 +02:00
Georgi Gerganov 54a0e66ea0 Check for vdotq_s32 availability 2023-03-13 01:21:03 +02:00
Georgi Gerganov 543c57e991 Ammend to previous commit - forgot to update non-QRDMX branch 2023-03-13 01:05:24 +02:00
Georgi Gerganov 113a9e83eb 10% performance boost on ARM 2023-03-13 00:56:10 +02:00
Georgi GerganovandGitHub 1a0a74300f Update README.md 2023-03-12 23:39:01 +02:00