nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 30 Packages Projects Releases Wiki Activity
5,303 Commits 1 Branch 0 Tags
bc4e1128f78be0fbb4e2fa630adb6a04b969ac68
Commit Graph
35 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Radoslav GerganovandGitHub 553a5c3a9f rpc : do not wait for response when sending RPC_CMD_SET_TENSOR (#12943) 
RPC_CMD_SET_TENSOR always returns an empty response and we send this 4
times per token. We can improve TG speed if we don't wait for this empty
response.

The performance impact of this change depends on the network latency.
 2025-04-25 10:08:08 +03:00
Radoslav GerganovandGitHub 2cca6c01e4 rpc : add command line option for number of threads for the CPU backend (#13060) 
closes #13051
 2025-04-23 10:32:49 +03:00
Radoslav GerganovandGitHub 2db9ba1464 rpc : add RPC_CMD_HELLO (#12955) 
Add RPC_CMD_HELLO for getting the version of the protocol implemend by
the server. Follow the semantic versioning rules at https://semver.org

Hopefully this bring better user experience when we make breaking
changes at the protocol level and avoid issues like #12465
 2025-04-18 10:13:42 +03:00
Radoslav GerganovandGitHub c772d54926 rpc : use ggml_context_ptr (#12938) 2025-04-14 13:59:34 +03:00
Radoslav GerganovandGitHub ef03229ff4 rpc : update README for cache usage (#12620) 2025-03-28 09:44:13 +02:00
Radoslav GerganovandGitHub ab6ab8f809 rpc : send hash when tensor data is above some fixed threshold (#12496) 
* rpc : send hash when tensor data is above some fixed threshold

ref #10095

* rpc : put cache under $HOME/.cache/llama.cpp

* try to fix win32 build

* another try to fix win32 build

* remove llama as dependency
 2025-03-28 08:18:04 +02:00
Radoslav GerganovandGitHub 1bef571f6a arg : list RPC devices first when using --list-devices (#11655) 
List devices in the same order as they appear when evaluating the model
and splitting tensors across devices, i.e. RPC devices come first in the
list.

ref #11435
 2025-02-04 18:16:20 +02:00
Radoslav GerganovandGitHub 6da5bec81c rpc : better caching of the base buffer pointer (#11331) 
There is no need to use map, just store the base pointer in the buffer
context.
 2025-01-21 15:06:41 +02:00
Radoslav GerganovandGitHub 667d72846c rpc : early register backend devices (#11262) 
Early register RPC devices and do not propagate RPC specifics in the
llama model structures.

ref: #10609
 2025-01-17 10:57:09 +02:00
Radoslav GerganovandGitHub 1244cdcf14 ggml : do not define GGML_USE_CUDA when building with GGML_BACKEND_DL (#11211) 
Build fails when using HIP and GGML_BACKEND_DL:
```
/usr/bin/ld: ../ggml/src/libggml.so: undefined reference to `ggml_backend_cuda_reg'
collect2: error: ld returned 1 exit status
```
This patch fixes this.
 2025-01-13 13:31:41 +02:00
Radoslav GerganovandGeorgi Gerganov c792dcf488 ggml : allow loading backend with env variable (ggml/1059) 
ref: #1058
 2025-01-08 13:40:18 +02:00
Radoslav GerganovandGitHub a4dd490069 rpc : code cleanup (#11107) 
Remove duplicated macros, use GGML_LOG_ERROR for errors
 2025-01-07 08:37:02 +02:00
Radoslav GerganovandGitHub 86bf31cfe6 rpc-server : add support for the SYCL backend (#10934) 2024-12-23 10:39:30 +02:00
Radoslav GerganovandGitHub d5ebd79c76 rpc : pack only RPC structs (#9959) 2024-10-21 13:35:40 +03:00
Radoslav GerganovandGitHub afd9909a64 rpc : backend refactoring (#9912) 
* rpc : refactor backend

Use structs for RPC request/response messages

* rpc : refactor server
 2024-10-18 14:33:58 +03:00
Radoslav GerganovandGitHub 841713e1e4 rpc : enable vulkan (#9714) 
closes #8536
 2024-10-03 13:00:52 +03:00
Radoslav GerganovGitHub0cc4m
 		00b7317e63 vulkan : do not use tensor->extra (#9407) 
* vulkan : do not use tensor->extra

This patch allows using the Vulkan backend with the RPC backend as
tensor->extra is no longer used.

Ref: #8536

* Adapt GGML_VULKAN_CHECK_RESULTS to extra removal (#2)

---------

Co-authored-by: 0cc4m <picard12@live.de>
 2024-10-02 13:49:16 +03:00
Radoslav GerganovGitHubslaren
 		293bebe077 rpc : fix segfault with nkvo (#9389) 
* rpc : fix nkvo

* rpc : buf_size must not be static

ref: #9337

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2024-09-09 18:40:10 +03:00
Radoslav GerganovandGitHub 54f376d0b9 rpc : update README [no ci] (#9320) 
Update README with instructions how to offload model layers to both
local and remote devices
 2024-09-09 11:04:39 +03:00
Radoslav GerganovandGitHub 82e3b03c11 rpc : make RPC servers come first in the device list (#9296) 
* rpc : make RPC servers come first in the device list

* rpc : disable options for non-RPC builds

* rpc : rpc_count always zero for non-RPC builds
 2024-09-04 11:08:32 +03:00
Radoslav GerganovandGitHub 1b6ff90ff8 rpc : print error message when failed to connect endpoint (#9042) 2024-08-19 10:11:45 +03:00
Radoslav GerganovandGitHub 18eaf29f4c rpc : prevent crashes on invalid input (#9040) 
Add more checks which prevent RPC server from crashing if invalid input
is received from client
 2024-08-19 10:10:21 +03:00
Radoslav GerganovandGitHub 1f67436c5e ci : enable RPC in all of the released builds (#9006) 
ref: #8912
 2024-08-12 19:17:03 +03:00
Radoslav GerganovandGitHub e65bbf606c llama-bench : fix RPC indication (#7936) 
Show "<backend_name>+RPC" when RPC offloading is used
 2024-06-14 16:47:41 +03:00
Radoslav GerganovandGitHub 172c825684 rpc : fix ggml_backend_rpc_supports_buft() (#7918) 2024-06-13 15:18:44 +03:00
Radoslav GerganovGitHubslaren
 		bde7cd3cd9 llama : offload to RPC in addition to other backends (#7640) 
* llama : offload to RPC in addition to other backends

* - fix copy_tensor being called on the src buffer instead of the dst buffer

- always initialize views in the view_src buffer

- add RPC backend to Makefile build

- add endpoint to all RPC object names

* add rpc-server to Makefile

* Update llama.cpp

Co-authored-by: slaren <slarengh@gmail.com>

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2024-06-03 20:03:26 +03:00
Radoslav GerganovandGitHub 210d99173d llama-bench : add support for the RPC backend (#7435) 2024-05-29 14:45:44 +03:00
Radoslav GerganovandGitHub 2b737caae1 rpc : resource management rework (#7562) 
* rpc : resource management rework

* address review comments
 2024-05-28 18:13:36 +03:00
Radoslav GerganovandGitHub db10f01310 rpc : track allocated buffers (#7411) 
* rpc : track allocated buffers

ref: #7407

* rpc : pack rpc_tensor tightly
 2024-05-20 16:36:55 +03:00
Radoslav GerganovandGitHub f4bd8b3d26 rpc : set SO_REUSEADDR for the server socket (#7320) 
ref: #7293
 2024-05-17 17:25:44 +03:00
Radoslav GerganovandGitHub ee94172d33 server : add support for the RPC backend (#7305) 
ref: #7292
 2024-05-17 10:00:17 +03:00
Radoslav Gerganov 9afdffe70e rpc : get available mem for the CPU backend 
This can be overridden with the -m command line option

ref: #7293
 2024-05-16 12:04:08 +03:00
Radoslav Gerganov 3b3963c55c rpc : add command line arg for specifying backend memory 
ref: #7293
 2024-05-16 09:58:29 +03:00
Radoslav GerganovandGitHub 5e31828d3e ggml : add RPC backend (#6829) 
* ggml : add RPC backend

The RPC backend proxies all operations to a remote server which runs a
regular backend (CPU, CUDA, Metal, etc).

* set TCP_NODELAY

* add CI workflows

* Address review comments

* fix warning

* implement llama_max_devices() for RPC

* Address review comments

* Address review comments

* wrap sockfd into a struct

* implement get_alignment and get_max_size

* add get_device_memory

* fix warning

* win32 support

* add README

* readme : trim trailing whitespace

* Address review comments

* win32 fix

* Address review comments

* fix compile warnings on macos
 2024-05-14 14:27:19 +03:00
Radoslav GerganovandGitHub 60f819a2b1 Add section to README on how to run the project on Android (#130) 2023-03-14 15:30:08 +02:00