293bebe077
rpc : fix segfault with nkvo ( #9389 )
...
* rpc : fix nkvo
* rpc : buf_size must not be static
ref: #9337
---------
Co-authored-by: slaren <slarengh@gmail.com >
2024-09-09 18:40:10 +03:00
Radoslav Gerganov and GitHub
54f376d0b9
rpc : update README [no ci] ( #9320 )
...
Update README with instructions how to offload model layers to both
local and remote devices
2024-09-09 11:04:39 +03:00
Radoslav Gerganov and GitHub
82e3b03c11
rpc : make RPC servers come first in the device list ( #9296 )
...
* rpc : make RPC servers come first in the device list
* rpc : disable options for non-RPC builds
* rpc : rpc_count always zero for non-RPC builds
2024-09-04 11:08:32 +03:00
Radoslav Gerganov and GitHub
1b6ff90ff8
rpc : print error message when failed to connect endpoint ( #9042 )
2024-08-19 10:11:45 +03:00
Radoslav Gerganov and GitHub
18eaf29f4c
rpc : prevent crashes on invalid input ( #9040 )
...
Add more checks which prevent RPC server from crashing if invalid input
is received from client
2024-08-19 10:10:21 +03:00
Radoslav Gerganov and GitHub
1f67436c5e
ci : enable RPC in all of the released builds ( #9006 )
...
ref: #8912
2024-08-12 19:17:03 +03:00
Radoslav Gerganov and GitHub
e65bbf606c
llama-bench : fix RPC indication ( #7936 )
...
Show "<backend_name>+RPC" when RPC offloading is used
2024-06-14 16:47:41 +03:00
Radoslav Gerganov and GitHub
172c825684
rpc : fix ggml_backend_rpc_supports_buft() ( #7918 )
2024-06-13 15:18:44 +03:00
bde7cd3cd9
llama : offload to RPC in addition to other backends ( #7640 )
...
* llama : offload to RPC in addition to other backends
* - fix copy_tensor being called on the src buffer instead of the dst buffer
- always initialize views in the view_src buffer
- add RPC backend to Makefile build
- add endpoint to all RPC object names
* add rpc-server to Makefile
* Update llama.cpp
Co-authored-by: slaren <slarengh@gmail.com >
---------
Co-authored-by: slaren <slarengh@gmail.com >
2024-06-03 20:03:26 +03:00
Radoslav Gerganov and GitHub
210d99173d
llama-bench : add support for the RPC backend ( #7435 )
2024-05-29 14:45:44 +03:00
Radoslav Gerganov and GitHub
2b737caae1
rpc : resource management rework ( #7562 )
...
* rpc : resource management rework
* address review comments
2024-05-28 18:13:36 +03:00
Radoslav Gerganov and GitHub
db10f01310
rpc : track allocated buffers ( #7411 )
...
* rpc : track allocated buffers
ref: #7407
* rpc : pack rpc_tensor tightly
2024-05-20 16:36:55 +03:00
Radoslav Gerganov and GitHub
f4bd8b3d26
rpc : set SO_REUSEADDR for the server socket ( #7320 )
...
ref: #7293
2024-05-17 17:25:44 +03:00
Radoslav Gerganov and GitHub
ee94172d33
server : add support for the RPC backend ( #7305 )
...
ref: #7292
2024-05-17 10:00:17 +03:00
Radoslav Gerganov
9afdffe70e
rpc : get available mem for the CPU backend
...
This can be overridden with the -m command line option
ref: #7293
2024-05-16 12:04:08 +03:00
Radoslav Gerganov
3b3963c55c
rpc : add command line arg for specifying backend memory
...
ref: #7293
2024-05-16 09:58:29 +03:00
Radoslav Gerganov and GitHub
5e31828d3e
ggml : add RPC backend ( #6829 )
...
* ggml : add RPC backend
The RPC backend proxies all operations to a remote server which runs a
regular backend (CPU, CUDA, Metal, etc).
* set TCP_NODELAY
* add CI workflows
* Address review comments
* fix warning
* implement llama_max_devices() for RPC
* Address review comments
* Address review comments
* wrap sockfd into a struct
* implement get_alignment and get_max_size
* add get_device_memory
* fix warning
* win32 support
* add README
* readme : trim trailing whitespace
* Address review comments
* win32 fix
* Address review comments
* fix compile warnings on macos
2024-05-14 14:27:19 +03:00
Radoslav Gerganov and GitHub
60f819a2b1
Add section to README on how to run the project on Android ( #130 )
2023-03-14 15:30:08 +02:00
