nikola/llama_cpp
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

server : optimize get_token_probabilities (#24796)

Browse Source 
Use std::partial_sort to order only the requested top-n tokens instead
of the full vocabulary

    logprobs sort: vocab=128000 n_top=0 iters=100
    full    sort:   8555.6 us/op
    partial sort:    704.3 us/op

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
This commit is contained in:
Adrien Gallouët
2026-06-19 23:26:54 +02:00
committed by GitHub
parent e475fa2b5f
commit 4b48a53b6c
3 changed files with 27 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/server/server-common.h
+1 -1
View File
@@ -326,7 +326,7 @@ json format_response_rerank(
// other utils
//
std::vector<llama_token_data> get_token_probabilities(llama_context * ctx, int idx);
std::vector<llama_token_data> get_token_probabilities(llama_context * ctx, int idx, size_t n_top);
std::string safe_json_to_str(const json & data);