2 Commits
Author SHA1 Message Date
Kevin WangandGitHub 470939d483 common : preallocate sampling token data vector (#8363)
`emplace_back` repeatedly-called is slower than preallocating the vector to the vocab size and directly inserting the data. Some rudimentary profiling with `chrono` improves the performance of this block of code from ~500us/op to ~40us/op.

Overall, this slightly improves the sampling performance which has a more substantial impact for the `examples/lookahead` implementation -- I am able to see a ~10% performance boost in lookahead inference.
2024-07-08 10:26:53 +03:00
Kevin WangandGitHub ffd00797d8 common : avoid unnecessary logits fetch (#8358) 2024-07-08 09:31:55 +03:00