255582687b
* spec: support MTP * fix batch size * rename files * cont : simplify (#7) * MTP: clean-up (#9) * MTP: clean-up * review: use llama_context_type instead of llama_graph_type * review: remove llama_model_has_mtp * review: fix convert issues * convert: fix pycheck * review: formatting * use `mtp-` for identifying mtp models * convert: fix mtp conversion * mtp -> draft-mtp * remove unused llama_arch * add need_embd in speculative * llama: allow partial seq_rm for GDN models for speculative decoding Currently speculative checkpoint needs to restart from a checkpoint after some draft tokens are not accepted, this leads to some wastage in running the target again. This PR adds the ability to rollback upto `draft_max` by storing the GDN intermediates. * fix pending state * vulkan: add GDN partial rollback * meta: extend check to axis 1 * metal: add GDN partial rollback Extend the gated delta net kernel to store intermediate states for partial rollback support on the Metal backend. - Add K (snapshot slot count) as a function constant - Read input state from slot 0 of the 3D state tensor - Write intermediate states to different slots during token loop - For K=1, maintain backward-compatible single-slot behavior Ref: https://github.com/ggml-org/llama.cpp/commit/8c05923630110223669f069af2000e9cf10c02bc Assisted-by: llama.cpp:local pi * delta_net_base: use ggml_pad instead of new_tensor * review: add need_rs_seq * review: rename part_bounded to n_rs * review: deslop comments * review: rename, add asserts * server : adjust checkpoint logic (#11) * server : adjust checkpoint logic * cont : rm asserts * server-context: fix early exit * spec : fix compatibility with n-gram and add TODOs (#13) * metal : cleanup * llama : fix faulty bitwise check in recurrent memory * server : disable RS-based MTP in combination with other spec types * spec : add TODOs * cont : fix comment * cont : update comment * common : fix logic for ngram + mtp compat * llama-memory: enable checkpointing with partial rollback * cont: add test-case for loading into a dirty ctx * llama-memory-recurrent: clear rs_idx in clear * download: fix mtp path * llama-arch: fix enorm op * docs: update docs * conversion: fix type annotations --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
51 lines
1.5 KiB
C++
51 lines
1.5 KiB
C++
#pragma once
|
|
|
|
#include "llama.h"
|
|
|
|
#include <cstdint>
|
|
|
|
#define LLAMA_MAX_SEQ 256
|
|
|
|
struct llama_cparams {
|
|
uint32_t n_ctx; // context size used during inference
|
|
uint32_t n_ctx_seq; // context for a single sequence
|
|
uint32_t n_batch;
|
|
uint32_t n_ubatch;
|
|
uint32_t n_seq_max;
|
|
uint32_t n_rs_seq; // number of recurrent-state snapshots per seq for rollback
|
|
int32_t n_threads; // number of threads to use for generation
|
|
int32_t n_threads_batch; // number of threads to use for batch processing
|
|
|
|
float rope_freq_base;
|
|
float rope_freq_scale;
|
|
|
|
uint32_t n_ctx_orig_yarn;
|
|
// These hyperparameters are not exposed in GGUF, because all
|
|
// existing YaRN models use the same values for them.
|
|
float yarn_ext_factor;
|
|
float yarn_attn_factor;
|
|
float yarn_beta_fast;
|
|
float yarn_beta_slow;
|
|
|
|
bool embeddings;
|
|
bool embeddings_pre_norm; // also extract the hidden state before the final output norm
|
|
bool causal_attn;
|
|
bool offload_kqv;
|
|
bool flash_attn;
|
|
bool auto_fa;
|
|
bool fused_gdn_ar; // use fused gated delta net (autoregressive)
|
|
bool fused_gdn_ch; // use fused gated delta net (chunked)
|
|
bool auto_fgdn;
|
|
bool no_perf;
|
|
bool warmup;
|
|
bool op_offload;
|
|
bool kv_unified;
|
|
bool pipeline_parallel;
|
|
|
|
enum llama_context_type ctx_type;
|
|
enum llama_pooling_type pooling_type;
|
|
|
|
ggml_backend_sched_eval_callback cb_eval;
|
|
void * cb_eval_user_data;
|
|
};
|