Class Branch

Child branch handles

Whether this branch has been disposed

Position at which this branch was forked from its parent (0 for root branches)

Internal handle (for debugging)

True if this branch holds a KV lease

True if this branch has no children

Parent branch handle, or null if root

Branch's perplexity (exp of mean surprisal)

Branch's current position (number of tokens decoded)

Sampling-level perplexity (from filtered distribution)

Returns perplexity from the distribution actually sampled from (after top-k/p/temp/penalties). Useful for policy priors and monitoring sampler chain impact.

Compare with perplexity which is model-level (raw logits).

Async iterator — generate tokens until EOG

Commit-before-yield semantics: every yielded token is already written to KV and accepted into the sampler. Breaking out of the loop is clean — no orphaned uncommitted tokens, perplexity reflects all yielded tokens.

For inspect-before-commit (speculative decoding, tree search), use the produce/commit protocol directly.

Record token in the sampler's repeat/presence penalty window

Clear all static logit biases from this branch

Clear all steer biases from this branch

Removes any dynamic logit adjustments set by steer(). Call this after each generation step if your steer constraints are computed per-step (e.g., N-gram blocking where the blocked set changes as text grows).

Accept and decode — update branch state, then write token to KV

Accepts the token into the sampler penalty window (for correct PPL measurement), then decodes (writing to KV cache via AsyncWorker on the libuv thread pool) and captures the resulting logits for the next produce() call. Accept-first ordering with rollback: if decode throws, sampler/grammar/metrics are restored from clones.

Fork this branch to a new sequence (async)

Async contract: local branches resolve immediately; cloud branches may perform an HTTP round-trip. Use forkSync when you know the branch is local and want zero-overhead forking.

Fork this branch to a new sequence (sync)

The child shares the parent's KV prefix in memory (metadata-only under unified KV, no KV buffer copy). Logits, sampler state, and perplexity tracker are cloned so the child can diverge independently. Fork from any branch — root or intermediate — to build arbitrarily deep trees.

Call reseedSampler() on each child for stochastic diversity.

Get a copy of this branch's captured logits snapshot.

Returns n_vocab floats — the raw logit distribution from the last prefill() or commit() call.

Returns an independent copy of the branch's internal snapshot. The returned Float32Array is safe to hold across async boundaries and is not affected by subsequent decode operations.

Compute entropy of the branch's logits distribution

Measures model uncertainty from the branch's captured logits snapshot:

• Low entropy: Model is confident (peaked distribution)
• High entropy: Model is uncertain (flat distribution)

Operates directly on state->logits_snapshot — no JS round-trip.

Compute surprisal (negative log-likelihood) for a specific token

Measures how "surprising" the model finds the given token from the branch's captured logits snapshot:

• Low surprisal: Model expected this token (high probability)
• High surprisal: Model didn't expect this token (low probability)

Operates directly on state->logits_snapshot — no JS round-trip.

Bulk-decode tokens into the branch's KV cache and capture logits.

tokens.length is the total count to process; the branch's nBatch (set at Branch.create) controls how many are sent per llama_decode call. E.g. 500 tokens with nBatch=64 → 8 calls (7×64 + 1×52).

Advances position by tokens.length. Stores final logits into the branch's internal snapshot — the next produce()/sample() reads from it.

Does NOT accept tokens into the repeat-penalty window — for external tokens (user input between turns), not model-generated tokens. For model output, use commit() which does accept + decode.

The primary way to feed tokens into a branch's KV cache.

Sample next token without advancing state (async)

Async contract: local branches resolve immediately; cloud branches may perform an HTTP round-trip. Use produceSync when you know the branch is local and want zero-overhead sampling.

Sample next token without advancing state (sync)

Same as produce but synchronous. Use when you know the branch is local and want to avoid the microtick overhead of a promise.

Discard this branch (async)

Async contract: local branches resolve immediately; cloud branches may perform an HTTP round-trip. Use pruneSync when you know the branch is local.

RESTRICT mode: throws if children exist. Use pruneSubtree to cascade-delete an entire subtree.

Discard this branch and all its descendants (async)

Async contract: local branches resolve immediately; cloud branches may perform an HTTP round-trip. Use pruneSubtreeSync when you know the branch is local.

Discard this branch and all its descendants — CASCADE delete (sync)

Iterative post-order traversal: prunes children first, then this branch. Use when tearing down an entire subtree (e.g. abandoned search path). Sets disposed synchronously.

Discard this branch — remove its divergent KV entries and free the handle (sync)

Only removes KV entries divergent from the shared prefix; sibling branches are unaffected. The disposed flag is set synchronously — any call to produce(), commit(), etc. after prune() will throw immediately.

RESTRICT mode: throws if children exist. Use pruneSubtreeSync to cascade-delete an entire subtree.

Reseed the sampler's PRNG for diversity after fork()

CRITICAL for parallel generation: Without reseeding, all forked branches produce identical outputs because they share the same PRNG state.

Only affects stochastic samplers (temperature > 0). Greedy samplers are unchanged.

Sample next token from branch's logits snapshot

Applies the branch's full sampler chain (top-k, top-p, temperature, repeat/presence penalties) to the captured logits.

Replace or remove the grammar constraint

Pass a GBNF grammar string to constrain generation. Pass empty string or undefined to remove the constraint. The grammar state is cloned on fork(), so sibling branches can diverge independently after hot-swap.

Set lazy grammar — unconstrained until trigger, then grammar-constrained

Generation runs freely until a trigger pattern or token fires, at which point the grammar activates and constrains subsequent tokens. Used for tool-call generation: model writes freely until <tool_call>, then grammar forces valid XML structure.

The grammar state is cloned on fork(), so sibling branches can diverge independently. Call again after a tool result prefill to reset.

Set static logit biases on this branch

Unlike steer (which is NOT inherited on fork), logit biases ARE cloned when forking. Use for persistent constraints that should propagate to child branches.

Applied during sample() in order: Grammar -> Logit Bias -> Steer -> Sampler Chain

Replace the sampler chain with new parameters (memoized)

If the new params match the current chain's params, this is a no-op. Otherwise the old chain is freed and a new one is created. Use for Entropy-Driven Temperature (EDT) and other adaptive sampling strategies that adjust parameters per-step.

Apply dynamic logit adjustments for this branch only

Unlike logit_bias in sampling params (which is cloned on fork), steer biases are NOT inherited by child branches. Each branch manages its own steer state independently. This makes steer ideal for path-dependent constraints.

Use cases:

• tsampler: Block tokens that would create repeated N-grams based on this branch's specific generation history
• Diverse beam search: Penalize tokens already chosen by sibling beams to encourage output diversity across the beam
• Dynamic constraints: Apply token restrictions that change per-step

Sampling order: Grammar → Logit Bias → Steer → Sampler Chain

Create a root branch at the given position

The branch takes ownership of the sequence and creates its own sampler chain from the provided params. Call prefill() to decode prompt tokens and capture the logit distribution before forking.