Consolidated mutable state for a single branch. More...

#include <lloyal/branch.hpp>

Public Member Functions
void	poison ()

void	capture_logits (int32_t index)
	Copy the logits at batch row `index` into this branch's private snapshot.

Public Attributes
llama_context *	ctx = nullptr
	Llama context (not owned, must outlive branch)

const llama_model *	model = nullptr
	Llama model (not owned, must outlive branch)

llama_seq_id	seq_id = NO_LEASE
	KV cache sequence identifier (NO_LEASE when inactive)

llama_pos	position = 0
	Current decode position in the sequence.

llama_pos	fork_head = 0
	Parent's position at fork time (0 for root branches)

uint64_t	revision = 0
	Counts every change to the state fast_forward() moves: decodes, logits, sampling, grammar and bias.

uint64_t	fork_revision = 0
	The parent's revision at fork time (0 for root branches).

bool	poisoned = false
	A decode failed and left the KV ahead of these books (see decode::leaves_branch_intact).

uint32_t	img_slack_own = 0
	Embedding-row slack: cells beyond position-delta from decode_embd() (an M-RoPE image adds nx*ny cells but advances position by max(nx,ny)).

uint32_t	img_slack_total = 0

SamplerChainHandle	sampler_chain = 0
	Handle into BranchStore's sampler registry.

GrammarHandle	grammar = 0
	Handle into BranchStore's grammar registry.

bool	grammar_sample_first = false
	Grammar order for sample(): false masks the whole vocabulary before the chain (exact under truncation); true lets the chain pick first and asks the grammar about that one token, masking only when it is refused.

CachedSamplingParams	cached_params
	Params used to create current chain (for memoization)

boundaries::BoundaryTracker *	boundary_tracker = nullptr
	Token boundary detector (owned, optional)

std::vector< llama_logit_bias >	logit_bias
	Static token biases, cloned on fork.

std::function< void(llama_token_data_array &)>	steer_fn
	Dynamic logit callback, NOT cloned on fork.

MetricsHandle	metrics = 0
	Handle into BranchStore's metrics registry.

llama_token	last_token = -1
	Last token returned by sample()

std::vector< llama_token_data >	last_candidates
	Filtered candidates from last sample()

std::vector< float >	logits_snapshot
	Captured logit distribution (n_vocab floats)

bool	has_logits = false
	True once logits_snapshot holds a live distribution: capture_logits(), set_logits(), or a fork that cloned it.

std::vector< llama_token_data >	candidates_buffer
	Reusable scratch buffer for sampling (avoids O(n_vocab) allocs per sample call).

int	n_batch = DEFAULT_N_BATCH
	Batch size for decode operations.

int	n_vocab = 0
	Vocabulary size (cached for buffer pre-allocation)

uint16_t	generation = 0
	Slot generation counter (for ABA prevention)

bool	in_use = false
	True when slot is allocated to an active branch.

BranchHandle	parent = INVALID_HANDLE
	Parent branch (INVALID_HANDLE if root)

std::vector< BranchHandle >	children
	Child branches forked from this one.

Detailed Description

Consolidated mutable state for a single branch.

Each branch encapsulates all state needed for independent generation:

Forkable state (cloned by fork()):

KV cache sequence (via llama_memory_seq_cp)
Sampler chain (handle into BranchStore registry)
Grammar constraints (handle into BranchStore registry)
Boundary tracker (token boundary detection)
Metrics (handle into BranchStore registry)
Logits snapshot (captured distribution for deferred sampling)
Logit bias (static token-level adjustments)

Non-forkable state (NOT cloned by fork()):

steer_fn (dynamic callback — may capture references, unsafe to copy)

Sampling application order in sample(): Grammar → Logit Bias → Steer → Sampler Chain or, with grammar_sample_first: Logit Bias → Steer → Sampler Chain → Grammar check of the pick, falling back to the first order only when the pick is refused.

Definition at line 293 of file branch.hpp.

Member Function Documentation

◆ capture_logits()

void lloyal::branch::BranchState::capture_logits ( int32_t index )

inline

Copy the logits at batch row index into this branch's private snapshot.

The context's buffer is shared and overwritten by the next llama_decode; the snapshot is what sample()/get_logits() read. A branch with no vocab has nowhere to put them — refused loud, never copied into an empty buffer.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 354 of file branch.hpp.

◆ poison()

void lloyal::branch::BranchState::poison ( )

inline

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 312 of file branch.hpp.

Member Data Documentation

◆ boundary_tracker

boundaries::BoundaryTracker* lloyal::branch::BranchState::boundary_tracker = nullptr

Token boundary detector (owned, optional)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 335 of file branch.hpp.

◆ cached_params

CachedSamplingParams lloyal::branch::BranchState::cached_params

Params used to create current chain (for memoization)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 333 of file branch.hpp.

◆ candidates_buffer

std::vector<llama_token_data> lloyal::branch::BranchState::candidates_buffer

Reusable scratch buffer for sampling (avoids O(n_vocab) allocs per sample call).

Memory footprint per branch for large vocab models (128k tokens):

logits_snapshot: n_vocab * 4 bytes (~512KB)
candidates_buffer: n_vocab * sizeof(llama_token_data) (~1.5–2MB)
last_candidates: typically ~40 entries with top_k=40 (~480 bytes)

Todo:
Move to per-thread or SessionContext scratch arena for deep search trees.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 372 of file branch.hpp.

◆ children

std::vector<BranchHandle> lloyal::branch::BranchState::children

Child branches forked from this one.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 382 of file branch.hpp.

◆ ctx

llama_context* lloyal::branch::BranchState::ctx = nullptr

Llama context (not owned, must outlive branch)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 294 of file branch.hpp.

◆ fork_head

llama_pos lloyal::branch::BranchState::fork_head = 0

Parent's position at fork time (0 for root branches)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 299 of file branch.hpp.

◆ fork_revision

uint64_t lloyal::branch::BranchState::fork_revision = 0

The parent's revision at fork time (0 for root branches).

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 306 of file branch.hpp.

◆ generation

uint16_t lloyal::branch::BranchState::generation = 0

Slot generation counter (for ABA prevention)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 377 of file branch.hpp.

◆ grammar

GrammarHandle lloyal::branch::BranchState::grammar = 0

Handle into BranchStore's grammar registry.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 327 of file branch.hpp.

◆ grammar_sample_first

bool lloyal::branch::BranchState::grammar_sample_first = false

Grammar order for sample(): false masks the whole vocabulary before the chain (exact under truncation); true lets the chain pick first and asks the grammar about that one token, masking only when it is refused.

Reset to false whenever a grammar is attached; cloned by fork().

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 331 of file branch.hpp.

◆ has_logits

bool lloyal::branch::BranchState::has_logits = false

True once logits_snapshot holds a live distribution: capture_logits(), set_logits(), or a fork that cloned it.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 346 of file branch.hpp.

◆ img_slack_own

uint32_t lloyal::branch::BranchState::img_slack_own = 0

Embedding-row slack: cells beyond position-delta from decode_embd() (an M-RoPE image adds nx*ny cells but advances position by max(nx,ny)).

own = slack accrued on THIS branch since its fork (release() adds it to the position-delta when subtracting unique cells); total = inherited + own for the whole prefix (retainOnly()'s winner absorbs the prefix via fork_head=0 and needs it). fork() copies total, zeroes own.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 323 of file branch.hpp.

◆ img_slack_total

uint32_t lloyal::branch::BranchState::img_slack_total = 0

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 324 of file branch.hpp.

◆ in_use

bool lloyal::branch::BranchState::in_use = false

True when slot is allocated to an active branch.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 378 of file branch.hpp.

◆ last_candidates

std::vector<llama_token_data> lloyal::branch::BranchState::last_candidates

Filtered candidates from last sample()

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 343 of file branch.hpp.

◆ last_token

llama_token lloyal::branch::BranchState::last_token = -1

Last token returned by sample()

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 342 of file branch.hpp.

◆ logit_bias

std::vector<llama_logit_bias> lloyal::branch::BranchState::logit_bias

Static token biases, cloned on fork.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 337 of file branch.hpp.

◆ logits_snapshot

std::vector<float> lloyal::branch::BranchState::logits_snapshot

Captured logit distribution (n_vocab floats)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 345 of file branch.hpp.

◆ metrics

MetricsHandle lloyal::branch::BranchState::metrics = 0

Handle into BranchStore's metrics registry.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 340 of file branch.hpp.

◆ model

const llama_model* lloyal::branch::BranchState::model = nullptr

Llama model (not owned, must outlive branch)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 295 of file branch.hpp.

◆ n_batch

int lloyal::branch::BranchState::n_batch = DEFAULT_N_BATCH

Batch size for decode operations.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 374 of file branch.hpp.

◆ n_vocab

int lloyal::branch::BranchState::n_vocab = 0

Vocabulary size (cached for buffer pre-allocation)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 375 of file branch.hpp.

◆ parent

BranchHandle lloyal::branch::BranchState::parent = INVALID_HANDLE

Parent branch (INVALID_HANDLE if root)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 381 of file branch.hpp.

◆ poisoned

bool lloyal::branch::BranchState::poisoned = false

A decode failed and left the KV ahead of these books (see decode::leaves_branch_intact).

Prune and replay; a fork shares the damage and inherits the mark.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 310 of file branch.hpp.

◆ position

llama_pos lloyal::branch::BranchState::position = 0

Current decode position in the sequence.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 298 of file branch.hpp.

◆ revision

uint64_t lloyal::branch::BranchState::revision = 0

Counts every change to the state fast_forward() moves: decodes, logits, sampling, grammar and bias.

Compared for equality only, so a fork can tell whether its parent is still the branch it was forked from.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 304 of file branch.hpp.

◆ sampler_chain

SamplerChainHandle lloyal::branch::BranchState::sampler_chain = 0

Handle into BranchStore's sampler registry.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 326 of file branch.hpp.

◆ seq_id

llama_seq_id lloyal::branch::BranchState::seq_id = NO_LEASE

KV cache sequence identifier (NO_LEASE when inactive)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 297 of file branch.hpp.

◆ steer_fn

std::function<void(llama_token_data_array&)> lloyal::branch::BranchState::steer_fn

Dynamic logit callback, NOT cloned on fork.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 338 of file branch.hpp.

The documentation for this struct was generated from the following file:

include/lloyal/branch.hpp

Public Member Functions

Public Attributes