Namespaces
namespace	detail

namespace	shape_guard

Classes
struct	DecodeError

struct	EachItem
	Input item for decode::each — one token for one sequence. More...

struct	EmbdItem
	Input item for decode::embd — embedding rows for one sequence. More...

struct	LandedSegment
	A segment of a stream once it is in the KV: what moved, for the caller's books. More...

struct	PackedChunk
	A chunk of item indices produced by bin_pack() More...

struct	ScatterItem
	Input item for decode::scatter — multiple tokens for one sequence. More...

struct	Scratch
	Reusable scratch buffers for multi-sequence batch construction. More...

struct	Segment
	One run of input in a heterogeneous prefill. More...

struct	SegmentProgress
	How far one input of a stream has landed: a reading of progress. More...

struct	SegmentSource
	Supplies the segment sequence for one branch's prefill. More...

struct	StreamReading
	How a context reads what is placed into it: the one read of its traits that placement decides from. More...

class	TokenSource
	A token span as a stream of one TEXT segment, so text is placed by the rail like any other input. More...

Functions
bool	leaves_branch_intact (int32_t rc, bool partial)
	llama_decode failure carrying the raw return code

void	require_fits_batch (llama_context ctx, int32_t n_tokens, const char who)
	The context's batch is a hard limit: the pinned llama.cpp asserts `n_tokens <= n_batch` inside llama_decode — an abort, not an rc.

int	many (llama_context ctx, const llama_token tokens, int32_t n_tokens, int32_t n_past, int32_t n_batch, llama_seq_id seq_id=0, int32_t *n_committed=nullptr)
	Decode multiple tokens into the KV cache with auto-chunking.

int	many (llama_context *ctx, const std::vector< llama_token > &tokens, int32_t n_past, int32_t n_batch, llama_seq_id seq_id=0)
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

int	one (llama_context *ctx, llama_token tok, llama_pos pos, llama_seq_id seq_id=0, bool want_logits=true)
	Decode a single token into the KV cache.

void	require_row_width (llama_context ctx, int32_t n_embd_inp, const char who)
	Rows must be exactly the model's input width: a wrong-but-positive width starts later rows mid-row and reads past the caller's allocation, with nothing to signal it.

int	each (llama_context ctx, const EachItem items, int32_t n, Scratch &scratch)
	Decode one token per sequence in a single llama_decode() call.

int	each (llama_context *ctx, const std::vector< EachItem > &items, Scratch &scratch)
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

int	scatter (llama_context ctx, const ScatterItem items, int32_t n, Scratch &scratch)
	Decode multiple tokens per sequence in a single llama_decode() call.

int	scatter (llama_context *ctx, const std::vector< ScatterItem > &items, Scratch &scratch)
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

int	embd (llama_context ctx, const EmbdItem &item, int32_t n_batch, Scratch &scratch, int32_t n_committed=nullptr)
	Decode pre-computed embedding rows into one sequence's KV cache.

StreamReading	reading_of (llama_context *ctx)

template<typename OnLanded >
void	segments (llama_context *ctx, llama_seq_id seq, llama_pos base, SegmentSource &source, int32_t rows_n_batch, Scratch &scratch, std::vector< llama_pos > &positions, OnLanded &&landed)
	Place a source's segments into one sequence, in order.

std::vector< PackedChunk >	bin_pack (const std::span< const llama_token > *items, int32_t n, int32_t n_batch)
	Greedy first-fit bin-packing of token spans into n_batch-sized chunks.

Function Documentation

◆ bin_pack()

std::vector< PackedChunk > lloyal::decode::bin_pack	(	const std::span< const llama_token > *	items,
		int32_t	n,
		int32_t	n_batch
	)

inline

Greedy first-fit bin-packing of token spans into n_batch-sized chunks.

Pure packing algorithm — no decoding, no logit capture, no context. Callers use the returned chunks to drive their own dispatch logic (decode::scatter for normal chunks, decode::many for oversized).

Empty spans (size 0) are skipped. Items exceeding n_batch get a solo oversized chunk.

Parameters

items	Array of token spans (only .size() is inspected)
n	Number of items
n_batch	Maximum total tokens per normal chunk

Returns: Vector of PackedChunks with indices into the input array

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp, and /home/runner/work/liblloyal/liblloyal/include/lloyal/logits.hpp.

Definition at line 1215 of file decode.hpp.

◆ each() [1/2]

int lloyal::decode::each	(	llama_context *	ctx,
		const EachItem *	items,
		int32_t	n,
		Scratch &	scratch
	)

inline

Decode one token per sequence in a single llama_decode() call.

"each" = each sequence gets one token. Packs N tokens (each targeting a different seq_id) into one llama_batch. Amortizes GPU dispatch overhead across N sequences.

Parameters

ctx	Llama context (must not be null)
items	Array of (token, pos, seq_id, output_logits) tuples
n	Number of items
scratch	Reusable scratch buffers

Returns: 0 on success, non-zero on failure

Exceptions

std::runtime_error if ctx is NULL

See also: one() for single-sequence single-token decode; scatter() for multi-token-per-sequence variant

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 489 of file decode.hpp.

◆ each() [2/2]

int lloyal::decode::each	(	llama_context *	ctx,
		const std::vector< EachItem > &	items,
		Scratch &	scratch
	)

inline

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

Definition at line 521 of file decode.hpp.

◆ embd()

int lloyal::decode::embd	(	llama_context *	ctx,
		const EmbdItem &	item,
		int32_t	n_batch,
		Scratch &	scratch,
		int32_t *	n_committed = `nullptr`
	)

inline

Decode pre-computed embedding rows into one sequence's KV cache.

The embedding rail beside the token rail: rows enter via batch.embd with batch.token = nullptr, zero-copy from the caller's buffer. A legacy llama_batch carries tokens or rows, not both, so this is always its own llama_decode, distinct from scatter(). A stream that a context must read whole goes in one mixed batch through segments() instead.

Auto-chunks by n_batch like many() — and this is the MAIN path, not an edge: image_min_tokens metadata commonly puts an image above the default batch size. Each sub-chunk re-packs its positions section-major into the scratch buffers, so no view/slice buffer is needed.

Parameters

ctx	Llama context (must not be null)
item	Rows, positions, sequence and flags
n_batch	Max rows per llama_decode (sub-chunk bound)
scratch	Reusable scratch buffers (shared with each()/scatter())
n_committed	Optional out: rows landed before return — `n_rows` on success, fewer when a later chunk failed (see DecodeError::partial)

Returns: 0 on success, non-zero llama_decode rc on failure

Exceptions

std::runtime_error on null ctx or malformed item

See also: scatter() for the multi-sequence token-rail primitive; BranchStore::decode_embd for the branch-level wrapper (bookkeeping)

Restores causal attention on every exit path, including a throw.

Causal mode is CONTEXT-WIDE, not per-batch: leaving it off would make every subsequent TEXT decode on this context non-causal, so the damage outlives this call. scratch.resize() can throw between the disable below and the end of the loop, which is why restoring at the return points is not enough.

False for causal items — the guard is then inert

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 794 of file decode.hpp.

◆ leaves_branch_intact()

bool lloyal::decode::leaves_branch_intact	(	int32_t	rc,
		bool	partial
	)

inline

llama_decode failure carrying the raw return code

Two facts travel as DATA, because the caller acts on them and can infer neither from the message:

rc classifies the FAILING CALL (llama.h): 1 = no KV slot and -1 = invalid batch both leave the memory as it was before that call; 2 = aborted and < -1 = fatal leave partial ubatches behind.
partial says whether EARLIER calls of the same operation landed. Every chunked path (many, embd, BranchStore::decode_scatter, decode_segments) may have committed chunks before the one that failed, and the branch's books never move on failure.

The rule, for an operation on one branch: the branch is INTACT iff the failing call restored state (rc is 1 or -1) and nothing before it landed (!partial). Intact with rc 1 is a capacity wait — retry when the KV has room; intact with rc -1 is the caller's input — do not resend the same batch. Anything else ⇒ prune the branch and replay onto a fresh one.

BranchStore::decode_scatter spans branches: its partial says an earlier chunk landed, for some branch, whose books moved with it. Whether a given branch is intact is its own poisoned flag (branch::is_poisoned()).

The binding catches this type in C++ and forwards both fields structurally; the exception itself never crosses N-API.

Visibility caveat: safe while liblloyal is header-only (thrower and catcher compile into one TU). If liblloyal ever becomes a separate shared library, typed catches can miss under -fvisibility=hidden — attach the rc some other way before making that move.

The message also carries rc=N (and partial when set) for humans reading logs; nothing parses it. The rule above as a predicate: whether a failed call left its branch intact.

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp.

Definition at line 152 of file decode.hpp.

◆ many() [1/2]

int lloyal::decode::many	(	llama_context *	ctx,
		const llama_token *	tokens,
		int32_t	n_tokens,
		int32_t	n_past,
		int32_t	n_batch,
		llama_seq_id	seq_id = `0`,
		int32_t *	n_committed = `nullptr`
	)

inline

Decode multiple tokens into the KV cache with auto-chunking.

Orchestration logic:

Uses a thread_local batch (heap-allocated once per thread, grows on demand)
Chunks tokens into n_batch-sized pieces
For each chunk: clear batch, add tokens, call llama_decode

Sequence ID Parameter

The seq_id parameter specifies which KV cache sequence to update. Default is 0 (single-sequence mode, backward compatible).

Use different seq_ids for:

Parallel generations (multiple steppers, each with own seq_id)
Branching/tree search (System 2)
Shared prefix optimization (decode prefix to seq_id=0, copy to others)

IMPORTANT: n_seq_max Clarification

There are TWO different n_seq_max parameters - don't confuse them:

llama_batch_init(n_tokens, embd, n_seq_max)
- Controls how many sequences A SINGLE TOKEN can belong to
- Keep at 1 for normal decode (one token → one sequence)
- Only increase for beam search where one token updates multiple branches
llama_context_params.n_seq_max
- Controls max TOTAL sequences (distinct KV cache states)
- Increase for parallel generations or tree search

Example: 4 parallel steppers, each decoding its own branch

Context n_seq_max: 4 (four distinct sequences)
Batch n_seq_max: 1 (each token belongs to one sequence)
Call: decode::many(ctx, tokens, n, pos, batch, seq_id=stepper_id)

Parameters

ctx	Llama context (must be initialized)
tokens	Token array to decode
n_tokens	Number of tokens in array
n_past	Position to start decoding from (KV cache position)
n_batch	Batch size for chunking
seq_id	Sequence ID to update in KV cache (default: 0)
n_committed	Optional out: tokens landed before return — `n_tokens` on success, fewer when a later chunk failed (see DecodeError::partial)

Returns: 0 on success, non-zero on decode failure

Exceptions

std::runtime_error if ctx is NULL or tokens are invalid (validation errors)

CRITICAL: Call kv::remove_range() BEFORE this function, never after.

See also: one() for single-token decode (autoregressive generation); scatter() for multi-token decode across multiple sequences

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp, /home/runner/work/liblloyal/liblloyal/include/lloyal/kv.hpp, and /home/runner/work/liblloyal/liblloyal/include/lloyal/logits.hpp.

Definition at line 240 of file decode.hpp.

◆ many() [2/2]

int lloyal::decode::many	(	llama_context *	ctx,
		const std::vector< llama_token > &	tokens,
		int32_t	n_past,
		int32_t	n_batch,
		llama_seq_id	seq_id = `0`
	)

inline

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

Definition at line 322 of file decode.hpp.

◆ one()

int lloyal::decode::one	(	llama_context *	ctx,
		llama_token	tok,
		llama_pos	pos,
		llama_seq_id	seq_id = `0`,
		bool	want_logits = `true`
	)

inline

Decode a single token into the KV cache.

Fast path for autoregressive generation. Uses a thread_local batch (one-time init per thread) so repeated calls avoid allocation entirely.

Typical usage in a generation loop:

llama_token tok = sampler::sample(ctx, vocab);

if (decode::one(ctx, tok, n_past++) != 0) { handle error }

lloyal::decode::one

int one(llama_context *ctx, llama_token tok, llama_pos pos, llama_seq_id seq_id=0, bool want_logits=true)

Decode a single token into the KV cache.

Definition decode.hpp:354

Parameters

ctx	Llama context (must not be null)
tok	Token to decode
pos	KV cache position for this token
seq_id	Sequence ID to update (default: 0)
want_logits	Whether to compute logits after this token (default: true). Set to false when prefilling tokens that don't need sampling.

Returns: 0 on success, non-zero on decode failure

Exceptions

std::runtime_error if ctx is NULL

See also: many() for batched multi-token decode with auto-chunking; each() for single-token decode across multiple sequences

Definition at line 354 of file decode.hpp.

◆ reading_of()

StreamReading lloyal::decode::reading_of ( llama_context * ctx )

inline

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/embedding.hpp.

Definition at line 948 of file decode.hpp.

◆ require_fits_batch()

void lloyal::decode::require_fits_batch	(	llama_context *	ctx,
		int32_t	n_tokens,
		const char *	who
	)

inline

The context's batch is a hard limit: the pinned llama.cpp asserts n_tokens <= n_batch inside llama_decode — an abort, not an rc.

Chunk sizes come from the caller (a branch's n_batch, a free function's argument), so every primitive below checks the batch it is about to build against the context first. An oversize chunk is a clear error; it is never a dead process.

Definition at line 180 of file decode.hpp.

◆ require_row_width()

void lloyal::decode::require_row_width	(	llama_context *	ctx,
		int32_t	n_embd_inp,
		const char *	who
	)

inline

Rows must be exactly the model's input width: a wrong-but-positive width starts later rows mid-row and reads past the caller's allocation, with nothing to signal it.

Definition at line 462 of file decode.hpp.

◆ scatter() [1/2]

int lloyal::decode::scatter	(	llama_context *	ctx,
		const ScatterItem *	items,
		int32_t	n,
		Scratch &	scratch
	)

inline

Decode multiple tokens per sequence in a single llama_decode() call.

Single-batch primitive: packs token runs from multiple sequences into one llama_batch. Does NOT auto-chunk — total tokens must fit in n_batch.

Parameters

ctx	Llama context (must not be null)
items	Array of (tokens_span, start_pos, seq_id) tuples
n	Number of items
scratch	Reusable scratch buffers

Returns: 0 on success, non-zero on failure

Exceptions

std::runtime_error if ctx is NULL or items are invalid

Note: Does NOT auto-chunk. A total beyond the context's n_batch is refused (require_fits_batch), never dispatched.

See also: many() for single-sequence multi-token decode with auto-chunking; each() for single-token-per-sequence variant; BranchStore::decode_scatter for auto-chunking branch-level variant

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp, and /home/runner/work/liblloyal/liblloyal/include/lloyal/logits.hpp.

Definition at line 547 of file decode.hpp.

◆ scatter() [2/2]

int lloyal::decode::scatter	(	llama_context *	ctx,
		const std::vector< ScatterItem > &	items,
		Scratch &	scratch
	)

inline

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

Definition at line 596 of file decode.hpp.

◆ segments()

template<typename OnLanded >

void lloyal::decode::segments	(	llama_context *	ctx,
		llama_seq_id	seq,
		llama_pos	base,
		SegmentSource &	source,
		int32_t	rows_n_batch,
		Scratch &	scratch,
		std::vector< llama_pos > &	positions,
		OnLanded &&	landed
	)

inline

Place a source's segments into one sequence, in order.

The one owner of placement for a heterogeneous stream. The context decides how, never the caller:

One dispatch per segment for a context that keeps what it was given and attends causally, a reasoning model's: TEXT through many() at the context's batch, EMBD through embd() at rows_n_batch (a branch's own batch), so a prefill keeps exactly the dispatch shape it has always had.
The whole stream in one dispatch for a context that reads it only whole (see StreamReading), an embedding model's: tokens and rows as entries of one llama_batch_ext, one llama_process. It is refused before anything runs when it would not fit one ubatch, because llama.cpp aborts on an oversized one rather than returning an error.

The source is untrusted input: every segment is checked before it sizes a buffer or reaches positions(). A segment is placed at the position the previous one ended, and landed is told once it is in the KV, so a caller moves its books per segment and a later failure finds them true to what landed.

Moves nothing of the caller's itself. A failed dispatch throws DecodeError, partial when an earlier segment landed or a chunk of this one did: the stream is one operation to its caller. A throw from the source passes through unchanged.

Parameters

ctx	Llama context (must not be null)
seq	Sequence the stream lands in
base	Position of the stream's first entry
source	Yields the segments; owns production, not placement
rows_n_batch	Most rows per dispatch for an EMBD segment placed alone
scratch	Reusable batch buffers (shared with each()/scatter()/embd())
positions	Reusable buffer for an EMBD segment's positions
landed	Called with a LandedSegment after each segment lands

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/branch.hpp, and /home/runner/work/liblloyal/liblloyal/include/lloyal/embedding.hpp.

Definition at line 1170 of file decode.hpp.

Namespaces

Classes

Functions

Function Documentation

◆ bin_pack()

◆ each() [1/2]

◆ each() [2/2]

◆ embd()

◆ leaves_branch_intact()

◆ many() [1/2]

Sequence ID Parameter

IMPORTANT: n_seq_max Clarification

◆ many() [2/2]

◆ one()

◆ reading_of()

◆ require_fits_batch()

◆ require_row_width()

◆ scatter() [1/2]

◆ scatter() [2/2]

◆ segments()