One run of input in a heterogeneous prefill. More...

#include <lloyal/decode.hpp>

Public Types
enum class	Kind { Text , Embd }

Public Attributes
Kind	kind = Kind::Text

std::span< const llama_token >	tokens
	Text: ready token ids (never re-tokenized by the store)

const float *	rows = nullptr
	Embd: n_rows x n_embd_inp floats.

int32_t	n_rows = 0

int32_t	n_embd_inp = 0

llama_pos	n_pos = 0
	Position advance this segment costs (may be < n_rows under M-RoPE)

int32_t	n_pos_per_embd = 1
	1 (plain positions) or 4 (M-RoPE)

bool	non_causal = false
	Bracket this segment's decode in non-causal attention.

Detailed Description

One run of input in a heterogeneous prefill.

A prefill is a sequence of segments, placed in order by segments(). How they are dispatched is the context's to decide (see StreamReading): each its own dispatch, TEXT through many() and EMBD through embd(), or the whole stream as entries of one mixed batch.

The store is deliberately blind to what produced an EMBD segment: rows are rows, whether they came from a vision projector, an audio encoder, or a cached embedding.

Definition at line 646 of file decode.hpp.

Member Enumeration Documentation

◆ Kind

enum class lloyal::decode::Segment::Kind

strong

Enumerator
Text
Embd

Definition at line 647 of file decode.hpp.

Member Data Documentation

◆ kind

Kind lloyal::decode::Segment::kind = Kind::Text

Definition at line 648 of file decode.hpp.

◆ n_embd_inp

int32_t lloyal::decode::Segment::n_embd_inp = 0

Definition at line 657 of file decode.hpp.

◆ n_pos

llama_pos lloyal::decode::Segment::n_pos = 0

Position advance this segment costs (may be < n_rows under M-RoPE)

Definition at line 659 of file decode.hpp.

◆ n_pos_per_embd

int32_t lloyal::decode::Segment::n_pos_per_embd = 1

1 (plain positions) or 4 (M-RoPE)

Definition at line 661 of file decode.hpp.

◆ n_rows

int32_t lloyal::decode::Segment::n_rows = 0

Definition at line 656 of file decode.hpp.

◆ non_causal

bool lloyal::decode::Segment::non_causal = false

Bracket this segment's decode in non-causal attention.

Definition at line 663 of file decode.hpp.

◆ rows

const float* lloyal::decode::Segment::rows = nullptr

Embd: n_rows x n_embd_inp floats.

Valid only until the source's next at() call — see SegmentSource's in-order contract.

Definition at line 655 of file decode.hpp.

◆ tokens

std::span<const llama_token> lloyal::decode::Segment::tokens

Text: ready token ids (never re-tokenized by the store)

Definition at line 651 of file decode.hpp.

The documentation for this struct was generated from the following file:

include/lloyal/decode.hpp

Public Types

Public Attributes