lloyal::decode::detail Namespace Reference

Functions
void	batch_clear (llama_batch &batch)
	The two helpers llama.cpp's common library used to provide for a llama_batch_init()-allocated batch: reset the count, append one token targeting one sequence.
void	batch_add (llama_batch &batch, llama_token id, llama_pos pos, llama_seq_id seq_id, bool logits)
void	require_segment (const Segment &seg, size_t i)
	A segment the source yielded, checked before anything is sized from it or handed back to the source: the source is a public extension point, so its segments are untrusted input.
template<typename OnLanded >
void	segments_each (llama_context *ctx, llama_seq_id seq, llama_pos base, SegmentSource &source, int32_t rows_n_batch, Scratch &scratch, std::vector< llama_pos > &positions, OnLanded &&landed)
	Each segment its own dispatch: the shape of a prefill into a sequence that keeps what it was given.
template<typename OnLanded >
void	segments_whole (llama_context *ctx, const StreamReading &reading, llama_seq_id seq, llama_pos base, SegmentSource &source, std::vector< llama_pos > &positions, OnLanded &&landed)
	The whole stream in one dispatch: one llama_batch_ext, tokens and rows as entries in order, one llama_process.

Function Documentation

batch_add()

void lloyal::decode::detail::batch_add ( llama_batch &  batch, llama_token  id, llama_pos  pos, llama_seq_id  seq_id, bool  logits  )

inline

Definition at line 104 of file decode.hpp.

batch_clear()

void lloyal::decode::detail::batch_clear ( llama_batch &  batch )

inline

The two helpers llama.cpp's common library used to provide for a llama_batch_init()-allocated batch: reset the count, append one token targeting one sequence.

Definition at line 103 of file decode.hpp.

require_segment()

void lloyal::decode::detail::require_segment ( const Segment &  seg, size_t  i  )

inline

A segment the source yielded, checked before anything is sized from it or handed back to the source: the source is a public extension point, so its segments are untrusted input.

Definition at line 958 of file decode.hpp.

segments_each()

template<typename OnLanded >

void lloyal::decode::detail::segments_each	(	llama_context *	ctx,
		llama_seq_id	seq,
		llama_pos	base,
		SegmentSource &	source,
		int32_t	rows_n_batch,
		Scratch &	scratch,
		std::vector< llama_pos > &	positions,
		OnLanded &&	landed
	)

Each segment its own dispatch: the shape of a prefill into a sequence that keeps what it was given.

Definition at line 990 of file decode.hpp.

segments_whole()

template<typename OnLanded >

void lloyal::decode::detail::segments_whole	(	llama_context *	ctx,
		const StreamReading &	reading,
		llama_seq_id	seq,
		llama_pos	base,
		SegmentSource &	source,
		std::vector< llama_pos > &	positions,
		OnLanded &&	landed
	)

The whole stream in one dispatch: one llama_batch_ext, tokens and rows as entries in order, one llama_process.

The batch keeps its own copy of every row, so a source may reuse its row buffer while the stream is gathered.

Definition at line 1046 of file decode.hpp.