Supplies the segment sequence for one branch's prefill. More...
#include <lloyal/decode.hpp>
Public Member Functions | |
| virtual | ~SegmentSource ()=default |
| virtual size_t | size ()=0 |
| Number of segments in this prefill. | |
| virtual size_t | cells () const =0 |
| KV cells this whole sequence will consume, known BEFORE anything decodes. | |
| virtual Segment | at (size_t i)=0 |
Segment i. Invalidates any previously returned segment. | |
| virtual void | positions (size_t i, llama_pos base, llama_pos *out)=0 |
| Fill positions for an EMBD segment, given the absolute base the store chose. | |
| virtual int32_t | input_of (size_t) const |
The input segment i belongs to (an image, a video, a recording), or -1 for prompt text that belongs to none. | |
| virtual size_t | total_of (int32_t) const |
KV cells input input takes, summed over its segments: what progress on it counts towards. | |
Detailed Description
Supplies the segment sequence for one branch's prefill.
The caller owns placement (which rail, at what position, which segment captures logits); the source owns production (decoding bytes, encoding rows, and the model-specific position geometry). Nothing about the producing format crosses this interface — an implementation may pull in llama.cpp's mtmd, a platform encoder, or nothing at all.
In-order contract. Segments are requested strictly in order, and each is done with before the next is requested: dispatched, or copied into a batch that dispatches once the stream is whole. An implementation may therefore hand out a pointer into a buffer it reuses (mtmd's encode output is one), invalidating segment i-1 when at(i) is called. Callers must not hold a segment's rows across an at() call, prefetch, or revisit.
Definition at line 682 of file decode.hpp.
Constructor & Destructor Documentation
◆ ~SegmentSource()
|
virtualdefault |
Member Function Documentation
◆ at()
|
pure virtual |
Segment i. Invalidates any previously returned segment.
Implemented in lloyal::MtmdSource, and lloyal::decode::TokenSource.
◆ cells()
|
pure virtual |
KV cells this whole sequence will consume, known BEFORE anything decodes.
Exists so a caller can decide ADMISSION before touching a branch. decode_segments is not atomic: once the first segment dispatches the branch is mutated, so discovering mid-walk that the sequence does not fit costs the branch. Refusing up front costs nothing.
Text can already be measured by tokenizing it, but a source that encodes — a vision projector, an audio tower — knows a row count its caller cannot derive from the bytes it holds. Without this, such input is the one thing that bypasses a context-pressure gate.
Cells, not positions: a KV budget is spent in cells, and under M-RoPE a segment costs far more cells than it advances position. The unit matches DecodeSegmentsResult::cells, so a caller can compare what it was quoted against what it was charged.
Must equal the sum over at(0 .. size()-1) of tokens.size() for TEXT and n_rows for EMBD. A source that cannot know the count before encoding must not estimate: this is a budget promise, and an under-quote is spent out of someone else's budget.
Implemented in lloyal::decode::TokenSource, and lloyal::MtmdSource.
◆ input_of()
|
inlinevirtual |
The input segment i belongs to (an image, a video, a recording), or -1 for prompt text that belongs to none.
A source without inputs has none.
Reimplemented in lloyal::MtmdSource.
Definition at line 729 of file decode.hpp.
◆ positions()
|
pure virtual |
Fill positions for an EMBD segment, given the absolute base the store chose.
out has n_rows * n_pos_per_embd entries, section-major: [s0...][s1...].... Never called for a TEXT segment.
The base is passed rather than exposed, because how it applies is model-specific — M-RoPE freezes one section and leaves another at zero, so a caller-side rebase would need the producer's model taxonomy.
Implemented in lloyal::MtmdSource, and lloyal::decode::TokenSource.
◆ size()
|
pure virtual |
Number of segments in this prefill.
Implemented in lloyal::decode::TokenSource, and lloyal::MtmdSource.
◆ total_of()
|
inlinevirtual |
KV cells input input takes, summed over its segments: what progress on it counts towards.
Known before anything decodes, like cells().
Reimplemented in lloyal::MtmdSource.
Definition at line 733 of file decode.hpp.
The documentation for this struct was generated from the following file:
- include/lloyal/decode.hpp
