Turns a marker'd prompt + admitted media into a decode segment stream. More...
#include <lloyal/mtmd.hpp>
Public Member Functions | |
| MtmdSource (mtmd_context *ctx, const std::string &prompt, const std::vector< std::vector< uint8_t > > &images, std::span< const llama_token > sep, int32_t n_embd_inp, bool add_special=false) | |
| Existing image entry point; audio requires typed inputs and explicit limits. | |
| MtmdSource (mtmd_context *ctx, const std::string &prompt, std::span< const MediaInput > inputs, std::span< const llama_token > sep, int32_t n_embd_inp, AudioLimits limits, bool add_special=false) | |
| Inputs follow prompt marker order. | |
| size_t | size () override |
| Number of segments in this prefill. | |
| size_t | cells () const override |
| KV cells this prefill will consume — see decode::SegmentSource::cells. | |
| int32_t | input_of (size_t i) const override |
The input segment i belongs to (an image, a video, a recording), or -1 for prompt text that belongs to none. | |
| size_t | total_of (int32_t input) const override |
KV cells input input takes, summed over its segments: what progress on it counts towards. | |
| decode::Segment | at (size_t i) override |
Segment i. Invalidates any previously returned segment. | |
| void | positions (size_t i, llama_pos base, llama_pos *out) override |
| Fill positions for an EMBD segment, given the absolute base the store chose. | |
 Public Member Functions inherited from lloyal::decode::SegmentSource | |
| virtual | ~SegmentSource ()=default |
Detailed Description
Turns a marker'd prompt + admitted media into a decode segment stream.
The prompt carries one media marker (mtmd_default_marker(), "<__media__>") per input; mtmd splits it into interleaved text and media chunks. Text chunks come back as ready token ids — never re-tokenized.
Framing is the caller's, as on the text path. add_special says whether the model's special tokens (a BOS) frame the prompt, as tokenizer::tokenize(vocab, text, add_special, true) does for text: false for a chat prompt, which already carries its template's specials (a BOS there sits mid-conversation and desynchronizes the KV from what the model was trained to see); true for a standalone embedding input, as upstream tokenizes one. parse_special = true always: template specials are real tokens.
Lifetime. Owns its bitmaps and chunk list. The input bytes are decoded into those bitmaps during construction and never retained, so the caller may release them as soon as the constructor returns. ctx and sep ARE retained and must outlive this object. Satisfies SegmentSource's in-order contract: an Embd segment's rows point into mtmd's context-owned encode buffer, which the next at() overwrites.
Constructor & Destructor Documentation
◆ MtmdSource() [1/2]
|
inline |
◆ MtmdSource() [2/2]
|
inline |
Inputs follow prompt marker order.
Audio is integer PCM WAV, with optional INFO/JUNK metadata. Limits apply before decoding or preprocessing; the caller supplies policy. ctx and sep must outlive this source.
add_special adds the model's special tokens (a BOS) around the prompt, as mtmd_tokenize does for an embedding input. A chat prompt already carries its template's, so the default leaves them out.
Member Function Documentation
◆ at()
|
inlineoverridevirtual |
Segment i. Invalidates any previously returned segment.
Implements lloyal::decode::SegmentSource.
◆ cells()
|
inlineoverridevirtual |
KV cells this prefill will consume — see decode::SegmentSource::cells.
Knowable here, and never an estimate: counted during construction, after mtmd_tokenize and before any encoder execution. Media row counts are fixed at tokenize time, which is what the placeholder-bitmap counting flow in mtmd.h relies on.
Implements lloyal::decode::SegmentSource.
◆ input_of()
|
inlineoverridevirtual |
The input segment i belongs to (an image, a video, a recording), or -1 for prompt text that belongs to none.
A source without inputs has none.
Reimplemented from lloyal::decode::SegmentSource.
◆ positions()
|
inlineoverridevirtual |
Fill positions for an EMBD segment, given the absolute base the store chose.
out has n_rows * n_pos_per_embd entries, section-major: [s0...][s1...].... Never called for a TEXT segment.
The base is passed rather than exposed, because how it applies is model-specific — M-RoPE freezes one section and leaves another at zero, so a caller-side rebase would need the producer's model taxonomy.
Implements lloyal::decode::SegmentSource.
◆ size()
|
inlineoverridevirtual |
Number of segments in this prefill.
Implements lloyal::decode::SegmentSource.
◆ total_of()
|
inlineoverridevirtual |
KV cells input input takes, summed over its segments: what progress on it counts towards.
Known before anything decodes, like cells().
Reimplemented from lloyal::decode::SegmentSource.
The documentation for this class was generated from the following file:
- include/lloyal/mtmd.hpp
