mtmd.hpp File Reference

llama.cpp mtmd as a decode::SegmentSource (multimodal codec) More...

#include "decode.hpp"
#include <mtmd.h>
#include <mtmd-helper.h>
#include <algorithm>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <functional>
#include <limits>
#include <memory>
#include <numeric>
#include <optional>
#include <span>
#include <stdexcept>
#include <string>
#include <string_view>
#include <vector>

Go to the source code of this file.

Classes
struct	lloyal::VideoNote
	Text placed on a video's timeline, such as a transcript segment. More...
struct	lloyal::VideoFrames
	The sampled frames of one video, in order, each an encoded image with the time it was sampled at. More...
struct	lloyal::MediaInput
	A borrowed admitted representation. More...
struct	lloyal::AudioLimits
	Per-source audio budgets, across all inputs. More...
struct	lloyal::mtmd_detail::VideoLayout
	One video expanded into the parts mtmd tokenizes: decoded frames and text. More...
struct	lloyal::mtmd_detail::VideoLayout::Part
class	lloyal::MtmdSource
	Turns a marker'd prompt + admitted media into a decode segment stream. More...

Namespaces
namespace	lloyal
	Boundary Tracker Stub for OSS liblloyal.
namespace	lloyal::shape_guard
namespace	lloyal::mtmd_detail

Functions
void	lloyal::shape_guard::mtmd_input_text_has_four_members ()
bool	lloyal::mtmd_detail::tag (std::span< const uint8_t > bytes, size_t offset, const char *value)
uint16_t	lloyal::mtmd_detail::u16 (std::span< const uint8_t > bytes, size_t offset)
uint32_t	lloyal::mtmd_detail::u32 (std::span< const uint8_t > bytes, size_t offset)
bool	lloyal::mtmd_detail::encoded_audio (std::span< const uint8_t > bytes)
size_t	lloyal::mtmd_detail::wav_sample_bound (std::span< const uint8_t > bytes, uint32_t target_rate)
std::string	lloyal::mtmd_detail::timestamp_text (int64_t ms)
VideoLayout	lloyal::mtmd_detail::video_layout (std::vector<::mtmd::bitmap_ptr > frames, const VideoFrames &video)
std::vector< MediaInput >	lloyal::mtmd_detail::image_inputs (const std::vector< std::vector< uint8_t > > &images)

Detailed Description

llama.cpp mtmd as a decode::SegmentSource (multimodal codec)

Opt-in header. Including it takes a dependency on llama.cpp's tools/mtmd; liblloyal's core does not. branch.hpp and decode.hpp see only decode::SegmentSource and never an mtmd_* type — this file is the one place the two vocabularies meet.

Build requirement (the consumer's, not liblloyal's). liblloyal is header-only and does not link anything on your behalf — linking is the binding layer's job. A target that includes this header must itself link llama.cpp's mtmd target, which PUBLIC-propagates the tools/mtmd include path:

add_subdirectory(${LLAMA_CPP_DIR}/tools/mtmd mtmd EXCLUDE_FROM_ALL)
target_link_libraries(your_target PRIVATE liblloyal::liblloyal mtmd)

A consumer that never includes this header links neither, and carries no multimodal dependency at all.

The split it implements:

• Codec (here): media decode, tokenization, encoder execution, and the per-model position geometry. Everything format-specific.
• **Kernel (BranchStore::decode_segments):** which rail each segment takes, at what position, which one captures logits, and the cell/slack bookkeeping. Everything KV-specific.

A binding therefore constructs an MtmdSource and makes one kernel call. Every SessionContext implementation (N-API, Nitro/JSI, …) shares this file rather than reimplementing the walk — and a platform encoder (CoreML, say) can supply a different SegmentSource without touching the kernel.

Definition in file mtmd.hpp.