liblloyal - Composable Primitives for llama.cpp Inference

Introduction

liblloyal is a C++20 header-only library providing composable building blocks for llama.cpp inference. It offers clean abstractions over llama.cpp primitives with support for tokenization, sampling, embeddings, KV cache management, and advanced patterns like multi-sequence operations and handle-based APIs.

Core Features

• Tokenization - Two-pass safe buffer sizing, special token handling
• Decoding - Batch orchestration, sequence-aware operations
• KV Cache - Sequence operations, state snapshots, long-context patterns
• Sampling - Grammar-constrained, persistent chains, 52 parameters
• Metrics - Dual-level entropy/surprisal, rolling perplexity, cloneable state
• Embeddings - Pooled extraction, L2 normalization, similarity
• Chat Templates - Jinja2 formatting with fallbacks

Advanced Patterns

Handle-Based APIs

Create reusable sampler chains and grammar handles for efficient token generation:

auto chain = lloyal::sampler::create_chain(model, params);
lloyal::sampler::apply(chain, ctx, vocab);  // Reuse across tokens

Shared Model Weights

Multiple contexts can share the same loaded model via ModelRegistry:

auto model1 = lloyal::ModelRegistry::acquire(path, params);  // Loads model
auto model2 = lloyal::ModelRegistry::acquire(path, params);  // Cache hit
// model1 and model2 share weights, independent KV caches

Multi-Sequence Operations

All primitives support sequence IDs for parallel execution paths:

lloyal::kv::seq_cp(ctx, 0, 1);  // Branch sequence 0 to sequence 1
lloyal::kv::seq_cp(ctx, 0, 2);  // Branch sequence 0 to sequence 2
// Each sequence maintains independent state

Quick Start

#include <lloyal/model_registry.hpp>
#include <lloyal/tokenizer.hpp>
#include <lloyal/decoder.hpp>
#include <lloyal/sampler.hpp>
 
// Load model (shared weights)
auto model = lloyal::ModelRegistry::acquire("model.gguf", params);
llama_context* ctx = llama_init_from_model(model.get(), ctx_params);
 
// Tokenize and decode
auto tokens = lloyal::tokenizer::tokenize(model.get(), "Hello, world!");
lloyal::decoder::decode_tokens(ctx, tokens, 0, n_batch);
 
// Sample next token
auto token = lloyal::sampler::sample_with_params(ctx, vocab, params);

Architecture

• Header-only - All implementations inline in include/lloyal/*.hpp
• Composable primitives - Building blocks combine into diverse patterns
• Handle-based APIs - Persistent samplers, grammar chains for efficiency
• Shared model weights - Thread-safe registry enables multi-context with single model load
• Multi-sequence support - All primitives sequence-aware (default seq=0)
• llama.cpp binding - Compile-time dependency, validated by build system
• Zero runtime dependencies - Only requires C++20 standard library

Key Namespaces

• lloyal::tokenizer - Tokenization and detokenization
• lloyal::decoder - Batch and single-token decoding
• lloyal::sampler - Token sampling with configurable parameters
• lloyal::kv - KV cache sequence operations
• lloyal::metrics - Entropy, surprisal, and perplexity tracking
• lloyal::embedding - Embedding extraction and similarity
• lloyal::grammar - Grammar-constrained generation
• lloyal::chat_template - Chat formatting with Jinja2 templates
• lloyal::ModelRegistry - Model weight sharing and caching

Documentation

• Usage Guide: See docs/guide.md for comprehensive patterns, examples, and best practices
• API Reference: Navigate using the tabs above (Namespaces, Classes, Files)
• Examples: Check the Examples tab for usage patterns
• Headers: All APIs are fully documented inline in include/lloyal/*.hpp

Installation

Add as git submodule:

git submodule add -b v0.1.0 https://github.com/lloyal-ai/liblloyal.git

CMake integration:

add_subdirectory(liblloyal)
target_link_libraries(your_target PRIVATE lloyal llama)

License

Apache 2.0 - See LICENSE file for details