model_registry.hpp File Reference

Thread-Safe Model Cache. More...

#include "common.hpp"
#include <functional>
#include <llama/llama.h>
#include <memory>
#include <mutex>
#include <string>
#include <unordered_map>

Go to the source code of this file.

Classes
struct	lloyal::ModelKey
	Model cache key combining file path and GPU configuration. More...
struct	lloyal::ModelKeyHash
	Hash functor for ModelKey. More...
class	lloyal::ModelRegistry
	Thread-safe registry for sharing llama_model instances. More...

Namespaces
namespace	lloyal
	JSON Schema to Grammar Converter (Header-Only)
namespace	lloyal::detail

Detailed Description

Thread-Safe Model Cache.

Provides weak-pointer cache to avoid reloading same model multiple times. Uses inline static members (C++17) for header-only implementation with static state.

Cache key: (canonPath, n_gpu_layers, use_mmap) Cache value: weak_ptr to llama_model (auto-cleanup when last context releases)

Thread-safe via std::mutex for all cache operations.

Definition in file model_registry.hpp.