Functions
std::vector< llama_token >	tokenize (const llama_vocab *vocab, const std::string &text, bool add_special, bool parse_special)
	Tokenize text to token array.

std::string	detokenize (const llama_vocab *vocab, llama_token token, bool special)
	Detokenize SINGLE token to text (streaming use case)

std::string	detokenize_batch (const llama_vocab vocab, const llama_token tokens, int32_t n_tokens, bool remove_special, bool unparse_special)
	Detokenize TOKEN ARRAY to text (reconstruction use case)

const llama_vocab *	get_vocab (const llama_model *model)
	Get vocabulary from model.

bool	is_eog (const llama_vocab *vocab, llama_token token)
	Check if token is end-of-generation marker.

int32_t	vocab_size (const llama_vocab *vocab)
	Get vocabulary size (total number of tokens)

std::vector< llama_token >	tokenize (const llama_model *model, const std::string &text)
	Tokenize text to token array (model-accepting overload)

std::string	detokenize (const llama_model *model, llama_token token, bool special=true)
	Detokenize SINGLE token to text (model-accepting overload)

std::string	detokenize_batch (const llama_model *model, const std::vector< llama_token > &tokens, bool remove_special=false, bool unparse_special=true)
	Detokenize TOKEN VECTOR to text (convenience overload)

std::string	detokenize_batch (const llama_model model, const llama_token tokens, int32_t n_tokens, bool remove_special, bool unparse_special)
	Detokenize TOKEN ARRAY to text (model-accepting overload)

bool	is_eog (const llama_model *model, llama_token token)
	Check if token is end-of-generation marker (model-accepting overload)

int32_t	vocab_size (const llama_model *model)
	Get vocabulary size (model-accepting overload)

Function Documentation

◆ detokenize() [1/2]

std::string lloyal::tokenizer::detokenize	(	const llama_model *	model,
		llama_token	token,
		bool	special = `true`
	)

inline

Detokenize SINGLE token to text (model-accepting overload)

Parameters

model	Llama model
token	Token ID to convert
special	Enable special token rendering (default: true)

Returns: Text representation of token

Definition at line 289 of file tokenizer.hpp.

◆ detokenize() [2/2]

std::string lloyal::tokenizer::detokenize	(	const llama_vocab *	vocab,
		llama_token	token,
		bool	special
	)

inline

Detokenize SINGLE token to text (streaming use case)

Fast synchronous operation for per-token conversion during generation. AVOID CONFUSION: This is NOT llama_decode (KV cache update).

Parameters

vocab	Vocabulary from llama_model_get_vocab()
token	Token ID to convert
special	Enable special token rendering (e.g., "<\|im_start\|>")

Returns: Text representation of token

Definition at line 91 of file tokenizer.hpp.

◆ detokenize_batch() [1/3]

std::string lloyal::tokenizer::detokenize_batch	(	const llama_model *	model,
		const llama_token *	tokens,
		int32_t	n_tokens,
		bool	remove_special,
		bool	unparse_special
	)

inline

Detokenize TOKEN ARRAY to text (model-accepting overload)

Parameters

model	Llama model
tokens	Array of token IDs
n_tokens	Number of tokens in array
remove_special	Remove BOS/EOS tokens from output
unparse_special	Render special tokens as text

Returns: Complete text representation

Definition at line 348 of file tokenizer.hpp.

◆ detokenize_batch() [2/3]

std::string lloyal::tokenizer::detokenize_batch	(	const llama_model *	model,
		const std::vector< llama_token > &	tokens,
		bool	remove_special = `false`,
		bool	unparse_special = `true`
	)

inline

Detokenize TOKEN VECTOR to text (convenience overload)

Accepts std::vector instead of raw pointer for safer API.

Parameters

model	Llama model
tokens	Vector of token IDs
remove_special	Remove BOS/EOS tokens from output (default: false)
unparse_special	Render special tokens as text (default: true)

Returns: Complete text representation

Definition at line 316 of file tokenizer.hpp.

◆ detokenize_batch() [3/3]

std::string lloyal::tokenizer::detokenize_batch	(	const llama_vocab *	vocab,
		const llama_token *	tokens,
		int32_t	n_tokens,
		bool	remove_special,
		bool	unparse_special
	)

inline

Detokenize TOKEN ARRAY to text (reconstruction use case)

Batch operation for reconstructing complete text from token sequences. AVOID CONFUSION: This is NOT llama_decode (KV cache update).

Parameters

vocab	Vocabulary from llama_model_get_vocab()
tokens	Array of token IDs
n_tokens	Number of tokens in array
remove_special	Remove BOS/EOS tokens from output
unparse_special	Render special tokens as text (e.g., "<\|im_start\|>")

Returns: Complete text representation

Definition at line 137 of file tokenizer.hpp.

◆ get_vocab()

const llama_vocab * lloyal::tokenizer::get_vocab ( const llama_model * model )

inline

Get vocabulary from model.

Simple accessor that wraps llama_model_get_vocab(). Isolates direct llama.cpp model API dependency.

Parameters

model Llama model

Returns: Vocabulary pointer (never null if model is valid)

Examples: /home/runner/work/liblloyal/liblloyal/include/lloyal/grammar.hpp.

Definition at line 196 of file tokenizer.hpp.

◆ is_eog() [1/2]

bool lloyal::tokenizer::is_eog	(	const llama_model *	model,
		llama_token	token
	)

inline

Check if token is end-of-generation marker (model-accepting overload)

Parameters

model	Llama model
token	Token ID to check

Returns: True if token marks end of generation

Definition at line 375 of file tokenizer.hpp.

◆ is_eog() [2/2]

bool lloyal::tokenizer::is_eog	(	const llama_vocab *	vocab,
		llama_token	token
	)

inline

Check if token is end-of-generation marker.

Parameters

vocab	Vocabulary from get_vocab()
token	Token ID to check

Returns: True if token marks end of generation

Definition at line 218 of file tokenizer.hpp.

◆ tokenize() [1/2]

std::vector< llama_token > lloyal::tokenizer::tokenize	(	const llama_model *	model,
		const std::string &	text
	)

inline

Tokenize text to token array (model-accepting overload)

Convenience wrapper that handles:

Vocab extraction from model
add_bos detection from GGUF metadata
Special token parsing

Parameters

model	Llama model
text	Text to tokenize

Returns: Vector of token IDs

Definition at line 264 of file tokenizer.hpp.

◆ tokenize() [2/2]

std::vector< llama_token > lloyal::tokenizer::tokenize	(	const llama_vocab *	vocab,
		const std::string &	text,
		bool	add_special,
		bool	parse_special
	)

inline

Tokenize text to token array.

Parameters

vocab	Vocabulary from llama_model_get_vocab()
text	Text to tokenize
add_special	Add special tokens (BOS/EOS) if model configured
parse_special	Parse special token strings like "<\|im_start\|>"

Returns: Vector of token IDs

Definition at line 38 of file tokenizer.hpp.

◆ vocab_size() [1/2]

int32_t lloyal::tokenizer::vocab_size ( const llama_model * model )

inline

Get vocabulary size (model-accepting overload)

Parameters

model Llama model

Returns: Number of tokens in vocabulary

Definition at line 396 of file tokenizer.hpp.

◆ vocab_size() [2/2]

int32_t lloyal::tokenizer::vocab_size ( const llama_vocab * vocab )

inline

Get vocabulary size (total number of tokens)

Parameters

vocab Vocabulary from get_vocab()

Returns: Number of tokens in vocabulary

Definition at line 233 of file tokenizer.hpp.

Functions

Function Documentation

◆ detokenize() [1/2]

◆ detokenize() [2/2]

◆ detokenize_batch() [1/3]

◆ detokenize_batch() [2/3]

◆ detokenize_batch() [3/3]

◆ get_vocab()

◆ is_eog() [1/2]

◆ is_eog() [2/2]

◆ tokenize() [1/2]

◆ tokenize() [2/2]

◆ vocab_size() [1/2]

◆ vocab_size() [2/2]