metrics.hpp File Reference

Distribution Metrics for Test-Time Alignment. More...

#include <cmath>
#include <cstdint>
#include <limits>
#include <unordered_map>
#include <vector>

Go to the source code of this file.

Classes
struct	lloyal::metrics::detail::PerplexityState
struct	lloyal::metrics::detail::BranchMetricsState

Namespaces
namespace	lloyal
	JSON Schema to Grammar Converter (Header-Only)
namespace	lloyal::metrics
namespace	lloyal::metrics::detail

Typedefs
using	lloyal::metrics::PerplexityHandle = int32_t
using	lloyal::metrics::BranchMetricsHandle = int32_t

Enumerations
enum class	lloyal::metrics::Base { lloyal::metrics::Nats , lloyal::metrics::Bits }

Functions
float	lloyal::metrics::detail::max_finite (const float *a, int n)
	Find maximum finite value in array Used for log-sum-exp shift to prevent overflow.
float	lloyal::metrics::detail::log_sum_exp (const float *a, int n, float shift)
	Numerically stable log-sum-exp Computes log(Σ exp(aᵢ)) using shift trick to avoid overflow.
std::unordered_map< PerplexityHandle, PerplexityState > &	lloyal::metrics::detail::get_registry ()
PerplexityHandle &	lloyal::metrics::detail::get_next_handle ()
float	lloyal::metrics::model_surprisal (const float *logits, int n_vocab, int picked_id, Base base=Base::Nats)
float	lloyal::metrics::model_entropy (const float *logits, int n_vocab, Base base=Base::Nats)
float	lloyal::metrics::sampling_surprisal (const float *candidate_logits, const int32_t *candidate_ids, int n_candidates, int picked_id, Base base=Base::Nats)
	Compute sampling-level surprisal for picked token.
float	lloyal::metrics::sampling_entropy (const float *candidate_logits, int n_candidates, Base base=Base::Nats)
	Compute sampling-level entropy of candidate distribution.
PerplexityHandle	lloyal::metrics::create_perplexity ()
void	lloyal::metrics::add_surprisal (PerplexityHandle handle, float surprisal)
	Add token surprisal to running average.
float	lloyal::metrics::get_ppl (PerplexityHandle handle)
	Get current perplexity.
int	lloyal::metrics::get_count (PerplexityHandle handle)
	Get number of tokens added to tracker.
void	lloyal::metrics::reset_perplexity (PerplexityHandle handle)
	Reset tracker to initial state (start new sequence)
PerplexityHandle	lloyal::metrics::clone_perplexity (PerplexityHandle handle)
void	lloyal::metrics::free_perplexity (PerplexityHandle handle)
	Free perplexity tracker.
std::unordered_map< BranchMetricsHandle, BranchMetricsState > &	lloyal::metrics::detail::get_branch_metrics_registry ()
BranchMetricsHandle &	lloyal::metrics::detail::get_next_branch_metrics_handle ()
BranchMetricsHandle	lloyal::metrics::create_branch_metrics ()
	Create unified branch metrics tracker.
void	lloyal::metrics::free_branch_metrics (BranchMetricsHandle handle)
	Free branch metrics tracker.
BranchMetricsHandle	lloyal::metrics::clone_branch_metrics (BranchMetricsHandle handle)
	Clone branch metrics tracker (for fork/branching)
void	lloyal::metrics::add_model_surprisal (BranchMetricsHandle handle, float surprisal)
	Add model-level surprisal (from raw logits before filters)
float	lloyal::metrics::get_model_ppl (BranchMetricsHandle handle)
	Get model-level perplexity (from raw logits)
void	lloyal::metrics::add_sampling_surprisal (BranchMetricsHandle handle, float surprisal)
	Add sampling-level surprisal (from filtered distribution)
float	lloyal::metrics::get_sampling_ppl (BranchMetricsHandle handle)
	Get sampling-level perplexity (from filtered distribution)
int	lloyal::metrics::get_model_count (BranchMetricsHandle handle)
	Get number of tokens in model-level tracker.
int	lloyal::metrics::get_sampling_count (BranchMetricsHandle handle)
	Get number of tokens in sampling-level tracker.

Variables
constexpr float	lloyal::metrics::detail::LN2 = 0.693147180559945309417232121458176568f

Detailed Description

Distribution Metrics for Test-Time Alignment.

Computes surprisal, entropy, and perplexity from logits (no attention needed). All metrics derive from softmax(logits) with numerically stable log-sum-exp.

Two measurement levels:

• Model metrics: Raw logits (before filters) - model's inherent belief
• Sampling metrics: Post-filter logits (after top-k/p/temp) - actual distribution sampled

Use cases:

• KV eviction gates: High entropy -> trigger retrieval or cache pruning
• Adaptive sampling: Collapsed distribution -> widen search
• Quality monitoring: Track surprisal/perplexity for confidence estimates
• Dashboard signals: Real-time uncertainty visualization

References:

• Shannon entropy: https://www.emblaustralia.org/wp-content/uploads/2023/11/information_theory.pdf
• Perplexity: https://huggingface.co/docs/transformers/perplexity

Ported from tsampler/metrics.ts - identical algorithms, validated implementation.

Definition in file metrics.hpp.