lloyal.node API Reference - v1.0.7
    Preparing search index...

    Class Branch

    Forkable inference handle for covalent generation

    A Branch owns everything needed for independent generation: a KV cache sequence, sampler chain, logits snapshot, and perplexity tracker.

    Forking is cheap — the KV prefix is shared in memory (metadata-only operation under unified KV — no KV tensor buffers are copied), so sibling branches read from the same physical KV entries. Only tokens decoded after the fork point are exclusive to each branch.

    Branches form trees, not just flat lists. Fork from root for best-of-N, fork from children for MCTS/beam search, fork from a draft for speculative decoding.

    The produce/commit protocol separates sampling from state advancement: produce() samples without writing to KV, letting you inspect the result before deciding to commit().

    const root = Branch.create(ctx, 0, tokens.length, { temperature: 0.8 });
    root.captureLogits();

    const candidates = [1, 2, 3, 4, 5].map((seqId, i) => {
      const branch = root.fork(seqId);
      branch.reseedSampler(1000 + i);
      return branch;
    });

    for (let t = 0; t < 50; t++) {
      for (const branch of candidates) {
        const { token, isStop } = branch.produce();
        if (isStop) continue;
        branch.commit(token);
      }
    }

    const best = candidates.reduce((a, b) => a.perplexity < b.perplexity ? a : b);
    for (const c of candidates) { if (c !== best) c.prune(); }
    Index

    Constructors

    constructor

    Properties

    disposed handle perplexity position seqId

    Methods

    accept captureLogits commit decodeAndCaptureOne destroy fork produce prune reseedSampler sample create

    Constructors

    Properties

    disposed: boolean

    Whether this branch has been disposed

    handle: number

    Internal handle (for debugging)

    perplexity: number

    Branch's perplexity

    position: number

    Branch's current position

    seqId: number

    Branch's sequence ID

    Methods

    • Accept token for repeat-penalty tracking

      Parameters

      • token: number

      Returns void

    • Freeze the current logit distribution into this branch. Essential before fork().

      Returns void

    • Accept and advance — write token to KV and update branch state.

      Parameters

      • token: number

      Returns void

    • Decode a single token, write to KV, and capture resulting logits

      Parameters

      • token: number

      Returns void

    • Release handle but keep KV entries intact (use for winners, continue with raw ops)

      Returns void

    • Fork this branch to a new sequence

      The child shares the parent's KV prefix in memory (metadata-only under unified KV, no KV buffer copy). Logits, sampler state, and perplexity tracker are cloned so the child can diverge independently. Fork from any branch — root or intermediate — to build arbitrarily deep trees.

      Parameters

      • newSeqId: number

        Sequence ID for the forked branch

      Returns Branch

    • Discard branch — remove its divergent KV entries and free the handle (use for losers)

      Returns void

    • Reseed the sampler's PRNG for diversity after fork()

      CRITICAL for parallel generation: Without reseeding, all forked branches produce identical outputs because they share the same PRNG state.

      Only affects stochastic samplers (temperature > 0). Greedy samplers are unchanged.

      Parameters

      • seed: number

        New seed for the PRNG

      Returns void

    • Sample next token from branch's frozen logits snapshot

      Returns number

    • Create a root branch at the given position

      The branch takes ownership of the sequence and creates its own sampler chain from the provided params. Call captureLogits() after prefill to freeze the logit distribution before forking.

      Parameters

      • ctx: SessionContext

        SessionContext to create branch on

      • seqId: number

        Sequence ID for this branch

      • position: number

        Starting position (typically prompt token count)

      • Optionalparams: SamplingParams

        Sampling parameters (temperature, topP, etc.)

      Returns Branch

    Settings

    Member Visibility

    On This Page

    Constructors
    Properties
    Methods