lloyal.node API Reference - v1.0.7
    Preparing search index...

    Function withLogits

    • Safe logits access with automatic lifetime management

      Ensures logits are only accessed synchronously within the callback. The callback MUST NOT:

      • Store the logits reference
      • Return a Promise (will throw)
      • Call decode() (would invalidate logits)

      This prevents common bugs where logits become invalid due to async operations between access and usage.

      How it works:

      • Memoization: Multiple getLogits() calls in same step return same buffer
      • Revocation: Next decode() invalidates previous buffer

      Type Parameters

      • T

        Return type of the callback

      Parameters

      • ctx: SessionContext

        The session context

      • fn: (logits: Float32Array) => T

        Synchronous callback that uses logits - must not return a Promise

      Returns T

      The result from the callback

      Error if callback returns a Promise (async usage not allowed)

      // Compute entropy synchronously
      const entropy = withLogits(ctx, (logits) => {
        let maxLogit = logits[0];
        for (let i = 1; i < logits.length; i++) {
          if (logits[i] > maxLogit) maxLogit = logits[i];
        }

        let sumExp = 0;
        for (let i = 0; i < logits.length; i++) {
          sumExp += Math.exp(logits[i] - maxLogit);
        }

        let entropy = 0;
        for (let i = 0; i < logits.length; i++) {
          const p = Math.exp(logits[i] - maxLogit) / sumExp;
          if (p > 0) entropy -= p * Math.log(p);
        }
        return entropy;
      });

      // Now safe to decode (previous logits buffer is revoked)
      await ctx.decode([nextToken], position++);

      // This will throw!
      withLogits(ctx, async (logits) => {
        await something();  // NOT ALLOWED
        return logits[0];
      });

    Settings

    Member Visibility