PES: A Parrondo-Inspired Entropy Switcher for Hysteretic Adaptive Decoding

Abstract

1. Introduction

Temperature is one of the most widely used controls in LLM decoding. Low temperature tends to improve local coherence and determinism, but can become brittle, repetitive, or prematurely committed — a failure mode we might call the Curse of Recursion: the model locks into a high-probability attractor and cannot escape. High temperature broadens exploration and diversity, but can also inject drift, syntactic instability, and incoherence — Variance Amplification that erodes the generation's informational health.

A fixed temperature is therefore a compromise, and compromises are often weakest exactly when the decoding process changes phase. That observation already has support in the recent decoding literature: EDT dynamically selects temperature from token-level entropy; AdapT uses larger temperature for challenging code tokens; entropy-aware decoding intervenes when entropy breaches upper and lower bands; AdaDec learns model-specific entropy thresholds; Top-H treats entropy as a direct control target.

The core intuition is Parrondo-inspired. In the classical paradox, two individually losing games become winning when alternated under the right switching rule. The analogy for LLMs is not literal — we are not composing losing strategies into a winning one in a strict mathematical sense. But the spirit is the same: using low temperature alone loses on diversity; using high temperature alone loses on coherence. A state-conditioned switcher with memory can outperform both.

2. Claim and Framing

PES is Parrondo-inspired, not Parrondo-proved. In the classical Parrondo literature, both random and deterministic mixtures of losing games can become winning under the right conditions, so the analogy must not be overstated. The right empirical claim for LLM decoding is narrower:

That framing is stronger scientifically because it points directly to the ablations that matter. The question is not just whether PES improves pass@1, but whether it changes the occupancy of low-, medium-, and high-entropy states in a way matched non-stateful policies do not.

3. Theoretical Foundations

3.1 The Parrondo Mapping

In the original Parrondo setting, a player's capital performs a random walk. Two games $A$ and $B$ are individually drift-negative (losing), yet their alternation can be drift-positive (winning). The key ingredient is a state-dependent coupling: game $B$'s transition matrix depends on the player's current capital modulo some integer $M$.

In our setting, the analog of capital is the informational negentropy of the generation sequence — a measure of how far the current decoding trajectory is from maximum entropy (pure noise). Low negentropy means the model is collapsed; high negentropy means it is drifting. The controller acts as an informational ratchet, preventing the trajectory from collapsing to either extreme.

3.2 Basic Notation

Let $z_t \in \mathbb{R}^V$ be the raw logits at decoding step $t$, before any sampling transformation. Define the base distribution:

We compute token-level Shannon entropy, optionally restricted to a top-$K$ set $S_t$ with renormalized probabilities $\tilde{p}_t$:

We normalize by support size to obtain a bounded entropy signal $h_t \in [0,1]$:

To reduce jitter, we maintain an exponentially smoothed entropy state:

The parameter $\lambda$ controls the memory horizon. $\lambda = 0$ is memoryless (pure reactive); $\lambda \to 1$ is infinite memory (ignores current token). A typical choice is $\lambda \approx 0.9$, giving an effective window of roughly 10 tokens.

3.3 State Space and Transition Structure

Define three controller modes and their associated temperatures:

The state-transition graph is not a free random walk — it is a hysteretic finite automaton. Transitions require sustained evidence (dwell count $\geq N_{\text{low}}$ or $N_{\text{high}}$), giving the system its Markov memory.

4. The PES Algorithm & Hysteresis Dynamics

The full controller pseudocode, annotated:

# PES controller — runs once per decoded token
# Params: h_low, h_high, N_low, N_high, λ, T_HOT, T_MID, T_COLD

state   = MID
counter = 0
m       = 0.5    # smoothed entropy

for each token step t:
    p_t  = softmax(z_t)              # or renorm top-K approx
    h_t  = H(p_t) / log|S_t|        # normalised entropy ∈ [0,1]
    m    = λ·m + (1-λ)·h_t          # EMA smoothing

    if   m < h_low  → counter_low  += 1;  counter_high = 0
    elif m > h_high → counter_high += 1;  counter_low  = 0
    else            → counter_low   = 0;  counter_high = 0

    if   counter_low  >= N_low:   state = HOT
    elif counter_high >= N_high:  state = COLD
    # else: state unchanged (hysteresis — no chatter)

    y_t ~ softmax(z_t / T(state))    # sample with active temperature

4.1 Why Hysteresis is the Mechanism, Not a Detail

Without dwell times, PES reduces to simple thresholding and can chatter near the boundary — switching at every token when $m_t \approx h_{\text{low}}$ or $m_t \approx h_{\text{high}}$. Chatter is not just noisy; it fundamentally breaks the Parrondo analogy. The paradox requires that switching decisions carry information about the current state, not just reactive one-token observations.

4.2 Extended Policy: Reranking Under High Uncertainty

High entropy is ambiguous. Sometimes it marks semantic drift — cooling may help. Sometimes it marks a pivotal decision where the correct token is present but misranked — cooling may lock in the wrong choice. We therefore define an extended three-action controller with a top-token margin signal $\Delta_t = p_{t,(1)} - p_{t,(2)}$:

This turns PES from a sampler into a self-correcting inference meta-policy that monitors its own informational health and escalates its intervention when margin evidence suggests the distribution is not merely uncertain but genuinely ambiguous at the top.

5. Why Entropy is the Right State Variable

For fixed logits $z$, define the temperature-scaled distribution:

Let $\beta = 1/T$ (inverse temperature). Then entropy under $p_T$ can be written via the log-partition function $\log Z(\beta) = \log \sum_i e^{\beta z_i}$:

Differentiating with respect to $\beta$ and converting back to $T$:

This is the mathematical foundation for using entropy as a control signal. When the controller detects sustained low entropy, it increases $T$, which is guaranteed to increase $H_t$ on the next step (for the same logit context). The feedback loop is sign-stable by the theorem above.

The monotonicity argument does not imply that a single global target entropy is optimal, since the logits evolve with context as new tokens are committed. But it does justify entropy as a meaningful, actionable signal rather than a passive diagnostic.

6. Implementation Notes

PES is a minimal, infrastructure-free intervention. The computational overhead is $O(K)$ per token for top-$K$ entropy estimation — near zero latency relative to the forward pass.

In vLLM, logits processors operate on batch-sized tensors of raw logits, can maintain per-request state, and can be loaded as custom processors without recompiling the engine. The current implementation fix is to use base temperature $T=1$ and absorb all scaling into the processor, since temperature is currently still hard-coded in some vLLM sampler paths.

For lightweight local prototyping, Ollama's generate API exposes logprobs and top_logprobs, which is sufficient to drive an external token-by-token controller using approximate top-$K$ entropy. This prototype should be described honestly as an approximation — it only sees the most likely tokens rather than the full vocabulary distribution.

# Minimal vLLM logits processor sketch
class PESProcessor:
    def __init__(self, h_low=0.3, h_high=0.75, N_low=5, N_high=5,
                 lam=0.9, T_hot=1.4, T_mid=1.0, T_cold=0.6):
        self.state = "MID"
        self.m     = 0.5
        self.count_low = self.count_high = 0
        # ... store params

    def __call__(self, input_ids, logits):
        # 1. estimate top-K entropy from logits
        topk = torch.topk(logits, 50, dim=-1)
        probs = torch.softmax(topk.values, dim=-1)
        h = -(probs * probs.log()).sum(-1).mean().item()
        h /= math.log(50)          # normalise

        # 2. update smoothed state
        self.m = self.lam * self.m + (1 - self.lam) * h

        # 3. update dwell counters
        if   self.m < self.h_low:  self.count_low  += 1; self.count_high = 0
        elif self.m > self.h_high: self.count_high += 1; self.count_low  = 0
        else:                      self.count_low   = 0; self.count_high = 0

        # 4. transition (hysteresis)
        if   self.count_low  >= self.N_low:  self.state = "HOT"
        elif self.count_high >= self.N_high: self.state = "COLD"

        # 5. apply temperature
        T = {"HOT": self.T_hot, "MID": self.T_mid, "COLD": self.T_cold}[self.state]
        return logits / T

7. Experimental Protocol: Proving the Gain

The primary evaluation target should be code generation, where decoding choices are consequential and correctness can be measured precisely. The main reported number should be pass@1 under identical prompts, context windows, and stopping rules, with fixed compute budgets across methods.

A convincing experiment compares PES against five baseline families:

The paper should pre-register the mechanism tests and log: token-wise entropy, smoothed entropy $m_t$, controller state, switch counts, dwell durations, top-token margin $\Delta_t$, repetition statistics, and failure locations. The strongest evidence is not merely "PES scored higher" — it is: PES entered better uncertainty regimes, stayed there longer, and those trajectory differences explain the performance gain.

8. Extensions: The Meta-Policy Vision

8.1 From Switching to Continuous Control

Instead of choosing from $\{T_{\text{HOT}}, T_{\text{MID}}, T_{\text{COLD}}\}$, PES can solve at each step for the temperature that places the post-scale distribution inside a target entropy band, then use hysteresis only to prevent chatter. This turns PES into an entropy-band controller — a continuous generalization of the binary switcher.

8.2 Learned Switching Policies

Moving from scalar thresholds to a neural policy that looks at $\Delta H_t$, repetition markers, KL divergence from prior tokens, and syntactic position. This is the transition from a hand-engineered ratchet to a trained inference policy, consistent with the direction of AdaDec's learned thresholds.

8.3 Multi-Parameter PES

The switcher logic can be applied jointly to $T$, top-$p$, top-$k$, and repetition penalties — a coordinated hyperparameter policy rather than a single-axis controller. Each axis has its own Parrondo-like trade-off between precision and diversity.

8.4 Hierarchical and Agentic PES

A token-level controller nested inside a higher-level agentic strategy: tool-using loops, long-form reasoning traces, and code-editing agents all exhibit even more distinct decoding phases than single-pass generation. In those settings, PES becomes a general inference policy layer rather than a sampling trick.

9. Conclusion

Fixed temperature is a static answer to a dynamic inference process. PES is a principled way to add memory to decoding: it treats Shannon entropy as an observable, controllable state variable, and uses a hysteretic finite-state controller to keep the generation trajectory inside productive uncertainty regimes.

The real contribution is not that hot and cold temperatures become magical in combination. The real contribution is the trajectory hypothesis: that the sequence of uncertainty states visited during decoding matters, that stateful control shapes that sequence, and that the right ablation table — testing against matched non-stateful switching — can isolate memory as the source of any gains.

PES should be presented as a Parrondo-inspired adaptive decoding policy, not as a literal instantiation of the original paradox. That is a feature, not a weakness.