Jev vs pplx-decider flash at the Elite Four

Every run uses the same team, built by hand on October 1 and never changed: Swampert, Gardevoir, Manectric, Salamence, Metagross and Breloom, at level 57 to 58. Each Pokémon knows the moves the game gives it at that level. The bag holds 10 Hyper Potions and 3 Full Restores.

The models never choose the team. They only choose what happens in battle: which move, which switch, which item. The opponents are the game's own Elite Four and Champion teams.

Hidden stats (IVs and nature) roll when the team is created, so the team's total max HP ranges from 991 to 1,009 across runs.

We gave Jev a level 57 to 58 team, 10 Hyper Potions and 3 Full Restores, and sent it through the Elite Four with no healing in between. Each turn, code listed the legal moves, switches and items with the facts that matter (type matchups, a KO estimate, HP), and Jev returned odds for each option.

Jev beat Sidney, Phoebe and Glacia, then lost to Drake. Its 63 decisions took 15.3 seconds in total, at a 225 ms median, and the run cost $0.0038. Jev never picked a potion and never gave one more than 11%.

That run suggested a small decision model can play real battles, provided code hands it a clean list. A one-model result can't tell you whether the list or the model did the work. A second model can.

We plugged Perplexity's pplx-decider into the same harness with the same list and the same instructions, and changed only the API call. In its first run, it beat Sidney and Phoebe, then picked Hyper Potion 20 times against Glacia. Not one potion was used.

The screenshots showed why. The bag reports that it's open while it is still fading in, and it ignores button presses until the fade ends. Our first A press vanished, so the second A opened the item's menu without choosing USE. The harness then backed out and read the fade as "Battle ended." Every potion in that run did nothing, and none of Jev's runs had picked one, so the bug never showed.

Jev's October result stands. The bug still mattered, though. The harness had offered a choice it couldn't carry out, and a model that wanted to heal paid for that. We fixed the harness, checked a potion restoring HP on a test battle, and set the buggy run aside.

Flash and pplx-decider picked the same kinds of actions. Over ten seeds, in the first two battles, both attacked about two-thirds of the time and switched about a third. Yet flash's team lost 9.3 HP per decision, and pplx-decider's lost 14.3.

In the first three seeds, most of pplx-decider's losses came from two repeated matchups. On two seeds, Swampert stayed in against Sidney's Shiftry and lost about 148 HP each time, even though Grass hits Swampert 4×. On two seeds, Breloom attacked Sidney's Absol because the list said Sky Uppercut KOs. Absol moved first and hit Breloom for 110 to 135 HP. On seed 101, flash sent Metagross against Shiftry and Manectric against Absol. Flash wasn't perfect either: on seed 103, its Salamence took a 185 HP hit from Absol. Luck doesn't explain the gap: critical hits and misses came out about even.

That pointed at the list. The KO note only counts our damage. It never says who moves first or what the foe can do back.

We added the missing facts, and the gap stayed

We wrote down a prediction before the runs. If the missing facts caused the gap, pplx-decider's HP lost per decision in the first two battles would fall to 9.5 or below, and it would win more battles. Then every move option got two new notes: who moves first, and whether the foe's best move can KO your Pokémon. Same team, same seeds, three runs per model. The notes use the foe's real moves, which a human player wouldn't see.

The prediction failed. pplx-decider's HP lost per decision went from 15.0 to 14.1, and its wins stayed at 2.33 per run. With the notes, it switched Swampert away from Shiftry on two of three seeds. It still attacked Absol with Breloom on both seeds where they met, with "foe moves first" on the option.

The notes did change one thing. Every model healed far more: pplx-decider used 37 items instead of 6, and Jev used 13 instead of 0. Win counts moved closer together, flash 2.67, pplx-decider 2.33, Jev 2.33 per run. At three seeds, a single result flips that order, so we ran ten.

Ten seeds per model

We pre-registered three calls before the rerun, then ran seeds 104 to 110 for every model in both setups. That gives ten seeds per model in each setup, 60 runs in all.

Flash leads under game rules. It won 3.0 battles per run, against 2.4 for both pplx-decider and Jev. The gap is 0.6, with a 90% interval of 0.1 to 0.9.

The threat notes close that lead. With them, pplx-decider won 2.8 per run, flash 2.6 and Jev 2.4. Flash's gap went from +0.6 to −0.2. The change has a 90% interval of −1.5 to −0.1, which clears zero by a small margin.

The HP gap held in both setups. Flash lost 9.3 HP per decision in the first two battles, and 7.3 with the notes. The other two lost 13 to 14 either way.

So flash plays the early battles better with or without a warning. pplx-decider still takes the same early hits with the notes, but it heals through them and goes further: it beat Glacia 7 times in 10 with the notes, against 4 without. The notes made every model heal more. That helped pplx-decider. Flash healed more too, 122 items against 50, and won less.

Flash heals far more than the other two. With the full bag it used 50 items in ten runs, against 15 for pplx-decider and none for Jev. To test whether that mattered, we ran the same ten seeds with two other bags: 3 Hyper Potions, and 13 Hyper Potions.

Two of our predictions were wrong. We expected flash to waste the small bag early. It reached Glacia with 2.7 potions on average, about the same as the others. We also can't explain one result. pplx-decider beat Glacia 8 times in 10 with either all-potion bag, and 4 times with the full one.

Same states, different odds

Each model reaches different battles, so the runs can't show whether flash likes healing or just lands where healing looks good. We took 40 logged decisions, changed the active Pokémon's HP and the potions left, and asked all three models about every version.

Those odds rarely put the potion first. Above half HP, flash ranked it first in 3.3% of answers and the other two never did. We had predicted a gap of at least 10 points, so part of flash's healing in the runs comes from the battles it reached.

Is it the count or the extra line?

The small bag changed two things at once. It held fewer items, and it had no Full Restore line. Harrison guessed the count mattered. Claude guessed the extra line did, since it gives flash a second item to rank first. We wrote both guesses down before testing.

Then we ran the ten seeds with 13 Hyper Potions on one line. Flash healed at half HP or more in 10.6% of decisions, like the full bag and unlike 3 potions (first chart). The count drives flash's healing. Splitting the probe by potions left points the same way, though we did it after the fact. Above half HP, flash ranked the potion first in 6.7% of answers with 10 left and 0.8% with 3.

Flash's lead fell from 0.6 wins to −0.1. Our rule called that a shrink only if the 90% interval stayed below zero. Its top end landed at exactly zero, so by the rule there's no clear change.

Does the foe take the KO?

My guess is that the game's AI doesn't always take a KO it has, which would make aggressive healing pay. The Elite Four use the game's try-to-faint AI. In the runs with threat notes, we counted turns where the list said "foe KOs you". That note means even the foe's weakest damage roll knocks our Pokémon out.

We counted this after the runs and didn't pre-register it. Treat it as a lead to test.

On October 1, Jev beat Glacia. Today it lost to her in all three runs. We can't say why yet. The October result is a single run. Since then, the battle list changed in two ways for other studies: the harness tracks its own switches, and the model now sees what the game said between choices. The game build changed too, and the team's hidden stats roll differently on every run.

We publish both results. The comparison on this page only uses today's runs, which share one harness, one build and three seeds.

The October idea held up. A second company's decision models, never tuned for this harness, plugged into the same list and played as well as Jev or better. The list carries across models, which makes small decision models a real option for this kind of agent.

The models still play differently. All three usually took a clear KO. They split on resource calls, like when to heal or switch, and on how much HP they kept for the next battle. Those calls decided the runs.

The bug says something too. A list harness is only as good as its list. Our first model never asked for a potion, so a broken potion never showed. Running a second model is a cheap way to test the harness.

Three seeds per model in the main comparison, ten in the rerun under Why flash won. One team, one gauntlet.