The table
No clear winner after 24 battles each
About 24 battles each is too few to rank them. Even Gemini 3 Flash's 15 wins in 24 could be luck: a coin would do that well about one time in seven. Treat the order as provisional.
The game decides each battle. Sorted by battles won. A missed KO is a turn where a guaranteed knockout was on the list and the model picked something else. 4 battles count as forfeits: DeepSeek V4 Flash spent its whole 8,000-token reply on thinking and never chose a move, though every model was asked to keep thinking to 1,024 tokens.
| Model | Won | Missed KOs | Per decision | Per battle |
|---|---|---|---|---|
| Gemini 3 Flash |
63% 15 of 24 |
2% 6 of 410 |
12.4 s | $0.033 |
| Jev TypeSafe |
50% 12 of 24 |
1% 2 of 376 |
208 ms | $0.001 |
| GPT-5 mini OpenAI |
50% 12 of 24 |
0% 2 of 566 |
11.9 s | $0.049 |
| Haiku 4.5 Anthropic |
46% 11 of 24 |
0% 1 of 398 |
9.0 s | $0.074 |
| DeepSeek V4 Flash DeepSeek |
42% 10 of 24 |
2% 8 of 437 |
68.5 s | $0.01 |
- Gemini 3 Flash Google
- Won
- 63% (15/24)
- Missed KOs
- 2%
- Per decision
- 12.4 s
- Per battle
- $0.033
- Jev TypeSafe
- Won
- 50% (12/24)
- Missed KOs
- 1%
- Per decision
- 208 ms
- Per battle
- $0.001
- GPT-5 mini OpenAI
- Won
- 50% (12/24)
- Missed KOs
- 0%
- Per decision
- 11.9 s
- Per battle
- $0.049
- Haiku 4.5 Anthropic
- Won
- 46% (11/24)
- Missed KOs
- 0%
- Per decision
- 9.0 s
- Per battle
- $0.074
- DeepSeek V4 Flash DeepSeek
- Won
- 42% (10/24)
- Missed KOs
- 2%
- Per decision
- 68.5 s
- Per battle
- $0.01
By scenario: Gemini 3 Flash Draft duel 5/8, Mirror match 4/8, Underdog 6/8 · Jev Draft duel 4/8, Mirror match 5/8, Underdog 3/8 · GPT-5 mini Draft duel 5/8, Mirror match 4/8, Underdog 3/8 · Haiku 4.5 Draft duel 2/8, Mirror match 3/8, Underdog 6/8 · DeepSeek V4 Flash Draft duel 4/8, Mirror match 4/8, Underdog 2/8.
Side check: the player's side won 60% of 60 battles. Sides swap every game, so the gap reflects the game, and it favors no model.
Says vs thinks
The stated reason and the thinking disagreed
Every model wrote a one-line reason for each move, and most also thought first. A script flagged 28 of 1,947 decisions where the two seemed to disagree. Reading all 28 in full, 8 held up; 17 were false alarms, mostly hedged reasons like "toward a KO", and 3 couldn't be judged. Three of the real ones:
Says
Slash deals the most damage (25-30) to finish off the severely damaged Altaria at 62 HP.
Thinks
If I do 25 damage, Altaria has 37 HP left. If I do 30 damage, Altaria has 32 HP left. This doesn't KO outright, but it's the best damage.
Does
Slash
The game
Move log ▶
A critical hit, but Altaria survives and attacks back.
Says
Lugia is likely faster and two Gusts will KO Sharpedo while surviving the incoming Skull Bash.
Thinks
Skull Bash is coming, and a direct Gust won't secure a KO. Recover offers a safe play against the incoming hit.
Does
Gust
The game
Move log ▶
Sharpedo survives the Gust and lands Skull Bash.
Says
Earthquake should KO Surskit now, so finish it quickly.
The list said
Earthquake: 0.5×, not very effective against Surskit.
Does
Earthquake
The game
Move log ▶
Surskit survives and attacks on the next turn.
Head to head
Wins, row against column
| Gemini 3 Flash | Jev | GPT-5 mini | Haiku 4.5 | DeepSeek V4 Flash | |
|---|---|---|---|---|---|
| Gemini 3 Flash | · | 5–1 | 3–3 | 2–4 | 5–1 |
| Jev | 1–5 | · | 3–3 | 5–1 | 3–3 |
| GPT-5 mini | 3–3 | 3–3 | · | 2–4 | 4–2 |
| Haiku 4.5 | 4–2 | 1–5 | 4–2 | · | 2–4 |
| DeepSeek V4 Flash | 1–5 | 3–3 | 2–4 | 4–2 | · |
Scenarios
Three ways to fight
- Draft duel
- Both pick 3 from the same 12, snake order, level 50. Pool seed 1001.
- Mirror match
- The same three Pokémon on both sides, so only the decisions differ.
- Underdog
- A draft, but one side plays five levels down. Each model takes a turn as the underdog. Pool seed 1002.
Every battle
60 battles, each with its full log
| Scenario | Matchup | Teams | Winner | Turns | Cost |
|---|---|---|---|---|---|
| Draft duel game 1 · seed 7091 |
Haiku 4.5 vs Jev Haiku 4.5 played the player's side |
Shelgon, Wooper, Gligar Altaria, Nidoqueen, Exeggcute |
Jev | 20 | $0.058 |
| Draft duel game 2 · seed 7092 |
Haiku 4.5 vs Jev Haiku 4.5 played the opponent's side |
Exeggcute, Gligar, Farfetchd Wooper, Nidoqueen, Altaria |
Jev | 23 | $0.061 |
| Draft duel game 1 · seed 7051 |
DeepSeek V4 Flash vs Haiku 4.5 DeepSeek V4 Flash played the player's side |
Nidoqueen, Shelgon, Sunflora Wooper, Altaria, Gligar |
DeepSeek V4 Flash | 33 | $0.099 |
| Draft duel game 2 · seed 7052 |
DeepSeek V4 Flash vs Haiku 4.5 DeepSeek V4 Flash played the opponent's side |
Nidoqueen, Wooper, Shelgon Altaria, Exeggcute, Sunflora |
DeepSeek V4 Flash | 30 | $0.09 |
| Draft duel game 1 · seed 7041 |
DeepSeek V4 Flash vs GPT-5 mini DeepSeek V4 Flash played the player's side |
Nidoqueen, Parasect, Surskit Exeggcute, Wooper, Altaria |
DeepSeek V4 Flash | 65 | $0.12 |
| Draft duel game 2 · seed 7042 |
DeepSeek V4 Flash vs GPT-5 mini DeepSeek V4 Flash played the opponent's side |
Gligar, Exeggcute, Wooper Nidoqueen, Surskit, Altaria |
GPT-5 mini | 26 | $0.048 |
| Draft duel game 1 · seed 7061 |
DeepSeek V4 Flash vs Jev DeepSeek V4 Flash played the player's side |
Nidoqueen, Altaria, Gligar Wooper, Surskit, Shelgon |
DeepSeek V4 Flash | 26 | $0.005 |
| Draft duel game 2 · seed 7062 |
DeepSeek V4 Flash vs Jev DeepSeek V4 Flash played the opponent's side |
Exeggcute, Nidoqueen, Shelgon Wooper, Gligar, Altaria |
Jev | 31 | $0.015 |
| Draft duel game 1 · seed 7021 |
Gemini 3 Flash vs Haiku 4.5 Gemini 3 Flash played the player's side |
Altaria, Wooper, Exeggcute Nidoqueen, Shelgon, Farfetchd |
Haiku 4.5 | 29 | $0.11 |
| Draft duel game 2 · seed 7022 |
Gemini 3 Flash vs Haiku 4.5 Gemini 3 Flash played the opponent's side |
Wooper, Altaria, Granbull Nidoqueen, Sunflora, Shelgon |
Gemini 3 Flash | 22 | $0.085 |
| Draft duel game 1 · seed 7001 |
Gemini 3 Flash vs DeepSeek V4 Flash Gemini 3 Flash played the player's side |
Altaria, Nidoqueen, Granbull Shelgon, Gligar, Wooper |
Gemini 3 Flash | 18 | $0.041 |
| Draft duel game 2 · seed 7002 |
Gemini 3 Flash vs DeepSeek V4 Flash Gemini 3 Flash played the opponent's side |
Gligar, Altaria, Exeggcute Nidoqueen, Shelgon, Wooper |
Gemini 3 Flash | 36 | $0.061 |
| Draft duel game 1 · seed 7011 |
Gemini 3 Flash vs GPT-5 mini Gemini 3 Flash played the player's side |
Altaria, Wooper, Gligar Shelgon, Nidoqueen, Parasect |
GPT-5 mini | 34 | $0.07 |
| Draft duel game 2 · seed 7012 |
Gemini 3 Flash vs GPT-5 mini Gemini 3 Flash played the opponent's side |
Gligar, Exeggcute, Shelgon Nidoqueen, Surskit, Altaria |
GPT-5 mini | 30 | $0.078 |
| Draft duel game 1 · seed 7031 |
Gemini 3 Flash vs Jev Gemini 3 Flash played the player's side |
Nidoqueen, Exeggcute, Altaria Wooper, Surskit, Gligar |
Gemini 3 Flash | 35 | $0.03 |
| Draft duel game 2 · seed 7032 |
Gemini 3 Flash vs Jev Gemini 3 Flash played the opponent's side |
Exeggcute, Altaria, Shelgon Wooper, Nidoqueen, Gligar |
Gemini 3 Flash | 25 | $0.035 |
| Draft duel game 1 · seed 7071 |
GPT-5 mini vs Haiku 4.5 GPT-5 mini played the player's side |
Parasect, Wooper, Exeggcute Gligar, Nidoqueen, Altaria |
Haiku 4.5 | 40 | $0.15 |
| Draft duel game 2 · seed 7072 |
GPT-5 mini vs Haiku 4.5 GPT-5 mini played the opponent's side |
Nidoqueen, Wooper, Shelgon Altaria, Gligar, Exeggcute |
GPT-5 mini | 30 | $0.12 |
| Draft duel game 1 · seed 7081 |
GPT-5 mini vs Jev GPT-5 mini played the player's side |
Nidoqueen, Parasect, Granbull Wooper, Surskit, Gligar |
GPT-5 mini | 26 | $0.033 |
| Draft duel game 2 · seed 7082 |
GPT-5 mini vs Jev GPT-5 mini played the opponent's side |
Parasect, Exeggcute, Altaria Wooper, Nidoqueen, Gligar |
Jev | 34 | $0.045 |
| Mirror match game 1 · seed 7191 |
Haiku 4.5 vs Jev Haiku 4.5 played the player's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Jev | 42 | $0.10 |
| Mirror match game 2 · seed 7192 |
Haiku 4.5 vs Jev Haiku 4.5 played the opponent's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Jev | 28 | $0.057 |
| Mirror match game 1 · seed 7151 |
DeepSeek V4 Flash vs Haiku 4.5 DeepSeek V4 Flash played the player's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
DeepSeek V4 Flash | 31 | $0.068 |
| Mirror match game 2 · seed 7152 |
DeepSeek V4 Flash vs Haiku 4.5 DeepSeek V4 Flash played the opponent's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
DeepSeek V4 Flash | 44 | $0.11 |
| Mirror match game 1 · seed 7141 |
DeepSeek V4 Flash vs GPT-5 mini DeepSeek V4 Flash played the player's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
GPT-5 mini DeepSeek V4 Flash forfeited | 31 | $0.052 |
| Mirror match game 2 · seed 7142 |
DeepSeek V4 Flash vs GPT-5 mini DeepSeek V4 Flash played the opponent's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
GPT-5 mini DeepSeek V4 Flash forfeited | 38 | $0.068 |
| Mirror match game 1 · seed 7161 |
DeepSeek V4 Flash vs Jev DeepSeek V4 Flash played the player's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Jev DeepSeek V4 Flash forfeited | 35 | $0.015 |
| Mirror match game 2 · seed 7162 |
DeepSeek V4 Flash vs Jev DeepSeek V4 Flash played the opponent's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
DeepSeek V4 Flash | 29 | $0.003 |
| Mirror match game 1 · seed 7121 |
Gemini 3 Flash vs Haiku 4.5 Gemini 3 Flash played the player's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Haiku 4.5 | 30 | $0.086 |
| Mirror match game 2 · seed 7122 |
Gemini 3 Flash vs Haiku 4.5 Gemini 3 Flash played the opponent's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Haiku 4.5 | 36 | $0.11 |
| Mirror match game 1 · seed 7101 |
Gemini 3 Flash vs DeepSeek V4 Flash Gemini 3 Flash played the player's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Gemini 3 Flash | 22 | $0.027 |
| Mirror match game 2 · seed 7102 |
Gemini 3 Flash vs DeepSeek V4 Flash Gemini 3 Flash played the opponent's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
DeepSeek V4 Flash | 38 | $0.047 |
| Mirror match game 1 · seed 7111 |
Gemini 3 Flash vs GPT-5 mini Gemini 3 Flash played the player's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Gemini 3 Flash | 30 | $0.062 |
| Mirror match game 2 · seed 7112 |
Gemini 3 Flash vs GPT-5 mini Gemini 3 Flash played the opponent's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Gemini 3 Flash | 36 | $0.068 |
| Mirror match game 1 · seed 7131 |
Gemini 3 Flash vs Jev Gemini 3 Flash played the player's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Gemini 3 Flash | 45 | $0.051 |
| Mirror match game 2 · seed 7132 |
Gemini 3 Flash vs Jev Gemini 3 Flash played the opponent's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Jev | 30 | $0.023 |
| Mirror match game 1 · seed 7171 |
GPT-5 mini vs Haiku 4.5 GPT-5 mini played the player's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
GPT-5 mini | 35 | $0.12 |
| Mirror match game 2 · seed 7172 |
GPT-5 mini vs Haiku 4.5 GPT-5 mini played the opponent's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Haiku 4.5 | 40 | $0.14 |
| Mirror match game 1 · seed 7181 |
GPT-5 mini vs Jev GPT-5 mini played the player's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
GPT-5 mini | 43 | $0.055 |
| Mirror match game 2 · seed 7182 |
GPT-5 mini vs Jev GPT-5 mini played the opponent's side |
Swampert, Salamence, Gardevoir Swampert, Salamence, Gardevoir |
Jev | 29 | $0.043 |
| Underdog game 1 · seed 7291 |
Haiku 4.5 vs Jev Haiku 4.5 played the player's side |
Sharpedo, Machoke, Tangela Glalie, Snorlax, Cacturne |
Jev | 16 | $0.053 |
| Underdog game 2 · seed 7292 |
Haiku 4.5 vs Jev Haiku 4.5 played the opponent's side |
Lugia, Snorlax, Sharpedo Machop, Machoke, Geodude |
Haiku 4.5 | 16 | $0.046 |
| Underdog game 1 · seed 7251 |
DeepSeek V4 Flash vs Haiku 4.5 DeepSeek V4 Flash played the player's side |
Lugia, Machoke, Sharpedo Geodude, Glalie, Cacturne |
Haiku 4.5 | 50 | $0.08 |
| Underdog game 2 · seed 7252 |
DeepSeek V4 Flash vs Haiku 4.5 DeepSeek V4 Flash played the opponent's side |
Sharpedo, Cacturne, Glalie Lugia, Machoke, Tangela |
Haiku 4.5 | 19 | $0.062 |
| Underdog game 1 · seed 7241 |
DeepSeek V4 Flash vs GPT-5 mini DeepSeek V4 Flash played the player's side |
Lugia, Machoke, Cacturne Sharpedo, Glalie, Tangela |
DeepSeek V4 Flash | 33 | $0.047 |
| Underdog game 2 · seed 7242 |
DeepSeek V4 Flash vs GPT-5 mini DeepSeek V4 Flash played the opponent's side |
Sharpedo, Glalie, Snorlax Lugia, Machoke, Tangela |
GPT-5 mini | 39 | $0.051 |
| Underdog game 1 · seed 7261 |
DeepSeek V4 Flash vs Jev DeepSeek V4 Flash played the player's side |
Lugia, Machoke, Snorlax Glalie, Sharpedo, Cacturne |
DeepSeek V4 Flash | 26 | $0.005 |
| Underdog game 2 · seed 7262 |
DeepSeek V4 Flash vs Jev DeepSeek V4 Flash played the opponent's side |
Lugia, Geodude, Ursaring Machop, Machoke, Snorlax |
Jev | 30 | $0.004 |
| Underdog game 1 · seed 7221 |
Gemini 3 Flash vs Haiku 4.5 Gemini 3 Flash played the player's side |
Lugia, Snorlax, Machoke Sharpedo, Geodude, Glalie |
Gemini 3 Flash | 30 | $0.11 |
| Underdog game 2 · seed 7222 |
Gemini 3 Flash vs Haiku 4.5 Gemini 3 Flash played the opponent's side |
Lugia, Machoke, Sharpedo Snorlax, Geodude, Glalie |
Haiku 4.5 | 41 | $0.14 |
| Underdog game 1 · seed 7201 |
Gemini 3 Flash vs DeepSeek V4 Flash Gemini 3 Flash played the player's side |
Lugia, Snorlax, Machoke Sharpedo, Geodude, Glalie |
Gemini 3 Flash | 26 | $0.028 |
| Underdog game 2 · seed 7202 |
Gemini 3 Flash vs DeepSeek V4 Flash Gemini 3 Flash played the opponent's side |
Sharpedo, Snorlax, Glalie Lugia, Machoke, Cascoon |
Gemini 3 Flash DeepSeek V4 Flash forfeited | 33 | $0.045 |
| Underdog game 1 · seed 7211 |
Gemini 3 Flash vs GPT-5 mini Gemini 3 Flash played the player's side |
Lugia, Machoke, Cacturne Snorlax, Sharpedo, Machop |
Gemini 3 Flash | 41 | $0.10 |
| Underdog game 2 · seed 7212 |
Gemini 3 Flash vs GPT-5 mini Gemini 3 Flash played the opponent's side |
Lugia, Machoke, Cacturne Snorlax, Geodude, Sharpedo |
GPT-5 mini | 35 | $0.095 |
| Underdog game 1 · seed 7231 |
Gemini 3 Flash vs Jev Gemini 3 Flash played the player's side |
Lugia, Machoke, Snorlax Sharpedo, Glalie, Machop |
Gemini 3 Flash | 32 | $0.041 |
| Underdog game 2 · seed 7232 |
Gemini 3 Flash vs Jev Gemini 3 Flash played the opponent's side |
Lugia, Snorlax, Sharpedo Machop, Machoke, Geodude |
Gemini 3 Flash | 16 | $0.014 |
| Underdog game 1 · seed 7271 |
GPT-5 mini vs Haiku 4.5 GPT-5 mini played the player's side |
Lugia, Tangela, Machoke Geodude, Sharpedo, Snorlax |
Haiku 4.5 | 159 | $0.26 |
| Underdog game 2 · seed 7272 |
GPT-5 mini vs Haiku 4.5 GPT-5 mini played the opponent's side |
Snorlax, Sharpedo, Glalie Lugia, Tangela, Machoke |
Haiku 4.5 | 20 | $0.087 |
| Underdog game 1 · seed 7281 |
GPT-5 mini vs Jev GPT-5 mini played the player's side |
Lugia, Machoke, Tangela Sharpedo, Glalie, Cacturne |
Jev | 30 | $0.039 |
| Underdog game 2 · seed 7282 |
GPT-5 mini vs Jev GPT-5 mini played the opponent's side |
Lugia, Tangela, Snorlax Machop, Machoke, Sharpedo |
GPT-5 mini | 24 | $0.034 |
Method
How the bench works
- Harness
- A list of legal actions for every model: code lists the legal actions (moves with type, power and a damage estimate; switches with matchups) and the model returns a choice. Chat models answer with a one-line reason; Jev returns probabilities.
- Sides
- One model plays the player's side, the other the trainer's (the game's own opponent controller, driven over a socket). Sides swap every game.
- What both sides see
- Both sides see the other's active mon's moves and stats (the game exposes them); symmetric.
- Not covered
- Singles only. No items. Draft facts are name and types. Cheap models only; a frontier model would cost 10–50× more per battle.
- Settings
- Each provider's defaults through OpenRouter; host not logged. Jev through TypeSafe.
- Harness
- Battles ran on 8 harness versions as bugs were found and fixed, so this is not a single-version controlled comparison. Each battle page names its version. 143a2b7ef+local changes (harness.diff); 20862524a+local changes (harness.diff); 2159ee4a4+local changes (harness.diff); 276b4e24d+local changes (harness.diff); 54816309a+local changes (harness.diff); 7b793a5ec+local changes (harness.diff); 9398de01c+local changes (harness.diff); abcf1ef04+local changes (harness.diff)
Data: bench v2 JSON. Frames on the battle pages are a few still images from our own runs.