PEGASUS EVAL / RESEARCH NOTE
Pattern IQ: What We Observed
Seven models. Fifty deterministic pattern-reasoning questions. A closer look at scores, shared failures, disagreement, and why a failed grade does not necessarily mean failed reasoning.
We evaluated seven models on a 50-question deterministic pattern-reasoning benchmark covering numeric sequences, letter patterns, grids, symbolic relationships, and mixed rules.
GPT-5.6 solved all 50 questions. Among the six comparison models, Qwen 32B AWQ achieved the highest observed score at 44/50 (88%), followed by GPT-4o-mini at 40/50 (80%).
These results describe performance on this benchmark only. They are not measurements of general intelligence or overall model capability.
| Model | Score | Accuracy |
|---|---|---|
| GPT-5.6 | 50/50 | 100% |
| Qwen 32B AWQ | 44/50 | 88% |
| GPT-4o-mini | 40/50 | 80% |
| Qwen 2.5 14B | 39/50 | 78% |
| Gemma 2 9B IT | 38/50 | 76% |
| Qwen 2.5 14B AWQ | 36/50 | 72% |
| Phi-3 Medium | 33/50 | 66% |
The failures were more interesting than the ranking
Across the six comparison models, 28 questions were solved by every model. The remaining 22 produced at least one deterministic failure.
Three questions were failed by all six comparison models. But shared failure did not imply a shared wrong answer. On one grid problem, for example, six models produced four different incorrect answers: 19, 20, 23, and 29.
Other questions showed striking convergence. On a letter-sequence problem whose correct answer was V, four of six models independently returned U. On a two-dimensional grid problem, three models returned the correct answer 8, while the other three converged on 10.
A deterministic failure is evidence—not an explanation
Human review revealed an important distinction.
One model decoded:
3, 1, 20, 20, 1
C, A, T, T, A
The expected answer was CATTA, so the deterministic evaluator marked the response as a failure. The grade is correct under the defined response contract—but the underlying decoding was also correct.
This is why Pegasus Eval preserves the original response, evaluation rule, model metadata, and evidence rather than reducing evaluation to a leaderboard.
For selected failures, we also used GPT-5.6 to independently derive the correct rule and propose possible mechanisms that could produce the observed wrong answers. These explanations are explicitly treated as hypotheses: final answers can support a diagnostic theory, but they cannot reveal another model's hidden reasoning.
Evaluation tells us what happened. Investigation begins with why.