Simon’s LLM Ranking
Methodology Sources GitHub
Formula v4 · capability-first

Multi-dimensional LLM ranking

A weighted composite of human preference, automated benchmarks, pricing efficiency, multimodal coverage, and context length — scored from a single canonical dataset.

Arena 30% AI Index 30% Price 15% Multimodal 15% Context 10%
⌕
{{ filtered.length }} / {{ ranked.length }} models

Failed to load data/models.json — {{ err }}

Loading models…

Methodology

Scores are computed client-side from data/models.json

{{ r.t }}

{{ r.w }}

{{ r.d }}

{{ t.l }}

How the ranking works

Coverage model, data provenance, and how to read each row

1. Two leaderboards, not always unanimous

Each model is scored on two independent measurements: an Arena rating and an Artificial Analysis Intelligence Index. These boards do not always agree on which models exist. A model can be listed on both, listed on only one, or drop off one board after having been listed before. Requiring both boards to agree for a model to appear would silently delete legitimately ranked models — that is what this scheme replaces. A model is now ranked whenever it has an Arena rating, and its coverage is declared explicitly instead of being decided by deletion.

2. Filling a missing AI Index (k-nearest-neighbour imputation)

When a model has an Arena rating but no published AI Index, it borrows an estimate from models that sit close to it on the Arena scale:

  1. Collect every model that has both an Arena rating and a measured AI Index.
  2. Keep those whose Arena rating is within ±20 points of the target model.
  3. Its AI Index becomes the mean of that neighbourhood.
  4. If the neighbourhood is empty, nothing is guessed — that dimension is left blank.

Imputation only runs in one direction. Because Arena is a directly measured rating, models within a narrow Arena band are genuinely comparable, and their AI Index is the best available stand-in. The reverse — guessing an Arena rating from AI Index — is deliberately not done: the two boards measure different quantities, so their neighbourhoods are not interchangeable.

3. Missing dimensions are dropped, never zeroed

A model without an AI Index is not penalised to zero. That dimension is removed from the denominator and the remaining weights are re-normalised so they still sum to 100%. A missing score therefore costs a model its evidence, not marks.

4. Multimodal coverage is derived, not assigned

The multimodal dimension is not a hand-maintained label. It is recomputed on every run from the model's declared input modalities (image, file, video, audio) as published by OpenRouter, by counting how many non-text modalities the model accepts:

  • 0 (text only) → 50
  • 1 → 70
  • 2 → 85
  • 3 → 95
  • 4 (all of image, file, video, audio) → 100

Hand-assigned tags tended to drift: two models advertising the same modalities could carry very different values, which made the 15% weight impossible to compare fairly. Deriving the tier from a published list removes that inconsistency. A model with no OpenRouter listing keeps its previous tier, since there is no modality data to derive from.

5. The weighting

Capability first. Arena 30% · AI Index 30% · Price 15% · Multimodal 15% · Context 10%. Price, multimodal capability and context are always available, so they never need imputation.

6. Reading the badges

Every row states how much of it is actually measured. The overall total is only as trustworthy as its weakest input.

Verified

Both scores measured

The Arena rating and the AI Index are both live values read from their boards on the most recent run. Nothing on this row is inferred. These rows are directly comparable with one another.

Estimated

At least one score inferred

The AI Index was not published for this model, so it was imputed from Arena-nearby models (±20 points) and is shown with a leading ~. The total is provisional: it can be read as a reasonable placement, but it is not a measurement and should not be compared too finely against a Verified row. The tooltip names how many models contributed.

Stale

A score carried over

No longer listed on one or both boards, or no comparable neighbour existed to impute from. Its previous values are kept so it does not silently disappear, but the total is struck through and the row is dimmed. Treat it as an archived entry, not a current ranking. Shown by default; use “Hide stale” above the table to filter them out.

7. Why a model may be missing an AI Index column entry

A dashed — in the AI column means both that the index is unpublished and that no model within ±20 Arena points had one to borrow from. Rather than invent a number, the column is left empty and the row is labelled Stale.

Data sources

Canonical JSON · external leaderboards · price audit