Chess Play Quality Index

The definition of the play quality index is inspired by the work of Matej Guid and Ivan Bratko on analyzing quality of play in chess games. For a comprehensive overview and technical discussion on this topic one can read their paper Using Heuristic-Search Based Engines for Estimating Human Skill at Chess, 2011.

In short, the method compares the moves which were played in a chess game with the ones that a strong chess engine would play. The average difference of the evaluation of the moves played with the ones suggested by the engine is used as the indicator of how "good" the game was played. More detailed, yet short description of the method is the following:
  • The analysis of each game starts at move 12 (Thus opening moves that might just come out of theory are not evaluated).
  • The chess engine evaluates the best moves (according to the computer) and the moves played by the player.>
  • All engine's evaluations are obtained at the same depth of search.
  • The score is then the average difference between evaluations of the best moves and the moves played.
  • If the player's mistake (as seen by the engine) at particular move is greater than 3.00, the score for this particular move becomes 300 "centipawns" (to avoid unreasonably high penalties for gross mistakes).
  • Moves where both the move played and the move suggested by the computer had an evaluation outside the interval [-2.00, 2.00], are discarded. (In clearly won positions players are tempted to play a simple safe move instead of a stronger, but risky one. Such "inferior" moves are, from a practical viewpoint, perfectly justified. Similarly, in lost positions players sometimes deliberately play an objectively worse move.)
  • Once the average difference d (in "centipawns", i.e. 1/100-th of a pawn advantage) is computed, we calculate the player quality index q as q = 100 - d. A negative number q would be rounded to zero. Thus, the final player quality index in the analyzed game will be in the range of 0 to 100, where higher number indicates better play.
  • The overall quality index for a given chess player is the average quality of play from all his analysed games in the last 2 years.
The scores obtained only measure the average differences between the players' choices of move and the computer's choice. Several studies have shown that these scores that are relative to the chess engine used have good chances to produce sensible rankings of the players.
This article will be continiously extended with further examples that demonstrate the method and the results it produces for notable chess players and tournaments, as well as further technical details regarding the calculations we carried out.

Correlation between ELO and Play Quality Index

As our calculations are progresing (as of January 2016, we have fully analyzed over 100,000 chess games) we can already make some observations. One is that the play quality index of the players correlates with players' ELO ratings, even only if as little as just few games per player are taken into account. While exact statistical analysis still has to be done, the practical result we expect out of this finding is that we can estimate player strenght and performance in tournaments much faster (accurately) than ELO formulas would when only few games of a player are available. The graph below shows the quality of play index (Scaled, so that it can be easily correlated with the ELO rating graph). The QoP index graph is very closely correlating with ELO rating for the range of players that has at least 10 games analyzed. At the tail (for players with ELO rating smaller than 2300) the correlation still exists but the variance is higher. This is basically because much less players and their games have yet been analyzed in those ranges.