About the new datasets for WinPrediction

Hi, I think it does not matter whether or not these game positions were from real human players, grandmasters, or even from the TCEC. Given a board position and which side’s turn it is, there is a clear unique evaluation that Stockfish 12+ will give, which is the evaluation assuming best play from both sides.

Now, in such positions, when giving the win prediction, we have to assume best play from both the side. We cannot assume human play because it’s irregular. A human play can be from a 1200 ELO player or a 2100 ELO player, and we have no way to account for that. Even a 2100 ELO player can have a bad day and play with a drop of 100 points in performance rating.

Now that we have established that there is one unique answer, we come back to the above pictured position - and similarly in another position on this post - to state that we have contradictory information in the dataset (against what we get from Stockfish evaluating the position). And this is not rare. For the first 100 training samples we observed 20 of them with opposite win predictions. Even if we assume our OCR is wrong on half of them, that’s still a 10% error rate in the training dataset.

Moreover, another issue is that not all positions are few moves before checkmate, as the problem statement says on the main page. Several positions are already mated, where there’s no sense of giving whose turn it is. On the other hand, several positions are far from mated, as you can see in the linked post, the evaluation is a meagre approx +3. However, any position near checkmate will ceratinly have a \pm Mx evaluation from stockfish, which means mate in x moves by either white or black.

Let me know if any part is unclear, I will re-explain. But I hope - if the dataset is revised once again - these issues are taken care of, because as it stands, it is almost impossible to submit a better score if we follow standard Chess evaluation metrics.

View post 1