Hello,
I noticed a strange behavior about the last challenge ranking (Back to the code). You might want to look into it as i think there could be an underlying bug.
A few hours before the challenge dead line, I submitted my final code. Then I slowly moved up in the ranking and stabilized around the 20th position.
Then, when everybody was 100% completed (I was 19th), you relaunched in additional batch of games. I instantly droped to the 45th position and stayed there until the end. My final ranking was 45th. I tought, woohoo, there has been some shadow players with good code that submitted very late and made it to the top at the last moment, good for them.
But then, the game was moved to the multiplayer category, and I find myself being again 19th ! Over the days my ranking slowly evolves, sometimes I’m 21th, sometimes 20th,… but I’ve never seen it above 23th.
To be clear, I don’t care about the ranking too much as I had missed the T-shirt anyway
But you might want to have a look and understand what happend, because it smells weird.
I fully understand your frustration but the reason is the following: we can’t generate all the matches with everybody with all possible starting positions (just compute the binomial coefficient or 4 among 2000). So we use a system (trueskill) that basically works with Bayesian inference, meaning probabilities. If during the last games you were really lucky, you could be over-rated. It’s not perfect, in particular for games where there are a lot of players with a similar ranking. The difference between 19th and 45th is less than 1.3%.
Thanks for the reply and the explanation.
But still…
Even if a ranking is an estimation of the true impossible to compute ranking, you would expect a coherent behavior. After 200 games have been played, the information brought in by the 201st game is relatively very small compared to the information already gathered. So at this point the ranking should move very slowly.
Here I say something like :
submit -> 5000e -> 3500e --> 2000e -> 1500e -> (going up faster and then slowly …) -> 200 -> 178 -> …(slowly going up) -> 100e -> (stable phase) -> 24e -> 21e -> 19e -> 20e -> 19e -> 22e -> 23e -> 20e -> everybody is at 100% !
Final batch of games is launched -> 20e -> 45e -> 46e -> 44e -> 45e -> 45e -> the end !
After all the games played and the weight of the previous games how could the ranking jump instantly with the start of the new batch ?
Why would I stick around the 45th position if previously from this ranking I would slowly reach the 20th area ?
Why when submiting to the multiplayer game afterwards I would again stabilize around 20 ?
maybe after everybody was at 100%, you started the final batch from a fresh start, meaning without taking into account the weight of the previous games. That could explain a big move. But then it would not make sense as the idea was to refine the estimation, not to start a new one…
maybe it’s not the ranking engine fault, and because of the server load I timed out a few games in a row and droped…
I’m not frustrated by this ranking at all (it’s already a while ago !) , I’m just trying to help as I geniously think I witnessed something weird. If you kept the logs you might want to look into it for the next challenge.
