Does it make sense to update the Trueskill score of ranked players when someone submits his bot?
Consider this extrem case: I am in gold league and I submit a bot which will be promoted to legend. Does it make sense to update any score of the gold league (I won’t even be part of that league)?
Why do I ask this question? When someone submits his code several times and is ranked around you, your last battles against him can be about 10% or 20% of all battles used to set your original rank. So your rank becomes a rank relative to one player instead of depending on a set of players in the same area. If you are strong against this player, you go up in the rankings (for example, you can go to the upper league without fighting the boss). If you are weak against this player, you go down in the rankings (but you can get your initial ranking back if you submit your code again - which proves that your ranking shouldn’t have been changed).
When you play a game in multiplayer mode, you just redesign the rankings depending on people which are strong or weak against you. In real life (where TrueSkill is used), I guess there is no such issue because:
- It is unusual that a player resets his skills. A human player always progresses continuously and his skill only changes slightly. While a bot restarts from an unknown skill each time it is submitted.
- Also if a strong human player suddendly shows up and defeats you several times, it means that you are an active player and you will immediately balance your score by doing other game sessions against other players. On Codingame, the ranked bots are not active. If an active bot battles with them, they won’t balance their ranking with other players in the same area.
Here are some ideas to improve the rankins (I don’t pretend to suggest solutions - this is a complex issue and I don’t know what the best way to fix it could be) (some suggestions are not related to the issue reported above but I take this opportunity to mention other ideas which could improve the rankins in general) (numbers are not priorities, they are here to help to refer to them in comments):
- Don’t update the score of ranked players.
- Trust the initial ten matches. This suggestion is to avoid to modify the rankings between 200th-100th positions if the player has the level of the 50th player. If the estimated rank is optimistic, the player will go quickly down. If it is pessimistic, it will be the same situation as currently. Currently we almost always start from the bottom of the rankings whatever strong is the bot. Probably because you reduce the initial Trueskill score by a conservative value which is too large in my opinion.
- Avoid the confidence to converge too fast. During the ranking process, the score of ranked players should be updated very slightly until the rank of the submitting player becomes accurate.
- At the end of the ranking process, battle with players close to the current rank. I observed that you can battle a player which is ten ranks below and if you lose (which can happen randomly) you can drop several positions on your last battle (probably the confidence decreases but the ranking process stops whatever the confidence is).
- Continue the ranking process until reaching a given confidence.