What is the purpose of splitting a league that has same rules into multiple leagues?
I understand the benefits of the sandbox wood leagues for warn-up and of leagues where we have to achieve a level to unlock new features, more “actions”. I wonder what are benefits of preventing players playing same rules from freely competing with each other during the whole contest.
I was automatically promoted today when silver was opened, so I am not jealous. But I get fewer people to compete against and already now, as always, there are players in bronze who are better than those in silver, some in bronze will be likely better than some in gold.
It would be more fun for everyone to have a flat leaderboard with a single league (after game rules stop changing) until the end of the challenge. Otherwise some get stuck at the top of a league and small improvements are insufficient to beat the boss, but they would be more inspired to keep up when seeing small hikes in ranking. Also otherwise some lose interest when they are at the bottom of a league and keep more or less same ranking even with a stale bot, they would be more inspired to tinker with bot if they saw their ranking drop every day.
Then when the final leaderboard is established in the final re-runs after the contest end, you would assign grading as legend, gold, silver, etc. based on the final ranking.
Multiple leagues are used to group people according to their level.
If there are too many differences in level in a single league, the ranking at the bottom of the league is too random, and depends on who was unlucky enough to face the best.
Let’s do an A/B experiment to know for sure. For just one event, let’s have a single league where everyone compete together, of course after completing the sandbox leagues Wood-1, Wood-2 for ramping up features. You can entitle the top 50, or however many are appropriate, as “Legend” after the final re-runs.
Then we know from experience whether we like this set-up.
The legend league’s main benefit is that the top bots don’t spend most of their games playing against opponents who would lose 95%+ of the games; even with ~60 people like the last time, the rankings we got from submits depended more on random unavoidable losses against low-rated opponents during the first 50% of the games than on bot strength; in spring 22 with 300+ people submits were completely pointless. If this happened, I would never submit a bot until the last few hours of the contest.
If anything, sometimes it’s too easy to get into legend because of gold boss’s rating variance.
I see that I missed something. I thought that the final ranking was based on the final re-runs and was reliable.
Can it happen, at least theoretically as an edge case, that bot “A” wins a contest (i.e gets the first position on the top) despite there’s a bot “B” that would win against “A” all the times, because the ranking is based on games outcome with different sets of opponents for “A” and “B”?
This itself might be a good challenge idea for a contest:
“Create algorithm for bot matching for quickest convergence of the leaderboard, i.e resulting in a leaderboard state where for any two bots at positions i, j, i < j, the bot at i consistently wins against bot at j.”
Such algorithm might need to internally cluster the bots somehow for selecting relevant opponents, but without silos.
That is only during the contest. Top N (50? 100?) shall be selected by a sorting the top, say 2 * N, bots with “<” defined as most wins of K games with random seed. And for selecting K, start by 8 and double it until increasing K does not affect the outcome of sorting.
The Legend league re-runs at the end I’ve witnessed were ‘everybody plays everybody else n times’ and that’s one of the benefits of having leagues. Exactly how that translates into rankings and how important the differing order of matches is I’m unsure. There’s some older discussions about this but I don’t think the process is documented anywhere.
A classic single-league Elo ranking doesn’t really work, because the thing being ranked - our code - changes dramatically as the contest progresses. That implies completely recalculating ranking on every submit, which is what happens (by resetting it and then playing a large amount of matches), but many more games would need to be played to achieve a halfway-stable ranking without leagues to ballpark an initial ranking. That’s more resources for CG and slower feedback for us.
I think there’s an important fun/psychological benefit too. If you’re not competing near the top then your goal is to improve ranking and also get to the next league and it’s very satisfying when you do. It’s sort of crude shiny dopamine hit but that’s why we’re here, right? Rather than learning from dry textbooks on our lonesome. [This aspect basically implies no demotions].
I don’t think it’s perfect. The ranking breaks down toward the top of leagues where promotion happens because the higher you go the more you’re denied matches against higher ranked opponents, as they ‘artificially’ disappear to the league above. It’s much easier to promote in the first batch when a new league opens. I wonder if this could be mitigated by adding some sort of super-bosses to each league which represent talent in the league above, or simply a handful of different but similarly ranked bosses and require outranking any one of them in order to promote.
I also dislike that you lose the ability to manually play bosses from lower leagues. Having a stable opponent to test against is very useful, and it just vanishes when you cross these arbitrary thresholds. In the current top league you don’t even get a replacement stable boss to test against. Fixing that would be easy.