Ranking calculations update

Platform Evolution
#10

I’m not convinced by the “game of the month”. We started with the idea to make the leaderboard more stable and now we have a suggestion that would scramble it every month (same with the last 3 contest: if you miss one, you will instantly drop some ranks).

I scraped some leaderboards (without codegolf and clash of code) to give an idea how the proposals would look like:
Marchete option A (rank all multis, max multiplayer score=60k): pastebin
Marchete option B (best 20 multis, 6 optims): pastebin
best 5 multis, 2 optims, bonus for most recent 3 contests, max multiplayer score = 15k: pastebin
best 5 multis, 2 optims, no bonus for recent contests: pastebin

The question is: shall the leaderboard show the most skilled players at the top, or those who had the most time to write a decent bot for all arenas (possibly without ever hitting top10 in a single game)?

5 Likes
#11

Clash of code should be moved in its own global category and be completely detached from the general leaderboard.
Perhaps the sprints (2-4h contests) should be added to this category though. Last one had around 700 participants and NO impact on contest points for those who participated in 3 week long contests.

Speaking of new players. If best 3 contests reward stays the way it is then new players might be forever prevented from obtaining maximum points on contests. Some of the older contests had 3000+ participants. You need a completely different way to scale the points perhaps.

For example rank 100 on the final leaderboard in a contest with 3600 participants yields more than 2800 points. You won’t reach this amount even if you place #1 in a contest with 2500 participants and the players only got better since then, but getting into top 100 in any contest means less CP regardless.
This means that even if someone was ranked #1 for the past 8 contests in a row they would be ranked lower than someone who got a good rank in 3 contests with 3000+ participants.

I also like the idea of having the last 3 contests give some bonus CP.

Continuing with the new player experience, imagine you had to write up 30+ bots and put some serious effort to compete on the general leaderboard. Exactly, most will prefer to not even bother. And as the multis pile up they will be disadvantaged on the ladder. Even 20 multiplayers as suggested seems like an awful lot.

I suggest everyone can pick something like 5-10 favorite multiplayers to be ranked on.
The multiplayer gained CP should scaled so that any 2 multiplayer games will reward the same amount of maximum CP.

Perhaps reward more CP for participating in the last (maybe) 3 multiplayers, to encourage remaining active.

I’ll finish this part with an example. So let’s say we have a codingamer “Hi_i_am_new” with the following stastics:
He picks 5 favorite games to count towards total score on the general leaderboard: CSB, Mean Max, CR, Fantastic Bits, Poker Chip Race
If each of those reward a maximum of 5000 points (if we stick to that limit) the total would be 25000.
The last 3 multiplayers are LOCM, CR and Kutulu, which would reward another 15000 points at most.

So “Hi_i_am_new” gets 40k CP if he is #1 in 7 multiplayers. That’s far better than having to write 20-30 bots; a very mundane task. Maybe the favorite multiplayers could reward less maximum points, but this is just one of the many little details that needs to be adjusted.

If you still want to take into account all multiplayers then anything beyond the 10th bot you have to write should give a much lower maximum CP.

Don’t really like code golf, but don’t care if it stays either as it doesn’t give that many extra points.

Optimization puzzles are fine the way they are imo, apart from the max number of puzzles that are counted towards the final score. If the number keeps growing then new players will end up facing the same daunting task they face in the multiplayer game section.

#12

I think the proposed formula is good. I agree that the BASE should be higher for some categories.

I think what is difficult here is the fact that we have two kinds of category, the one with time limit as contests and the multiplayer (past events) with no time limit. Here what I propose for the ranking:

  • Time Limit: Taking the Top N for each category is a way to reward the presence, I believe a number between 3 to 5 is good. But it’s important that the window for the selection of the Top N moves with time (I propose 1 year). Like this even if someone miss one or two contests it should not have much impact on his ranking. The sliding window selection is what will make it fair for new players, as even if eventually a new player will have the same selection as an old player so will be able to fairly compete with everyone.

  • Unlimited time: As there is not time limit and this category is growing over time, I would say that the point should be at least 1/3 rewarding than the contest. We could apply the same logic, Top N * 3 multiplayer and we divide the reward by 3.

With this approach, any new player should be able to achieve a ranking showing his skills in N contests only assuming he post his solutions on Multiplayer and solve 2N more Multiplayer.

For the established players, Multiplayer will not be a defining factor as everyone will have time to post his solutions, the only really thing that will matter will be the contests and the ranking in contest due to the sliding window.

I believe this should achieve a stable leaderboard but still leave room for any new player to go to the top. Past player who top playing for a year, will go down in ranking (only contest as multiplayer point will not be lost) and will be able to climb back if they participate in another contest.

EDIT:
There is no sliding window on the unlimited time category
Each category can have subcategory, the thing that is important is the weight of each of them.

Let’s say we want to have 50% of point on Time Limit, and 50% on unlimited time.
Each category can be further split in subcategories

2 Likes
#13

I think this leaderboard is a clear winner among the one proposed.
Skilled players get rewarded whether they are new, e.g. MSmits, or veterans, e.g. Recar.
Meanwhile, those who are known for copying other people’s code get ranked less high.

Just for the sake of having the best possible base formula, I’d like to change this:

(BASE*min(N/500,1))^​((N-​C+​1)/​N)

To this:

(BASE*min(sqrt(N)/sqrt(500),1))^​((N-​C+​1)/​N)

This will cause people who win a contest with less than 500 participants to be punished less severely.

1 Like
#14

I think that part should be addressed separately. Yes, there are some copy-pasters with a high ranking, but it’s not the job of the leaderboard to filter them (partially by giving twice the points for contests as for multiplayers in that pastebin) - their stolen bots should just be removed to really solve the root cause.

3 Likes
#15

Copy and paste has already been addressed:


It could benefit coders if it’s well used.

And about the best 5 multis, that will only make multi scoring useless for all kind of players. Copypasters will have a much much easier task, finding 5 is easier than finding ALL. On the other hand it simplifies a lot for tops challenger players. You just need to excel in 3 challenges, then reuse it on multi and you just need two more multi bots to have the max score (that they can C&P if they want). Having 3 good challenge bots it’s also 3 free multi bots.

Do you think all these 4 players must have the same score in global leaderboard?
https://cgmulti.azke.fr/players?p=_Royale%20Agade%20eulerscheZahl%20Recar
I don’t think so, sorry. Max Score in leaderboards should reflect the total score of all possible games a new player can do (no matter if it’s “boring” or “take too long”). Don’t fit Max Score to your needs, but to all available options of multiplayer games, for the theorical supercoder (top in all games, no matter what section)

Multi scoring is just a percentage of the global score (maybe 40%). Having just 100% in multi won’t get you to top leaderboard.

It’s just a matter of adjusting percentages of each section (multi, challenges, etc). You tend to promote challenges (fast, time-limited coding), but multis gives the option to test more things, with another pace. CSB multi bots are much much evolved than the challenge ones.

#16

Since it’s not strictly forbidden to copy someone else’s code, I’m going to be bold and say: yes, it is the job of the leaderboard to filter them.

Do you think all these 4 players must have the same score in global leaderboard?

No, but the suggested leaderboard does not give them the same rank so I don’t see what the issue is?
But yes, I absolutely do think Recar is roughly on the level of the others. It should not be needed for a skilled player to dedicate his life to writing a quick legend bot for every single multi to get rewarded with a good rank.

#17

In my opinion :

Copy/pasters in multiplayers puzzles are removed from the leaderboards by CodinGame (when they take the time to make a little clean up :smiley: ). I don’t think the global leaderboard should adress that. This is a separated issue. And since CodinGame is against it, it should be ignored.

The CodinGame goal should be to encourage people to stay on CodinGame (not all the day of course, but in a long term fashion). With that idea in mind, i really like the Marchete’s proposal to gives points for the last 3 contests. But i also think that every multiplayers puzzles (multi, code golf, optims …) must rewards points. I know that for a new player, create an AI for every multiplayer puzzles will be very hard. But it’s codingame goal to keep people here. So you have to find something to keep people here.

The Marchete’s OPTION A seems fine to me. Maybe tweak a little the bases.

For clash of code : i’m very biased because i don’t like CoC. But CoC is the only leaderboard where you lose points every day if you don’t do CoC every day … It’s very harsh and very time consuming. When you are top 10 in a multiplayer puzzles, you’ll stay in the top 10 for a very long time. Maybe one day you’ll be out of this top 10 but you’ll have to wait at least one year for that. That’s why i think CoC should not reward points in the global leaderboard. Just keep the CoC leaderboard separated.

In the end the question is : What the global leaderboard should represents ? If we only gives points for 3 contests and 5 multiplayers the global leaderboard will just represents your capacity to win a contest and to copy/paste your AI in the multiplayer puzzle.

If we want that the global leaderboard represents you overall level on codingame, you have to includes all availables multiplayers puzzles (we keep 3 contests becauses contests are not available anymore).

3 Likes
#18

My only problem with the current formula is how much the number of participant influence the score. Seems like the new formula proposed by Thibauld will address that.

Some people proposed to only keep the best X multiplayer, I don’t agree with that at all. It is a global leaderboard, it should represent how you rank globally. So I feel all multiplayer should still worth points and I feel since some games gives so much point while others are meaningless, it mostly represent how you did in those games and not in all games.

I like heuristic, I find it more fun than a search algo, but that’s not a popular opinion in here, meaning games like CodeBuster, one of my favorite, doesn’t worth much and I don’t have any reason to improve on that game and it’s not really appealing for new player to tackle it.

Same goes for contest, I gave up in the last week of Locm, in part cause I knew I wouldnt be in top 20 and with the amount of players, it would never be in my top 3.

I agree with the top 3 in contest, since it’s a 1 time thing and new player can’t replay them, but I feel the way the top 3 is determine should change to reflect how good you did. I get that more player = harder to be on top, but not to a scale of 5k players.

Anyway, I think this would motivate new and old players to keep improving and giving their best in new contests and new games.

3 Likes
#19

I do not agree with this. But i know many people will agree with you.

#20

Because statistically speaking it’s not true, and we talked about it in the previous thread How is the Coding Rank calculated?

player count is a matter of sample size on a statistical sample. Having more players in theory won’t move you a lot on your relative position. If I’m in the top 20% for 1000 players, I’ll be in the same 20% with a population of 20000 players (maybe with some confidence error
Sample size is the basics of most statistical analisys and quantitative research. This has a big background on research and real use cases in all fields. It’s not something I made up just for CG rankings.

#21

The idea of a sliding window is very important to even out the conditions under which everyone competes. Basically after 1 year everyone has the chance to go through exactly the same circumstances, same contests, same playerbase.

I would still exclude sprints or put them into a different category at least. Have sprints be a different category and not part of the contests.

Without the sliding window and by including all small contests (after you scale the formula to take into account 200 - 500 participant contests) you get a chaotic leaderboard which won’t have as much worth competing in. For example over time the amount of players with 3 contests where they achieved top 10 accumulates and takes away from the motivation to go all in on it. A seasonal ranking however brings everyone closer to a new start, a blank slate and is therefore more encouraging.

Participants who like the 10 days contests don’t necessarily like the few hour contests. Those might also not fit into everyone’s free time and counting them the same way as the 10 day contests only generates chaos, especially if the last 3 contests are to be considered for the contest rank calculation.

Alternatively there could be a seasonal leaderboard. Only what happened during the last selected period of time is displayed. Would very much prefer this as default leaderboard over the “hey someone got #1 in 3 contests that took place a decade ago” while today’s environment is completely different. Completely different competition and challenges. This is also more in favor of those who remain active.

#22

Guys don’t put me out of context haha.

I’m just saying I understand the current formula being something like: “You are better than more player, so you deserve more points”. But the number of player influencing the score is basically my only problem with the formula so I clearly don’t agree with that.

What I meant was on a really small scale, like 10 player, you could be lucky and be against only lazy coders who coded a dumb bot and gave up. So just a little bit more players and luck becomes quickly irrelevant and your relative rank wont change much between a let’s say 100 players versus 5k players.

The proposed 500 seems fair, small enough to include all games and big enough just so you don’t get millions of points for being in the first submitters of a new multiplayer on the first day.

#23

An annotation to Marchete’s Option A:

We will (soon?) have the option to create our own multiplayer arenas, see Vindinium. So it’s quite likely that there will be some arenas with less than 500 players one day. That will cause the leaderboard to drift towards contests rather than multiplayers as of now.
20/ALL is not an appropriate correction if we want to account for this.

2 Likes
#24

It’s true, aside from the gratuitous ad for Vindinium :laughing:. It would be better work with sum of 1st place scores

MaxScore(multi) / sum( f(C=1, game = multi))

In SQL you also can just get select multi,max(score) FROM nnn group by multi where … it’s “simple” to implement, or at least I know how to do all the calculations in SQL (some subqueries, but it shouldn’t be very slow).

#25

At the moment my opinion on (possibles) future community multiplayers puzzles is weird.

Don’t get me wrong, i’d love the possibility to create a multiplayer puzzle without the Community Contest step. But i’m unsure if this puzzles should reward codingpoints. I suppose CodinGame will decide on that point.

3 Likes
#26

Hey, just starting reading this post.

  • I do like the fact that the contest formula change because it was designed thinking that the number of contestant will always grow over time.

Quick question: What’s the benefit of having a ‘min(N, 500)’ and not just 500. With every game giving the same amout of points, people willing to get points will play either the one they perfer and/or the one with the less competition. It won’t change in the long term, but will add an incentive to the “low played” game :thinking:

#27

The benefit is because at the start of a new multiplayer puzzles we often have far less than 500 players. I suppose the goal is to avoid that a new a multiplayer puzzle reward 3000 points directly.

And maybe it’s here to prepare the field for future community multiplayer puzzles. But you are more informed than us.

#28

To avoid edge cases, like in very old challenges, and those are “final”, you can’t add more players. If we will have a single formula for all games, we must ensure that in any situation given points are fair. With only a handful of players you don’t have a representative sample of the real level of players. Maybe it’s only played by hardcore CGers so you end up in a lower percentile than you deserve.

#29

Hello,

we had a internal discussion on this topic. We didn’t have time to discuss everything though.

  1. First, we like the new formula to give the same amount of points to games of same type: (BASE * min(N/500,1))^((N-C+1)/N)
  2. We also like the idea of seasonal ranking for contests. From January to December, only top 3 contests taken into account. We even thought it could be the main leaderboard of CodinGame. (with possibility to check the leaderboards of previous years)
  3. Then, what to do with multi-players, optims, code golf… we’re not sure yet.
6 Likes