Ranking v2 suggestion

Platform Evolution
#1

Hi, after reading your feedback about ranking calculation, I would like to suggest the following computation.

Feedback source:

The following suggestion is work in progress and can be commented (feedback greatly appreciated as always).

The base formula

First, everyone seems to agree with this formula:

(BASE * min(N/500,1))^((N- C+ 1)/ N)

But a post written by Jolindien got a lot of likes. It’s about rewarding the top of the ladder and the the low/mid ladder with more points. Based on the suggested formula, I did some tests, and the following formula seems quite fair:

0.5*(1000*min(N/500,1))^(((N- C+ 1)/ N)^0.5) + 0.5*(1000*min(N/500,1))^(((N- C+ 1)/ N)^10)

It gives a huge amount of points when climbing in the top 10 while still rewarding more everyone at the start (giving a nice feeling of progression).

Here is a comparison between the first formula (red line) and the new one (blue line)

Pasted%20Graphic 

The X-axis is the ranking (here over 500 players, but changing the number of players doesn’t influence the curves). The Y-axis is the number of points. As you can see more points are given at the beginning and a lot more at the end.

Doing so will greatly boost the amount of points gained by top players, giving them more ranking in all aspects.

Repartition

One of the current main problem is that multiplayer games are giving more and more points over contests and other categories. To solve that, we can compute a ranking for each category and then, apply the above formula to calculate final points. As we also apply the complex formula, someone who is in the top of the contest category will have a lot of points compared to someone who is medium in all categories.

Here is the suggested repartition:

  • contests (30%) (base 30 000)
  • multi (30%) (base 30 000)
  • opti (15%) (base 15 000)
  • clash of code (15%) (base 15 000)
  • code size (10%) (base 10 000)

Note that the repartition is an example and a preview using a second repartition can be found at the bottom of the post. Feel free to share your thoughts about the repartition :wink:

As every category is partitioned, the base for each game (except contest) can be 1000. Once each game has been computed, a ranking is computed and the formula is applied again with the base specified in the repartition.

Contest (30%)

top 3 contests are taken into account. Remember that a lot of players copy paste their solution in the multi category, so this category rewards a lot of points (and not just 30%).
Instead of taking into account only last year’s contests (as it was suggested), we’re thinking of giving more and more points for each new contest. For example, the first contest ever done gives 1000 points. Each new contest could give 5% more additional points (rounded to 10).

BASE will be:

Xmas Rush: 8150
A*Craft: 7760
Legends of Code and Magic (M): 7390
Legends of Code and Magic (S): 7040
Code of Kutulu: 6700
...
Genome Sequencing: 1160
CodinGame March 2013: 1100
Chuck Norris: 1050
CodinGame October 2012: 1000

As each new contest gives more points, it’s interesting to give it a try and reach a new “best score”.

Multi (30%)

There is a lot of debate (even internally) about this. Some of us prefer to keep the “X best scores”, other tend to prefer “Keep them all”. The progression feeling is important and we don’t want to tell our active users “let’s focus on 10 multi games and let’s stop playing the new ones”. This is why I would like to suggest to “Keep them all”. But as the formula gives a lot of points for the best players, there are multiple strategies to reach the top of this leaderboard: be the best on few games or good on lot of them, giving opportunities for each of us to climb this leaderboard. This is why the formula gives an exponent of 10: like this less active but better players will be rewarded a lot of points making them higher in the leaderboard.

Opti (15%)

all games are taken into account

Clash of Code (15%)

I know that this one is really controversial. But as it has been stated, it’s a big part of CodinGame and played by a lot of players. Nevertheless, we’re thinking of reducing the loss of points over time making Clash of Code more interesting for climbing the leaderboard.

Code Size (10%)

all games are taken in account.

As each category is partitioned, we will probably rework the leaderboard so that each category can be displayed easily letting each of us to focus on one category (contest for example) or all of them using the main leaderboard (which reward all activities in the platform).

I computed a simulation that you can discover here (only the first 10k players):

The second tab is the following repartition (which is more contest oriented) to compare:

contests (50%) (base 50 000)

  • multi (25%) (base 25 000)
  • opti (15%) (base 15 000)
  • clash of code (5%) (base 5 000)
  • code size (5%) (base 5 000)

Remember that once coding points have been calculated for each game, a ranking is computed for each category and the formula is then reapplied using the base specified in the repartition.

If you read this, congrats, that was a huge post.

What do you think about this?
What repartition do you prefer?

9 Likes
Ranking calculations update
#2

As a CodinGamer, I like the second repartition better.

  • Contest is the most fair environnement (time limited, …). it should be the most important part of the ranking.
  • Multi is a lot of work too and it’s perfect for newcommers to catch up between contest
  • Opti is ok-ish. Most of the time there is a finite solution and we just have to hardcoded it. CG should add a lot more validators.
  • Code Golf & Coc are just for fun. I’m ok to count them in the General ranking but it shouldn’t take a big part.

So, Contest > Multi > Opti > Code Golf > Coc

4 Likes
#3
#4

The second repartition seems better to me too. The top of a contest (or a multi puzzle) is really hard to reach, it must be rewarding.

I’m looking at the new ranks file preview. Rank 21 in CoC reward 11863 ? I know CG seems to like CoC very much but wow … It’s more than a single contest … it’s absurd …

4 Likes
#5

nop, you can’t compare points between categories as they are then partitioned. If you have 500000 in clash of code, it will be 5% for the final score (in the 2nd repartition). It can’t be compared with a contest.

Moreover remember to look at the second tab of the preview (in the second tab Clash of Code worth 5% of the final score).

#6

Here’s my two cents:
I like the new curve. I think it does make the most out of the previous discussion.
The contest BASE is weird at first, but it works in the end (seeing how @MSmits is #1).

Another thing: since the CG points are computed per sections (all contests/multi/… as one for the global leaderboard), it makes all scores really close to each other, because there are now 1M people in the formula, instead of a few thousands. I’m not sure this is the best.

And I guess now is the time to debate about the distribution.
The first one is problematic (as a codingamer). The obvious example being @reCurse, ranked #65 while being without doubt one of the best/main codingamer. I might be biaised, but I think the platform mostly revolves around the bot programming competitons, as contests or multiplayer. 15% to opti is questionable, there are only 5 opti games vs 20ish multiplayers games.
The second one is definitely more appealing, but this time I think 50% to contests is too much. @MSmits rank might prove me wrong, but I think it is too hard to catch up (especially since great contest AI counts double with the following multi).

So my first guess would be a 25/50/15/5/5. I wanting to give it a try, but not having the raw data made it hard.

4 Likes
#7

reCurse is good in contests (and in some multiplayer puzzles), but the global leaderboard is a … global leaderboard. So if you are not good everywhere, there’s no reason you are at the top in the global leaderboard.

Multiplayer puzzles rewarding twice a contest ?

#8

I really hate bragging but it’s going to be hard for me to debate the merits of these proposals without bringing up my results, so I sincerely apologize.

A formula that gives me rank 60 (or even 30) when I rank 5th place in contests and 7th in multis, which are by far the hardest categories to do well in CG… just because I didn’t game the hardcoded validators in optim or code golf (e.g. I’ve seen the validators shared in chat, how CvZ has been hardcoded to death, etc), or played CoC with some who may or may not have legitimately memorized the questions? I don’t know about that…

As for the “you must play the whole site to do well” argument, why not be entirely coherent and give rankings for all puzzles (including community) as well? If you disagree with that, then there is certainly room to disagree with the others and their relative importance. We have to be honest about the relative level of difficulty and effort involved.

All in all, if the current system needs to be changed, then I think the idea of a global leaderboard reuniting so many different things is misguided. It should be separate per topic and ask you which topic you care about seeing. If there are concerns with the displayed rank in the profile badge, just pick the highest one or let the user decide which to display. That would be the fairest option for all in my opinion.

Also, I completely agree with the change for the slope of scoring a game. Having top 10 or rank 1 is not nearly rewarded enough compared to the effort it takes getting there.

9 Likes
#9

  • Contest 70%
    Why not taking top 5 instead of top 3 by the way, there are so many of them that limiting it to 3 is a bit sad.
    I like the +5% to keep things fresh.
    Contest is the most competitive category and also the category having the biggest attention from the community (look at chat / forum during contests), so they should be rewarded accordingly.

  • Multi 30%
    All taken equally into account to keep it simple.

3 Likes
#10

Quick side note :

Optims are getting lots of undeserved hate imho. Yes most of the optims top is hardcoded, but that doesn’t make them less worthy. Knowing the validators just turns the game into an offline-problem (very similar to what Google Hashcode is all about). Different problem, same difficulty to reach #1.

As for the solutions for this particular issue, they already have been mentioned quite a few times already (make validators public OR add shitloads of validators).

End of the digression, you can keep fighting :slight_smile:

3 Likes
#11

Looks like the blue line drops too fast. Imo issue right now is, especially for low count games, that your points are very much irrelevant until you get around #50 or so.

I thought this was about encouraging participation and increasing new player retention.

I would rather keep the orange line and then on top of the regular CP, give a bonus to the top whatever range is supposed to represent the best.

But there needs to be more motivation all along the way to the top to do even minor improvements. Otherwise what exactly is the driving force towards participation?

Unless i got it all wrong and a higher count of regulars isn’t the actual goal.

The score inflation for contests seems interesting, but please don’t count sprints with the regular contests. We had a huge lag fest back then and 4 hours isn’t for everyone.

Might sound cynical, but perhaps it’s best to make the leaderboard modular and let everyone pick, mix and match whatever they like. Also allow adjusting of category based values by percentage of course. Then it will be like doing statistical models to predict the best on the ladder.

PS: if it’s only either of the options presented then my votes:

  • orange line good
  • 50% contest version
#12

As long as it isn’t playercount based, and it doesn’t drift in the future it will work
But I think I disagree in pretty much everything.

I’m not a big fan of exponentials, I just used it in previous formulas because it was already there.
It seems that being top1 will be orders of magnitude better than say top20, OK if you think that. But there is an inherent noise in rankings, and that huge exponential just worsen it. That noise was clear on a challenge where a legend bot was leaked, the same bot ranked like 30th or 70th, and it was the very same bot.

About challenges, adding 1.05^x just renders challenge scores useless in the long term. A good player with 3 top1 3 years ago is much worse than some current active player in top 30. I know that’s the intention, but it makes the scoreboard very temporary, and again, exponentials are picky, it’s 5% over a cumulative scoring, I can’t forsee how it reacts in scoring of players. I don’t really like that good players of past years are completely removed from leaderboards.

Multis are commonly depreciated, like in the second tab of the Excel. Being top 1 in multis gives the same score than being 586th in challenges… OK.
For some players challenges are too time limited, and they don’t let fully explore and enjoy the game. Maybe for you it’s “fair” as it’s time limited. But what if, I don’t know, not all players have the same spare time in a 10 days span? Sure, challenges scores based on a time limited fashion, but multis scores based on the fully development of bots and strategies.
If you take the current CSB 1st bot in Multi against the Challenge winner bot that match will be categorized as child abuse. Why is that better a challenge ranking over multi ranking? I don’t get it
In addition a 50/25/x/x/x just mean challenges are in reality 65%/70% of the global score (by passing the challenge bot to multi).

The other thing I don’t like is that scoring ignores the fact of being much better than others in one category. If it happens than a player is 1st by a big margin, this score gap is ignored. Scores per category are just for player placement.

If I should pick a partitioning, I’d go more for 30/45/10/10/5, or Azkella’s proposal of 25/50/15/5/5 to see how it behaves. I think challenge top players will have at least 50/60% of the multi score directly without doing anything (as challenge bots can be passed to multi).
CoC, codegolf and optims are interesting games too, 25% of the total score could be good.

4 Likes
#13

Do you have a moment to talk about our savior “just pay a short domain name and win the thor code golf puzzle” ?

Imho, optims puzzles are fine. I agree with you. Just a different problem. But code golfs puzzles are just trick with short domain name or bash code with system call …

1 Like
#14

Too many changes!
In my opinion, the rules so far were generally appreciated by codingamers. Except for the imbalance between games where there are many participants and others.
Personally, I will only opt for one other modification: to take into account more contests (6 for example).

1 Like
#15

Lucky/unlucky rolls on the ladder should be taken care of too. A game is played for each bot at regular intervals, similar to what riddles.io has.

1 Like
#16

I bought a 4 letter x.yz domain for $1.99, giving a 9 char solution. It was a fun mini-game to find one for such negligible price, and to make it “solve” the puzzle. Anyway network access and such solutions were removed since, which is good thing, so that’s a non-issue.

I agree removing system calls would probably make golf more interesting and diverse, since currently a short bash solution can be reused in many languages. It might be tricky for CG to do reliably for all languages.

1 Like
#17

The more I read your feedback, the more I doubt about having a global leaderboard. In fact, a global leaderboard should be global and so all activities should be represented. Someone which is not good in all activities should not have a good ranking.

So, before trying to compute a good global leaderboard, let’s answer the fateful question, assuming we can have a leaderboard per category:

For or against a global leaderboard?

#18

I’ll answer for everyone: for of course, and here’s the formula to use that incidentally ranks me first.

4 Likes
#19

Against.

Reasons:

  • the discussion is now about % of how much each category matters and clearly not everyone likes every category. Actually, who does like every single category?
  • it’s best if everyone focuses on what they like most. Clearly some folks like clash of code and they only want to play that, why should their category be less valued on a general leaderboard when all they care about is clash of code? Instead it could have its own separate leaderboard with 100% of the value.
  • allows adding extra categories without much arguing, as you wouldn’t have to re-adjust percentages

Otherwise, i’d like to ask if it’s fun to play leaderboard optimization. It has been done before with the readjustment of max points one can get from code golf or optimization games. So, is it fun adjusting those values?

1 Like
#20

If i understand well, CodinGame need a “leaderboard” to be representative for companies. In the same way as the StackOverflow reputation rank. StackOverflow has a global reputation rank which is just the sum of all the reputation you have (for all tags and all activities). They don’t use any ratio but each activities rewards different points so this is the same in the end.

So the question is : When a recruiter/someone looks at our profile, what is the wanted information ?

I always feel that the global leaderboard just represents how much time we invest in CodinGame. It’s not always hard to reach a good ranking everywhere, it just takes a very long time. If this is what CodinGame want to represents, then any formula/ratio will be good enough (you can always tweak the formula, it will always represents how much time you invest in CodinGame).

If you want a leaderboard representing how good we are, it’s gonna be complex. What defines a good developer ? Reaching a high rank in a contest coding in “C++ without O3 so your code is ugly as hell” makes you a good dev ? Farming CoC makes you a good dev ? Copy/pasting your contest code into a multi puzzle makes you a good dev ? Reverse engineering a puzzle validators then submit a hardcoded solution makes you a good dev ?

Does using CodinGame makes you a good dev or a problem solver ?

But if we have no more global leaderboard, we could also just have a feature in our profile and choose what rank we want to display in our profile card (and in our profile as the first information). This way everyone can choose what to display.

I don’t really care of my global ranking. The only 2 things i don’t like is that some puzzles/multis rewards way too much points (and some puzzles are just useless) and CoC is just farming everyday. But CodinGame have to decide what they want to represents with a global leaderboard.

6 Likes