As the platinum rift is over, I wanted to collect a few of my feedbacks and suggestions concerning it, future contests, the system in general and CodinGame.
First up, a big thank you to CodinGame for organizing this contest, it was a very pleasant problem statement, complex enough to have a load of challenges and tricks to try out, yet simple enough to keep the barrier of entry low. So kudos for achieving this Similarly, I’d also like to shout out to the community for being very positive throughout the contest, with a lot of fun chat sessions.
The suggestions themselves (ordered by how they pop into my mind):
Be more transparent and communicative with your members. A lot of questions went unanswered on the forums, or answered after days of nagging. Consider that by organizing this contest, over 1000 people decided to gift you their time. That is something extremely valuable and you should the least try to do the same, and be there when they need you.
If you do mid contest changes, better have a darn good reason and come clean with it to your members (see transparency above). We can all agree to changes, and even if we don’t completely agree, we can accept them… if you are willing to trust the community enough to admit why they were necessary. Never ever ever change things so opaquely to people who already invested 2+ weeks of their time. Ever!
Currently (at least for Go), the uploaded source limit is 50K characters. Throughout the contest I heard about people complaining that they have to resort to source compression, minification, etc to fit into your allotted limits. Please consider, that Google Code Jam gives you a 100K allowance for a three hour contest (external libraries not counting into that) and TopCoder allows ~1MB sources for their marathon matches (3-4 weeks, standalone submission). These are not some accidental numbers, but were deemed necessary. Please learn from others’ trials and significantly raise your source caps. I personally would go with the 1MB limit. It is enough to allow anyone to explore all kinds of options, but still small enough to prevent system abuse.
Your scoring system sucks. Period Don’t get me wrong, I understand how TrueSkill works and appreciate your efforts to try and fix up some of it’s weaknesses, but I don’t find them [i.e. your solutions] well executed. As far as I know, what you tried to do was to run every test 5 times under a rank of 20 to prevent anomalies caused by randomness. This imho should be done always, for every test, for every contestant. Basically in your current config, if I managed to reach top 20, then immediately went up at least into top 10 if not top 5, just because my program started to be evaluated fairly and not based on some random luck. If I somehow slipped out of top 20, then 1-2 accidental losses and I was already 25-27, from where it’s near impossible to climb back into top 20. All in all, I believe your current setup gives a huge unfair advantage to your top 20 rankers. In the top 20, your solution is evaluated statistically, after the 20th position, you have to beat every single game to climb. If I’m 19th, I have to be better only about 50% of the time. If I’m 21st, better be always better, or else I’m going dooown. I still find the Google’s approach (lock, reset, simulate for 3 days, same for everybody, no exceptions above a certain rank).
The visualizer for the contest was imho quite an overkill. I couldn’t see anything in the general visualizer and always had to switch over to debug mode before I could do anything worthwhile. It’s nice that you’ve tried to spice it up visually, but you kinda’ filled it with so much visual noise, that it became pretty much useless to me
Keep the IRC history. I personally only accessed the chat room through the web interface. Given that most communication went down there, it would be nice to be able to browse back the history.
I’m sure there are a few I’m forgetting now, but these were ones I really wanted to share and hopefully get some consideration from the admin team. All in all you organized a great contest, it had a lot of rough edges, but hopefully in time these will get polished down.
Cheers,
Peter
PS: Before getting on the defensive side, consider that I’ve just spent 1 hour of my free time to provide you feedback. I tried to make it constructive, hopefully you’ll find something useful in it.
Ok I mostly agree. About the mid changes though, if you are talking about the map change, it was clearly said from start that it could change, the only restriction being only that the map is a graph. It was not even said that it would be flat, there could have been airports and so on…
Then, on this particular challenge, I think the results were a bit too chaotic and random. Especially, 100 matches were very far from enough to get a stable rank. Once I submitted twice the same code in a row ending 9th and then 40th…
Another intersesting suggestion was to force people’s AI to beat a set of validation matches to avoid an antarctic only spawn ending in top 100 for example. Just requiring to beat the default IA was enough for this one.
Ok, so i’m not an admin, but i can give you my answer with the little power that i have.
As @godeffroy_valet said, the map was said to change, it was written DAY1 and i’m pretty sure it was planned all along. The whole point of it is to see if you can adapt quickly and give a chance for late bloomers to rise up in the rankings.
When there was a change, they update the blog, they put a popup on the website (annoying one, you can’t haven’t seen it) and they sent a mail. And when it wasn’t enough, they replied on forum, i think that’s not bad considering the amount of work they have right now.
I can understand your concern, but maybe it is just that they don’t have THAT much space available right now.
The visualizer is bling-bling but that what attract you in the first place.
IRC history probably won’t happens, because conversation until someone join can be secret/private and it’s part of what IRC is (google it, i did).
The problem is, you are quite agressive. I don’t blaim you for having complains, you can’t just be nice about it.
Also consider that you did a free contest with prizes and jobs offers, they could easily make an entrance fee if they wanted.
All said. I hope i wasn’t myself too agressive, and i hope the questions I couldn’t answered will be sorted out.
Transparency doesn’t mean that you notify of changes, it means that you describe the reason for those changes. That they didn’t do at all As I said, I gladly accept changes and I’m aware of their escape clause in the problem description. I’m just saying, that unexplained changes leave a bad taste. It was meant as a constructive criticism to be more open next time.
About the space concern. I hope you also know your argument about not having enough space is bullshit Any cloud provider gives you 1GB / 2.6 cents (!). That’s 21 thousand submissions Let’s leave this at that.
I don’t really agree with the IRC logic. If it’s a public thread which anyone can read, and especially if it’s something where potentially important discussions happen, that an archive would definitely be nice. If IRC is not appropriate for such a feature, generate an archive with a chat bot and serve it outside of IRC. The solutions are endless.
Anyway, as I already emphasized, the reason I enumerated these is to provide feedback so they may polish their systems and processes.
Hello guys,
yes I agree @karalabe . Scoring system is not easy to follow, nor to trust. My last submit go up to rank 50 before falling down to rank 148.
Otherwise, I 'd loved this challenge, easy to understand but with a lot of little subtleties to code.
Bravo Codingame
I agree with most things karalabe said. Except that I really liked the visuals.
As for the Platinum Rift game itself, I was somewhat disappointed when I realized what it actually is. The game outcome depends too much on rather unpredictable initial spawn. This was pointed out many times. It also makes scores even more uncertain, in addition to sampling errors which karalabe mentioned. If you look at how final battles proceeded, you will see that scores kept changing, and ranks shuffling until the end.
Submitting the same code twice in a row could bring me to top10 at one time, and 60+ at another, within a short period. That’s not very nice for such a contest.
Compared to previous AI challenge, there were improvements on this side. I mean, at least, we knew exactly the way the game was simulated. I wasn’t there during Tron but I remember very well the coordinates issue in Games of Drones (floats on server but integers for clients, and there wasn’t any forum at that time) and the 500 steps per turn of Poker Chip Race. These examples were possibly breaking code thus making you lose time. But here, nothing like that happened as it was said from the very beginning that the map could change and the change was actually understandable by anyone who spent a bit of time on this challenge (Antarctica rush to be sure of being 2nd or 3rd in 4 players matches).
As I said in another topic, there are many ways to keep a clean and understandable code while staying under the 50k limit. Arguing that Google Code Jam has a 100k limit for a 3 hours contests is a joke: if you reach that limit in that amount of time, you either did a lot of copy/pasting from external sources (or worse, inside your own code) or never spent time thinking about what’s the best way to actually solve the puzzles.
I also think it’s part of a developper work to keep their code short as it’s better for maintainability. So having a low limit is also a good way to learn about concepts like DRY.
It doesn’t.
However, and that’s my only complaint about this challenge, the spawn ruined everything. You could win or lose after first or second turn because it’s impossible to know where others will spawn. As @yngvix said, scores were definitely not stable during final battles while they were supposed to be and I’m sure the ranking would be different after another series of matches. But that’s not because of the scoring system, only the game itself.
I also agree with having IRC history. It’s a public channel and sometimes important informations are given through it so I too think nothing should be kept secret/private. If people want to talk privately, there are other ways than public channels.
I was hooked by this contest. I love writing AI code and coming up with heuristics and this contest catered to this perfectly.
I was surprised that there were almost no changes to the map. The description just said something about “zones” and “links”, which could change for any game. I would have liked dynamically created maps. As previously said, airports would have been interesting.
It was possible to improve your ranking by submitting your code several times. Your initial ranking was based on luck and heavily influenced your final rank. So it would have been nice to have less FFA battles and more 1v1 games, because 1v1 is a lot less luck-based. In FFA if another player decides to spawn where you spawn and the third player spawns some place else, the third player automatically wins. It was a good idea to run matches 5 times with different seeds to compare the bots adaptability to different platinum distributions.
Thank you for this beautiful contest and bringing all of us enthusiasts together
@karalabe I think you took the game too seriously, pointed many times how valuable is your time. Come on, it’s been just for fun, even prizes were rather symbolic.
Definitely, a next contest should have:
a) much less random initial phase
OR
b) much more battles fought during score evaluation
TrueSkill gives the variance of each player score, it’s then easy to estimate how much games have to be played to get reliable estimate of the score at a given confidence level (i.e. probability of rank position error kept below given threshold).
Some sort of AI persistent state would be very helpful. We’re coding AI, and good AI can learn and adapt.
@NewboO Actually, I would argue that specifically by keeping the code clean, you can end up with larger code bases. Go for instance is a particularly verbose language. They specifically tried not to include magical constructs that hide runtime complexity. Btw, the reason I myself hit the limit was because I’m using a few helper libraries that I wrote (some previously, most during this contest), which are generic to be reusable between competitions. This though is something quite common in algo competitions, that some rather complex algorithms get extracted out into a lib and reused if needed. Simply because they are hard to get right, especially under pressure. However, if you strive for genericity, you sacrifice code length. Imho, this is exactly the DRY you mention. You are right that hitting 100K in 3 hours is extremely unlikely, but then again, not hitting 50K in three weeks is still a risky bet (that many many people hit)
@mrv A tad maybe I’m an all in guy, I either put a lot of effort behind my work, or just not start it. All in all I’ve spent this much time particularly because of the fun factor. However, this does not mean that there isn’t place for improvement.
@NewboO PS: I’ve just joined CodinGame for this contest, so I don’t know about previous challenges, but if you say that they made a lot of improvements since them, then it’s a very good thing to hear and hopefully this thread will lead to them further improving their platform. It’s a great place to be, even if frustrating from time to time
I was aware that there would be map changes. As a result, I tried to avoid any kind of hard-coded magic numbers in my solution from the very beginning. As a professional developer, I know that changes always happen so it’s critical to create software ready to embrace new changes.
Randomness
I was also very confused by the randomness in ranking. Submitting the same solution twice won’t obtain the same result. Finally, I’m convinced that randomness should be the main difficulty of this contest. As a result, in my solution I’ve taken into account the randomness and tried to cover the risks of my strategy.
After all, we have to admit that even in the real life, having some chances is also an important element for big success
Community
Finally, I’d like to thank the community. You guys are really very awesome, clever, passionate and most important fun to play against.
A big thanks to codingame for having made it happen. Looking forward to the next one.
By the way, could you please find some companies using scala for the next time ?
Randomness is ok generally. But:
a) it increases the number of games that need to be played to estimate the score;
b) there’s good randomness and bad randomness. Things like “roll a dice, if you get 1 you lose” are no fun really. And it was approaching that;
c) even a very strong strategy could be countered by choice of spawn. It’s impossible to make a strategy that will defeat all (look at the stats of the top10 in finals). This is a contest of programmer skill rather than luck, so such things are undesirable IMHO.
Randomness is of course OK, but should be used carefully.
In our case: at a given moment, with a given set of players and with the same map generation algorithm (i.e. constant parameters defining random distribution of platinum) there exists statically optimal spawn algorithm. But here come some troubles:
What is “optimal”? One has to define the goal function. It’s depend only on the coder skill, so it’s OK.
What does “statistically” mean? Just giving advantage while averaging over sufficiently large sample of games. Here’s the problem with small fights while ranking…
What if players changes their algorithms? We have to adapt to the new optimum. Without persistence of algorithm state between fights, reasonable adaptation is very inconvenient (has to be done manually)…
You are exaggerating the problem. Otherwise, all of us should praise to win this contest but clearly you see the difference between a good AI and an average one.
Honestly, this game is all about on how to make good purchases.
For example, I implemented one lazy strategy which plays only starting the second round. When I decide to make purchases, I take the continent which has most of available sources and fewest enemies. This works better than the one which purchases on a continent full of enemies and few resources. Most of time, this allows me to avoid taking the last place.
By the way, I trained many 1vs1 matches against you. You are really very good at those 1v1 games.
I was thinking about verbosity, but as I don’t know every language available on CodinGame, I’m not sure to which extent it’s making a difference. So if Go is particularly verbose, then it could indeed be a good idea to have a higher limit for AI challenges (maybe depending on the language then?)
Btw, it could be fun to have some kind of code golf on CodinGame to let people learn how short code could sometimes be.
Anyway, keeping the code clean includes things I’m seeing way too often like incredibly long variable or function names (though they could be shorter while still meaningful), dead code, and so on. You don’t necessarily have to push genericity as far as completely reusable code for this kind of contest.
And things like tabs instead of spaces can save a lot of chars. For instance if you consider lines with an average length of 60 columns (which is already high for an average, except for Python :trollface: ) and 2 indentations, you can save 10% of chars by using a single tab instead of 4 spaces.
Guys! You can’t avoid randomness everywhere and you don’t want, seriously.
But anyway good strategy can be found. For example, I spent a lot of time to found not so bad strategy to get second place in 3/4 player games (better than Antarctic rush). And this was really interesting and challenging.
The story of the TOP20 rankers is really strange for me, but now I understood something.
I agree that more games in the final stage would be helpful and would have a more accurate ranking.
Thanks to codingame and all competitors.
Keep coding, have fun (and luck).
I really wanted to thanks CodinGame for this amazing (free) Contest.
I really loved it and spent a lot of nights to work on to finish in the 100’s top.
Randomness and Limit of file’s size seem just normal for me. Like in life, there is always something that we don’t like and something which will never be under control… but we need to deal with it.
Another big thank to all of you, CodinGame and Gamers for this time, and wish to meet you again next contest!
Oh, I thought that in 3- or 4-player battle there is one winner and others are equal losers… You want to say that an exact place in battle was important? I did not find it in the rules
Challenge time should be twice longer : some of us cannot spend 100% time on it…
TrueSkill looks cool to me, may be underrun with only 100 battles but it’s nothing more than CPU time isn’t it ?
500 battles would be better and the final count (after the bell) is to apply to all participants : Some low rank AI’s may change the TOP 100 … It’s quite a big issue about TrueSkill, I spent long time to study that point on game of drones.
So the original concept is great ! congrats to codinGame team for the amazing integration job !
Now, the concept has to improve and grow up. Reproduce multi-players challenges many times may over repeat the same issues.
The main priority should be to allow the use of debuggers !
Writing trace logs is really an old way of programing and terribly time consuming.
It should be allowed to open sockets and write stdin/stdout on it … It shouldn’t be a big challenge to develop … Sound like a joke … : is there a computer dev to read this ?
Write and read via socket would let participants run their code on their own computer using their IDE while the server side would stay on the server as it worked that far.
EDIT : for some reasons, participants code is to be compiled on the server. Server must be able to launch as many instances of the code as it needs. Some of them can stay running as long as a debugging session is active with a timeout when devlopper fall asleep
Similarly, I’d also like to shout out to the community for being very positive throughout the contest, with a lot of fun chat sessions.
Don’t get me wrong, I understand how TrueSkill works and appreciate your efforts to try and fix up some of it’s weaknesses, but I don’t find them [i.e. your solutions] well executed. As far as I know, what you tried to do was to run every test 5 times under a rank of 20 to prevent anomalies caused by randomness. This imho should be done always, for every test, for every contestant. Basically in your current config, if I managed to reach top 20, then immediately went up at least into top 10 if not top 5, just because my program started to be evaluated fairly and not based on some random luck. If I somehow slipped out of top 20, then 1-2 accidental losses and I was already 25-27, from where it’s near impossible to climb back into top 20. All in all, I believe your current setup gives a huge unfair advantage to your top 20 rankers. In the top 20, your solution is evaluated statistically, after the 20th position, you have to beat every single game to climb. If I’m 19th, I have to be better only about 50% of the time. If I’m 21st, better be always better, or else I’m going dooown. I still find the Google’s approach (lock, reset, simulate for 3 days, same for everybody, no exceptions above a certain rank).
