My coworker and I have noticed that even at rank 300, the player AIs don’t even know how to go around walls. And the players at the bottom are much worse than that. Some will crash on their first move. So it seems that the leaderboard is full of AIs which really aren’t doing anything. Personally I would like to know how I rank against AIs which are actually doing something, and this also makes matchmaking much slower because a lot of time is wasted playing against AIs which aren’t really ready to fight in the arena.
Has there been any consideration that players must meet a minimum requirement before submitting code to the arena? For example, being able to play 10 games against default AI without crashing. Or having some preset matches that prove the AI is at least rudimentary enough to play a game.
Of course this isn’t something that could be done for the Great Escape, but something to consider for the next battle.
I love this idea. Basically all of the infrastructure to implement something like this is there - it could be borrowed from single player tasks as a couple of test cases. Taking The Great Escape as an example:
Test 1: reach exit as player 1
Test 2: reach exit as a random player
Test 3: reach exit while avoiding walls
After finishing all test cases the “submit to arena” button could be “unlocked”.
Oh man, this would be so awesome. I hate worrying about all of the ridiculous corner cases that broken bots create.
I really don’t see why this is a problem since a decent AI would quickly climb above the trully “broken” ones… And I suspect the reason you sometimes see AIs messing up some apparently trivial moves is bugs that manifest in some edge cases. That can happen event to the best AIs.
The reason I’m against test cases is because it would provide a “framework” that would guide everybody in creating pretty similar code.
Well, I agree with what you mentioned, especially about the edge cases (to be honest I was pretty subjective on that matter), but
depends on what you mean by “quickly”. In PR2 and TGE this took about 25% of the whole ranking process, that’s a lot of games wasted and server CPU time burnt away for nothing - for a result that’s predictable from the start for anyone deploying their code for the n-th time.
The problem with that approach is that you can then tailor your AI to do really well against the top players and their tactics and ignore the bottom players. So there needs to be some kind of run against the entire leaderboard to make sure evaluation and ranking is fair.
I don’t think test cases would result in similar code if they are designed properly. They would be very basic, for example for great escape:
Succesfully finish a game without walls starting from all 3 sides
Sucessfully play 20 turns of a game with a wall without crashing (doesn’t have to make it across)
Just some very basic tests to makes sure the code has had some work put into it. I think those two test cases alone would cut the leaderboard down by 70%.
By the way, there are a lot of ways to solve this problem, apart from test cases, off the top of my head:
-Create a lobby or a two-tier ranking, you push your AI to the first one, and every couple of hours the top 10, 20, 50 players are promoted to the main leaderboard. If you reach the top leaderboard once, you stay there for each next push until the contest ends or your code ranks in the bottom 10, 20, 50 players or so, then you fall back to the lobby.
-Push the “Default AI” to the leaderboard, which then gets cleared every so often based on its rank
There was an interesting discussion in the chat on that topic. Somebody pointed out that the higher number of AI submitted was positive for Codingame when communicating towards sponsors.
A “simple” fix for the leaderboard while maintaining a high number of submitted bots could be the following :
push the Default AI to the leaderboard
All new AI pushed to the leaderboard start at “Default AI” ranking
…And start by playing a couple of games with Default AI only. This really sounds neat because I thing the Default AI’s score would sort of become the peak of the score bell curve.
Also, high number of players as an advertising asset seems like a good point.
I think one cool thing would be to do a set of tests (like in solo competitions).
4-5 tests and you wouldn’t be able to push your AI before finishing the tests.
In the end, from what I have noticed, the ranking seems to be fairly good. With minimal changes to my code I generally end up back around were I was +/- depending on the changes. However, the time to get there is quite long and it does present an interesting situation. If I push my code up at the same time someone in say the top 10 pushes theirs. We will end up competing in the 900-1000 rank range. I will obviously get my butt kick, but because of this it thinks I lost to someone in the 1000 rank range. Like I said it seems to balance out in the end, but definitely delays it.
I think a simple test is more than fair to make sure the coder has actually done some work. Just have a randomly generated board with some walls and make the player reach his goal. No opponents. Just get from one side of the board to the other. This is a key element in this game and if your code can’t do that, then it is not ready for the arena.
I think a simple test is more than fair to make sure the coder has actually done some work. Just have a randomly generated board with some walls and make the player reach his goal. No opponents. Just get from one side of the board to the other. This is a key element in this game and if your code can’t do that, then it is not ready for the arena.
You would think so, but the #1 player for a little while yesterday (iMost) couldn’t even make it around basic walls.
