Problem: while many noble players advance in the open, some try to hide their strategies during all the contest. I think the latter is just rational (everybody wants to win!), but it has some strong anti-fun and anti-competition effects (I wholeheartedly agree with a related remark that @Agade put in his postmortem).
So maybe we could add some rational incentive to release early and keep the best results in the arena? For example, for each player, consider the whole evolution of their ranking and compute a bonus for being high in the leaderboard for longer periods of time. At the end of the contest this bonus could enter into the final score.
A few technical details should be decided (What exact formula for the bonus? When do we start the observation? from the Bronze League? What weight do we assign to this bonus at the end?..) But all this seems not too difficult, and IMHO the benefits outweigh the costs.
So what are the potential benefits? The game would be more open. More āreal timeā too. The bonus would also be a fair contribution to telling apart players who are often very close to each other in the raw score: the one who has been leading more of the time would be deemed the winner. Last but not least, itād would be an innovative approach to scoring AI contests that attempts to measure a kind of real life (āreal timeā) efficiency.
Isnāt the only goal to have the best AI at the end of the contest ?
I donāt see why someone with the best AI during 7 days but beaten by a better AI on the last day should win the contestā¦
Moreover it wouldnāt be interesting at all, someone who does not participate in the early days of the contest couldnāt recover, even with the best AIā¦ In my opinion, it would be a nonsense.
When you submit your AI in the arena, codingame generate a new AgentId for this AI. You can see this id when you use their API (my cgstats tool use them, so i know how to find this AgentId ). If you know how to tweak the codingame IDE, you can play a game against an old AgentId. AgendId are not destroyed during a contest (or maybe very old AgentId are destroyed, i didnāt check every AgendId i found).
If codingame want to fix AI hiding, itās simple. Just give us a way to play against an āold AIā from a player in the arena. This is already possible when you know how to change javascript variables of the IDE !
For information, this is how i ācounter attackā AI hiding. I keep the AgentId when someone submit a real AI (and i know he will hide it just after) and i play against this AI in the IDE.
Maybe someday iāll improve cgstats to store AgentId of every players in the arena during a contest. And iāll create a topic to explain how to modify the IDE to play against an old AgentId.
@anst I feel like it is dangerous to rank AIs on anything other than win rate.
The solution would have to be technical, like giving people a UI to play against all previous submits of a person. It might discourage it alot. But it would still be possible to never submit until the end. And what saiksy said, having an AI that plays badly if the time is later than 1 hour after my submit.
Well, I think I should agree. Winrate is the king and any modification could prove controversial (@Numb has let out their strong feelings). And yes, itād be quite easy to cheat anyway (Magusā option has at least the merit of being very simple).
We should be clear about a problem that needs solving.
A person submitting the AI for the first time at the very end sounds fine to me. He didnāt profit from trying to play vs others nor others profited from playing vs him. I doubt that this approach would produce top AIs, but I see no harm in it.
Where I do see harm is that one person feeds of other ppls good will. Everyone puts their AIs in the open, making them available for other to fight vs them. They profit from being able to observe how it does in the rankings, while others profit from being able to play vs that particular AI, test their strategies and improve accordingly.
In such a system, when minority decides to hide their AIs, they are in fact abusing the good will of those who donāt because there is no reciprocity. If I just use all those available and open AIs to test mine and see how it does, yet I actively try to prevent them doing the same, itās a pretty one-sided exchange.
Being able to see history of bots and previous ID seems like a very good idea, while making bot stop working after a fixed amount of time should be considered cheating and being punished (I know enforcing is a problem, but that doesnāt mean cheating should be ok).
Also, being able to see all those played games for a bot that was submitted and now is removed would also help - this doesnāt suffer from the timeout problem, but maybe logging each game replay is a storage problem.
In the end I am baffled by failing to understand the reasoning behind hiding. Is it:
āOh look I thought about this cool strategy noone else thought of!ā - Really?
āOthers mustāve thought of it too and decided itās not fair. What do I care, Iāll just do it.ā - Sadā¦
For the record, I have never done āAI hidingā because I donāt think I can make good use of it with the risk associated with it. I may have even lost a first place to it before, and I think itās great. All this to say, and being fully aware this is a controversial opinion, I see absolutely no problem with people making use of it, and I am not even defending it because I am using it, far from it.
In fact, I have a hard time understanding that perspective. You canāt force people to have their best version out. If through a lot of thinking and testing you discover an awesome strategy, why would you be forced to show it to everyone, why couldnāt you keep it secret and reap the benefits of your creative thinking? Since itās a competition, such mind games are part of a healthy, competitive community. If you are playing within the rules of the game and the competition (and this is the important part), it is your responsibility as a player to use everything at your disposal when you play to win. I have a lot of respect for that.
I strongly recommend reading this classic and great article on the topic of playing to win when having a competitive mindset. It addresses common arguments such as āthis is not fun/fair/etc.ā and the approach to take when you want to win. While itās mostly about game design, you can also apply it to a āmeta-gameā such as a CG competition.
One thing to clarify, while I donāt see any problem with it, it is possible that āAI hidingā is something that needs to be addressed as a whole, if it is the general desire of the CG staff or community and if there is a good case for it. But, and itās very important, this would need to be answered at the level of game mechanics / tournament rules that are clearly defined for everyone to see, and not with unwritten rules or something subjective as āitās not fairā.
I would also like to sincerely congratulate the players who performed very well on GitC, āhidingā or not, for playing to the very best of their abilities within the rules of the tournament. Thereās been some impressive displays of skill there.
My only argument against the AI hiding is the following : If every one use it, the arena will be empty until sunday 19h50. No fun at all. Every one will just quit and at the end youāll have something like 100 players. Less players, less sponsors, no more codingame, no more contest.
At the moment, only a minority of players use it. Codingame has no reason to deal with it. No impact on the players count for the contest.
I canāt blame people for AI hiding since thereās no rule against it. But i canāt really think this is harmless. Since it has an impact on the contest āfunā.
I think the simple fact most people donāt care about winning that much will ever make āonly 100 playersā a reality. Just look at the minority who make it to legend league.
@reCurse Because of the things you said I also wouldnāt diminish results of the person that did it this time in any way. I already congratulated him and I was sincere. His reasoning is questionable to me and I disagree with it, but it wasnāt against the rules (neither is hacking someoneās computer and trying to DDOS him so he canāt submit his solution for example, but I wouldnāt do that either).
I still stand behind my opinion that he used an āexploitā whose nature is such that if a lot of people start using it, it would ruin the āgameā and therefore needs fixing.
Since youāre mentioning competitive mindsets, there are a lot of similar examples from the world of competitive gaming (among which I followed SC very closely all the way from Broodwar).
Exploits are found. Whenever they threaten the ecosystem, they are regularly proclaimed against code of conduct or are fixed.
I love competing and have been a lot in my life. But if you are trying to make a very literal point about āuse everything at your disposalā I would strongly disagree. That can be taken to unreasonable extremes by unreasonable people. Most competitions donāt list all the things that are against the rules unless they existed for a very long time and had the chance to learn from mistakes.
Argh no idea why I have this many comments on this subject, no harm was done at all yet. Iāll just stop now
No offense intended, but I think comparing ādid not submit his very best strategy until the endā to āDDOS a competitor to prevent him submittingā is disingenuous. Are you seriously putting these two on the same level? This is mindboggling.
Most serious competitions actually have a very extensive ruleset of everything that is allowed and not allowed, precisely to avoid disputes like this. It is the responsibility of the organizers to do so, and not the players to āinterpretā what they can do and being blamed for going against some unwritten etiquette.
Just for the sake of fun, I like to see a well populated leaderboard during the entirety of the contest (no hiding)
As reCurse said, the aim is to win. AI hiding is a reasonable strategy in todayās situation. Unless it is explicitly and enforceably banned, competitive players will resort to AI hiding.
I can not think of any univocal rule to actively prevent AI-hiding.
If AI hiding can not be stopped, at least CG should not promote it.
One problem of AI hiding is the asymmetry of testing opportunities. Such asymmetry is facilitated by tools such as CG Spunk.
TL;DR : There is a need for explicit rules on the use of tools such as CG Spunk, because they make hiding easier.
I think it would be really unfortunate to restrict the use of CG Spunk-like tools just to discourage AI hiding. It is extremely valuable in other situations, as the submits before legend league do not have nearly enough games to have a good idea of the bot performance, and generally do not play enough matches against the top players even in legend.
For the sake of discussion I think a better solution to discourage AI hiding is to go along the lines of what Magus mentioned:
Make a 3rd party website like CG stats track the new submissions of top players along with their stats and agent ID.
Have a way to set the agent ID of an opponent inside the IDE.
Obfuscate the current date/time on the VM running the bot, to prevent the (rather clever) hack mentioned by Apo.
Again, I think this is putting a lot of effort on something thatās not really worth fighting all that much. But if it must be done, please at least leave CG Spunk usable.
I forgot to add. Agade mentioned he used local simulations to find improvements to his AI. Is this considered to be AI hiding as well if he doesnāt submit those improvements until the end of the contest? The definition of āhidingā is extremely vague and this is a problem when it comes to try and āaddressā it.
I donāt have strong feelings on the subject, but I wanted to raise a point: if CG decides to Do Something About Code Hiding, it should be something that eliminates the advantage that it gives rather than detecting or punishing those who practice it. It seems like something that could result in a lot of false positives. Who here has never reverted to an old bot because their new one turned out to be worse?
I canāt argue a lot about the subject, because this is first worldās problem (i.e. top 50 or less). I donāt think people at rank 100 or 200 or 2000 does that strategy, we are struggling to have some viable AI.
Said this, I think everybody shares a general idea of what āhidingā AI is:
AI Hiding: Take advantage of fighting against real AIās while trying to minimize the time your real AI is exposed to the public.
That minimization comes in several ways: Submit of real AI, then replace for a bad one ASAP; creating killswitches based on time; creating killswitches to timeout some % of sure victories, or just force segfaults; using extensive CG Spunk without any real submit (gray area),ā¦
I canāt consider offline AI testing against yourself any kind of AI hiding, you are not taking advantage of fighting real AIās of other players. I can win myself at football, but then compare yourself with Messi or Ronaldoā¦
If CG wants to deter these practices and promote a fair play, they should start as adding a fair play rules of conduct on the challenge guidelines. Once itās written āhidersā must measure the risk of being out of the challenge vs be a bit lower on ranks.
Iām not sure that those practices can be detected at 100%, so it shouldnāt be an script or anything that gave false positives. CG at any time can audit our codes, check ranks of each submit, times between submits, etcā¦ A player that submits a top1 bot for 20 minutes, and as soon as the submit ends change it to a top50 bot for X0 hours is kind of suspicious, thatās not a simple broken bot. Same with other tactics.
As this is a gray area, I donāt condone or agree with it. On a prisonerās dilemma players should not cooperate to get the most benefit, so while itās not against the rules these tactics can be questionable but completely right.
But CG should take care of its product and how they sell it to their sponsors.
And a fair play is more āsaleableā than a game-of-thrones like competition.