AI Hiding

Problem: while many noble players advance in the open, some try to hide their strategies during all the contest. I think the latter is just rational (everybody wants to win!), but it has some strong anti-fun and anti-competition effects (I wholeheartedly agree with a related remark that @Agade put in his postmortem).

So maybe we could add some rational incentive to release early and keep the best results in the arena? For example, for each player, consider the whole evolution of their ranking and compute a bonus for being high in the leaderboard for longer periods of time. At the end of the contest this bonus could enter into the final score.

A few technical details should be decided (What exact formula for the bonus? When do we start the observation? from the Bronze League? What weight do we assign to this bonus at the end?..) But all this seems not too difficult, and IMHO the benefits outweigh the costs.

So what are the potential benefits? The game would be more open. More ā€œreal timeā€ too. The bonus would also be a fair contribution to telling apart players who are often very close to each other in the raw score: the one who has been leading more of the time would be deemed the winner. Last but not least, itā€™d would be an innovative approach to scoring AI contests that attempts to measure a kind of real life (ā€œreal timeā€) efficiency.

What do you think?

Isnā€™t the only goal to have the best AI at the end of the contest ?
I donā€™t see why someone with the best AI during 7 days but beaten by a better AI on the last day should win the contestā€¦

Moreover it wouldnā€™t be interesting at all, someone who does not participate in the early days of the contest couldnā€™t recover, even with the best AIā€¦ In my opinion, it would be a nonsense.

4 Likes

http://www.strawpoll.me/12480814

4 Likes

There is a simple way to fix AI hiding.

When you submit your AI in the arena, codingame generate a new AgentId for this AI. You can see this id when you use their API (my cgstats tool use them, so i know how to find this AgentId :stuck_out_tongue: ). If you know how to tweak the codingame IDE, you can play a game against an old AgentId. AgendId are not destroyed during a contest (or maybe very old AgentId are destroyed, i didnā€™t check every AgendId i found).

If codingame want to fix AI hiding, itā€™s simple. Just give us a way to play against an ā€œold AIā€ from a player in the arena. This is already possible when you know how to change javascript variables of the IDE !
For information, this is how i ā€œcounter attackā€ AI hiding. I keep the AgentId when someone submit a real AI (and i know he will hide it just after) and i play against this AI in the IDE.

Maybe someday iā€™ll improve cgstats to store AgentId of every players in the arena during a contest. And iā€™ll create a topic to explain how to modify the IDE to play against an old AgentId.

8 Likes

You can always push an AI that timeout after a certain amount of time so even if you have an old agentID, it wonā€™t be useful.

3 Likes

At the moment, no one do that :smiley: But yes itā€™s possible.

@anst I feel like it is dangerous to rank AIs on anything other than win rate.

The solution would have to be technical, like giving people a UI to play against all previous submits of a person. It might discourage it alot. But it would still be possible to never submit until the end. And what saiksy said, having an AI that plays badly if the time is later than 1 hour after my submit.

2 Likes

Well, I think I should agree. Winrate is the king and any modification could prove controversial (@Numb has let out their strong feelings). And yes, itā€™d be quite easy to cheat anyway (Magusā€™ option has at least the merit of being very simple).

(That being said, I appreciate the poll.)

2 Likes

We should be clear about a problem that needs solving.

A person submitting the AI for the first time at the very end sounds fine to me. He didnā€™t profit from trying to play vs others nor others profited from playing vs him. I doubt that this approach would produce top AIs, but I see no harm in it.

Where I do see harm is that one person feeds of other ppls good will. Everyone puts their AIs in the open, making them available for other to fight vs them. They profit from being able to observe how it does in the rankings, while others profit from being able to play vs that particular AI, test their strategies and improve accordingly.
In such a system, when minority decides to hide their AIs, they are in fact abusing the good will of those who donā€™t because there is no reciprocity. If I just use all those available and open AIs to test mine and see how it does, yet I actively try to prevent them doing the same, itā€™s a pretty one-sided exchange.

Being able to see history of bots and previous ID seems like a very good idea, while making bot stop working after a fixed amount of time should be considered cheating and being punished (I know enforcing is a problem, but that doesnā€™t mean cheating should be ok).
Also, being able to see all those played games for a bot that was submitted and now is removed would also help - this doesnā€™t suffer from the timeout problem, but maybe logging each game replay is a storage problem.

In the end I am baffled by failing to understand the reasoning behind hiding. Is it:

  1. ā€œOh look I thought about this cool strategy noone else thought of!ā€ - Really?
  2. ā€œOthers mustā€™ve thought of it too and decided itā€™s not fair. What do I care, Iā€™ll just do it.ā€ - Sadā€¦
  3. ?? - Something else Iā€™ve missed ?

This is so lameā€¦

For the record, I have never done ā€œAI hidingā€ because I donā€™t think I can make good use of it with the risk associated with it. I may have even lost a first place to it before, and I think itā€™s great. All this to say, and being fully aware this is a controversial opinion, I see absolutely no problem with people making use of it, and I am not even defending it because I am using it, far from it.

In fact, I have a hard time understanding that perspective. You canā€™t force people to have their best version out. If through a lot of thinking and testing you discover an awesome strategy, why would you be forced to show it to everyone, why couldnā€™t you keep it secret and reap the benefits of your creative thinking? Since itā€™s a competition, such mind games are part of a healthy, competitive community. If you are playing within the rules of the game and the competition (and this is the important part), it is your responsibility as a player to use everything at your disposal when you play to win. I have a lot of respect for that.

I strongly recommend reading this classic and great article on the topic of playing to win when having a competitive mindset. It addresses common arguments such as ā€œthis is not fun/fair/etc.ā€ and the approach to take when you want to win. While itā€™s mostly about game design, you can also apply it to a ā€œmeta-gameā€ such as a CG competition.

One thing to clarify, while I donā€™t see any problem with it, it is possible that ā€œAI hidingā€ is something that needs to be addressed as a whole, if it is the general desire of the CG staff or community and if there is a good case for it. But, and itā€™s very important, this would need to be answered at the level of game mechanics / tournament rules that are clearly defined for everyone to see, and not with unwritten rules or something subjective as ā€œitā€™s not fairā€.

I would also like to sincerely congratulate the players who performed very well on GitC, ā€œhidingā€ or not, for playing to the very best of their abilities within the rules of the tournament. Thereā€™s been some impressive displays of skill there.

11 Likes

My only argument against the AI hiding is the following : If every one use it, the arena will be empty until sunday 19h50. No fun at all. Every one will just quit and at the end youā€™ll have something like 100 players. Less players, less sponsors, no more codingame, no more contest.

At the moment, only a minority of players use it. Codingame has no reason to deal with it. No impact on the players count for the contest.

I canā€™t blame people for AI hiding since thereā€™s no rule against it. But i canā€™t really think this is harmless. Since it has an impact on the contest ā€œfunā€.

5 Likes

I think the simple fact most people donā€™t care about winning that much will ever make ā€œonly 100 playersā€ a reality. Just look at the minority who make it to legend league.

3 Likes

@reCurse Because of the things you said I also wouldnā€™t diminish results of the person that did it this time in any way. I already congratulated him and I was sincere. His reasoning is questionable to me and I disagree with it, but it wasnā€™t against the rules (neither is hacking someoneā€™s computer and trying to DDOS him so he canā€™t submit his solution for example, but I wouldnā€™t do that either).
I still stand behind my opinion that he used an ā€˜exploitā€™ whose nature is such that if a lot of people start using it, it would ruin the ā€˜gameā€™ and therefore needs fixing.

Since youā€™re mentioning competitive mindsets, there are a lot of similar examples from the world of competitive gaming (among which I followed SC very closely all the way from Broodwar).
Exploits are found. Whenever they threaten the ecosystem, they are regularly proclaimed against code of conduct or are fixed.

I love competing and have been a lot in my life. But if you are trying to make a very literal point about ā€˜use everything at your disposalā€™ I would strongly disagree. That can be taken to unreasonable extremes by unreasonable people. Most competitions donā€™t list all the things that are against the rules unless they existed for a very long time and had the chance to learn from mistakes.

Argh no idea why I have this many comments on this subject, no harm was done at all yet. Iā€™ll just stop now :slight_smile:

Peace!

1 Like

No offense intended, but I think comparing ā€œdid not submit his very best strategy until the endā€ to ā€œDDOS a competitor to prevent him submittingā€ is disingenuous. Are you seriously putting these two on the same level? This is mindboggling.

Most serious competitions actually have a very extensive ruleset of everything that is allowed and not allowed, precisely to avoid disputes like this. It is the responsibility of the organizers to do so, and not the players to ā€œinterpretā€ what they can do and being blamed for going against some unwritten etiquette.

5 Likes

For once, Iā€™ll try to keep it short :

  • Just for the sake of fun, I like to see a well populated leaderboard during the entirety of the contest (no hiding)
  • As reCurse said, the aim is to win. AI hiding is a reasonable strategy in todayā€™s situation. Unless it is explicitly and enforceably banned, competitive players will resort to AI hiding.
  • I can not think of any univocal rule to actively prevent AI-hiding.
  • If AI hiding can not be stopped, at least CG should not promote it.
  • One problem of AI hiding is the asymmetry of testing opportunities. Such asymmetry is facilitated by tools such as CG Spunk.

TL;DR : There is a need for explicit rules on the use of tools such as CG Spunk, because they make hiding easier.

6 Likes

I think it would be really unfortunate to restrict the use of CG Spunk-like tools just to discourage AI hiding. It is extremely valuable in other situations, as the submits before legend league do not have nearly enough games to have a good idea of the bot performance, and generally do not play enough matches against the top players even in legend.

For the sake of discussion I think a better solution to discourage AI hiding is to go along the lines of what Magus mentioned:

  • Make a 3rd party website like CG stats track the new submissions of top players along with their stats and agent ID.
  • Have a way to set the agent ID of an opponent inside the IDE.
  • Obfuscate the current date/time on the VM running the bot, to prevent the (rather clever) hack mentioned by Apo.

Again, I think this is putting a lot of effort on something thatā€™s not really worth fighting all that much. But if it must be done, please at least leave CG Spunk usable.

5 Likes

I forgot to add. Agade mentioned he used local simulations to find improvements to his AI. Is this considered to be AI hiding as well if he doesnā€™t submit those improvements until the end of the contest? The definition of ā€œhidingā€ is extremely vague and this is a problem when it comes to try and ā€œaddressā€ it.

2 Likes

I donā€™t have strong feelings on the subject, but I wanted to raise a point: if CG decides to Do Something About Code Hiding, it should be something that eliminates the advantage that it gives rather than detecting or punishing those who practice it. It seems like something that could result in a lot of false positives. Who here has never reverted to an old bot because their new one turned out to be worse?

2 Likes

I canā€™t argue a lot about the subject, because this is first worldā€™s problem (i.e. top 50 or less). I donā€™t think people at rank 100 or 200 or 2000 does that strategy, we are struggling to have some viable AI.

Said this, I think everybody shares a general idea of what ā€œhidingā€ AI is:

AI Hiding: Take advantage of fighting against real AIā€™s while trying to minimize the time your real AI is exposed to the public.

That minimization comes in several ways: Submit of real AI, then replace for a bad one ASAP; creating killswitches based on time; creating killswitches to timeout some % of sure victories, or just force segfaults; using extensive CG Spunk without any real submit (gray area),ā€¦
I canā€™t consider offline AI testing against yourself any kind of AI hiding, you are not taking advantage of fighting real AIā€™s of other players. I can win myself at football, but then compare yourself with Messi or Ronaldoā€¦

If CG wants to deter these practices and promote a fair play, they should start as adding a fair play rules of conduct on the challenge guidelines. Once itā€™s written ā€œhidersā€ must measure the risk of being out of the challenge vs be a bit lower on ranks.
Iā€™m not sure that those practices can be detected at 100%, so it shouldnā€™t be an script or anything that gave false positives. CG at any time can audit our codes, check ranks of each submit, times between submits, etcā€¦ A player that submits a top1 bot for 20 minutes, and as soon as the submit ends change it to a top50 bot for X0 hours is kind of suspicious, thatā€™s not a simple broken bot. Same with other tactics.

As this is a gray area, I donā€™t condone or agree with it. On a prisonerā€™s dilemma players should not cooperate to get the most benefit, so while itā€™s not against the rules these tactics can be questionable but completely right.
But CG should take care of its product and how they sell it to their sponsors.
And a fair play is more ā€œsaleableā€ than a game-of-thrones like competition.

1 Like