I don’t think this qualifies as a definition at all. What is a ‘real’ AI? If you are testing multiple approaches or strategies, on what criteria is any of those versions considered as being real vs not, and that it must be present in the arena? What if I decide to revert to a previous strategy later on because of whatever reason, will this be considered as hiding? How much does an improvement need to be before it’s considered to be hiding as opposed to a minor tweak? At which level must it be monitored, is hiding a top 20 bot the same as a top 5? What about top 1? etc. etc.
It remains a poorly defined, highly subjective concept and therefore unenforceable as is. Disqualifying someone on those grounds would be far more damaging to the CG community and “the sponsors” than because “he did not show everyone the cool strategy he figured out as soon as he did”, which is a mind game present in pretty much every other strategy competition out there.
Since AI hiding is virtually indistinguishable from fixing code late in the contest or even starting late, I don’t think it is possible to police this behavior without huge manual moderation effort.
But what I think what should be doable is this:
For me, the main issue with AI hiding seems to be that only one of the two involved players gets the results of the testing. I think if someone uses a bot from the leaderboard for testing, the replay and the used ide:bot + ide configuration should be accessible for the owner of the bot from the leaderboard too.
Blatant hiding, reCurse, of course there is a big gray area on detecting those actions, that’s why I said that can’t be automated.
As I said, a guy that submits a 5min bot that goes to top 5 and suddenly he just goes top 50 for long hours is clear what is happening. Or a guy that its majority of loses are timeouts when there was a sure victory. I don’t know at what level this strategy works, maybe on top30/ top50.
Think about a referee on any sport. Sure, there will be tons of “gray” actions on a match, but he is here to detect what actions are against the rules. Not good or bad, but against the rules. There are a lot of rules on real sports made exclusively for a fair play. Why we need an offside on soccer/football? for a more amusing game.
And as far as sponsors I prefer an “Agade” or a “Magus” rather than a “MrHyde”, but it’s a personal preference.
It’s not about “he shows off how cool he is” but “he win in equally fair conditions against the rest of AI’s”
You mention rules, and this is exactly what I am getting at. A rule needs to be written carefully and clearly to determine exactly and with as little ambiguity as possible what you qualify to be ‘blatant’ hiding vs ‘normal’ activity. I do not think it is possible to do so because the concept is too abstract to judge on and actually be fair to everyone.
It is not about showing off or being cool, it is about being properly rewarded for figuring out the challenge better than others did. Which is the whole point of the competition.
How about after the contest ends, all the latest submissions are tested from 0 and need to grow up (aka: no matter what you do in the competition week, only the final submission is ranked from bottom of leaderboard all the way to the top)? And all submissions during contest time are available for all users, with no possibility to delete them? And, hook into the VM’s OS to make all time functions report bogus values. With no internet connectivity from the VM to the outside this should solve all the bot hiding problems. At the cost of requiring way more resources and efforts.
One way to encourage early competitive submissions would be to have a prize for the person who held the top place for the longest, or longest in the top 10. Similar to the finishing top in a language etc. It wouldn’t be as good as winning the competition, but it would be a decent insentive.
tl;dr version: 1. Ai hiding is bad for the platform in the medium term. 2. I believe there are some relatively simple means --more transparency-- to keep it in check.
Here are full details. 1. How do you want a CG competition to look like? My take is this:
The good players will find incredibly overpowering tactics and patterns. As they play the game more, they’ll be forced to find counters to those tactics. The majority of tactics that at first appear unbeatable end up having counters, though they are often difficult to discover. The counter tactic prevents the first player from doing the tactic, but the first player can then use a counter to the counter. The second player is now afraid to use the counter and they’re again vulnerable to the original overpowering tactic.
(Accidentally, this is a quote from Playing to win). I think this strategy evolution, often discussed in the chat, is what makes CG different from many other competitive programming platforms. But if, say, 30 best players keep hiding, this scenario will fade away. We’ll see fewer improvements during the week, less excitation etc. The unfairness coming from asymmetrical tests setting was already mentioned. These are real problems.
Yet, hiding is at present arguably the best tactic (and 100% valid too).
2. So can we do something smart about it? My first attempt above was admittedly lame… Here is another take:
In higher leagues, every bot that gets to top 20, has his 100 battles archived and easy to see for others (enough to keep the best bot for each top20 player). Also, this bot’s code should be easily reachable for others to test in the IDE. We make a clear rule that intentional (esp. time-based) crippling your bot on the display/arena is forbidden. Once you’ve tested, you stay there, that’s only fair play. Finalists (top 3 or 10, say) can be more closely examined at the end as in The Accountant.
People not pushing to the arena at all are already kept in check by the risk that the solitary tests in the IDE don’t guarantee much for the arena behavior. Additionally, we could restrict the IDE testing to no more than N adversaries per hour (N to be determined so that pushing to the arena from time to time would be simply the best option for everyone; consequently, during a contest, Spunk should be restricted too).
Lol, Magus used AI hiding for the last 4 days of FB
EDIT : The only people we should ask about AI hiding is the top 10 of the contest. I don’t see any of them in this topic. Why bother for something that doesn’t concern you ?
While i totally agree with the fact that i used AI hiding on FB, Agade and other people have a different point of view. I kept my best code hidden, but my code in the arena was still 1st. I revealed my best code saturday evening because reCurse was 1st. So yes it was AI hiding. But i never submitted a dummy code.
Magus wasn’t completely honest but he stayed first during the entire contest so he didn’t completely deny other people the opportunity to play against him and improve.
That makes no difference to me. What seem to bother you is that when one hides his best AI, opponents cannot train against one’s real level.
In fact, it’s a sneakier AI hiding when you make everyone think that your best version is in the arena.
If the only hiding that occurred was the #1 withholding his absolute best AI, the problem would already be much lessened. But I agree that it is still some kind of hiding.
Coming back to the original point, you can make an analogy with every kind of competitive esport :
In Starcraft for instance, players often prepare tactics that they do not display until the very end of a tournament; it allows them to surprise the opponent in crucial games and it’s considered normal to do so.
The main difference in CG contests in that while preparing such plans, you can still practice against the actual top players in the IDE whithout them knowing, while Starcraft players cannot test their hidden tactics against the people they’re aimed at.
So yes, as pb4 said, the asymmetrical aspect of these behaviors is the only real problem here. The two solutions I see :
Make the arena the only way to reliably test his AI :
Restrain the Play number a lot more in the IDE so players can no longer rely only on cgSpunk to test their AI without pushing anything in the arena
Add an agentId option in the IDE to test against withdrawn AIs
or
Accept this behavior, do the same and just prepare the best AI you can for the last day, knowing that someone is probably doing the same
I personally prefer the latter. The first one would harm everyone’s experience.
I don’t really prefer this solution but the former would also be completely impossible for CG. The servers are already overloaded near the end of a contest when everyone is submitting. Limiting the number of plays and relying too much on the arena would just kill the submits for everyone.
In this matter, I have to say I don’t really like game hiding. But I also agree that might be a “loser excuse” for me (as pointed out in the play to win article). Moreover I completely agree with reCurse that, if any action was to be taken, it would have to be written hard and plain in a set of rules. Which is very hard, because AI hiding is kind of ill-defined.
Then people could push in the arena to test their AI, and withdraw it the second it reaches 100%, and the situation would pretty much be the same as it already is.
My point is that you have to put everyone on the same level, either by forcing them to push their AI once in a while and prevent hiding with the agentId option, or accept these bahaviors.
Here is another take:
