On agentic AI, Codingame and incentives

In my personal opinion, it seems that Codingame is choosing quite poorly the direction that it is taking with this new “agentic coding” meta. I think that big changes will be required or this page will move away from its original purpose of “learning” and “having fun” coding. As I see it, what is being done may as well be the start of Codingame’s demise (not that I want that to happen, of course, that’s why I take my time to make this topic/rant).

Recently there was already a topic about code golf specifically and some worries about this new paradigm (I posted some thoughts there, although I don’t often participate in that mode, but it seems to be the most affected only after Clash of Code) and today I noticed too (from a discord comment) the new landing page and how the page is sold as an “agentic arena” with “real benchmarks” to compare each other and all that stuff… I won’t comment much on that, although it’s really not a good view especially for newcomers (but hey! At least we get to keep our natural reserve for humans in the arcade modes). The thing is more changes are being made: the next contest to test the API stability, making easier things for people that already use agents (I don’t know how that is the priority… as if agents didn’t do enough work already that we need to make it more accessible), Clash of Code was removed from the ranking points (okay? I don’t know if that will change anything), and the API itself that I don’t know what changes (there were ways already to submit or at least test code and get an API answer, check Number Shifting puzzle). Unless Codingame wants to change its business into being some kind of AI benchmark (fair enough if it’s good for them, I don’t know their company inner workings), I think that everything is going in the wrong direction.

Before, there were mostly three incentives to be on this page, depending on the person: one was the competitive programming aspect, the learning aspect and the “having fun” aspect (which could be thought as a mix of the other 2, varying from person to person).
For the learning aspect you could solve some puzzle or read other people’s comments and post-mortems (although this has always been quite lacking, and often relies on external resources or is poorly structured). You could also make your own discoveries/inventions which was quite neat!
The competitive aspect was obvious: there are leaderboards for games, in general, categories… and it was part of the motivation to “improve” on things that you could have already solved (or mostly solved).
With these new paradigms, the competitive aspect especially (and most of the learning) is now almost over: the most efficient way of getting around, solving something or climbing the leaderboard is to just pay some subscription and throw any problem to several agents in a loop until they find the answer. And in most games… that’s the optimal way too! In terms of how many things they can do and how precise they can end up being. If you’re efficient enough, you can even remove yourself from the loop!, isn’t that fun?

Some time ago at least when I got an answer from an AI, I could check what it did, correct something and know what was happening. Now the agents send you a block of insufferable and dense code that somehow works better than anything you could have ever come up with, and I can’t touch a single line because I don’t know anything that is happening (I’ve tried looking… here you have an excerpt, now imagine 1800 lines of that):

No way I try to understand all of this...
...
    def exhaust(self, batches, cap, maxlen=7):
        """walk the systematic skeleton list (cheapest first) band by band; every (skeleton, cost band, hole-space index, scan mode) is evaluated once ever
        (cursor stored in the database, so a later level only adds the new, costlier band; a stricter mode -- smaller --wmin / smaller --bal-* -- rescans
        only what earlier modes did not already cover).  True while unfinished work remains"""
        used_b = 0; (w, l), tag = self.mode_tag(); done_c = self._scan_cache()
        legacy_cov = self.full and w >= 10 and l >= 32
        for lo, hi in self.bands():
            for sk in self.systematic(1, maxlen):
                if sk.cost_min() > hi: continue
                if sk.cost() + 6 <= lo: continue
                total = space_size(sk)
                if total > cap: continue
                if legacy_cov and self.db.scan_get(self.scope + 'sk|1|' + sk.key)[2]: continue            # finished by the previous (band-less) version
                if any(w0 <= w and l0 <= l for w0, l0 in done_c.get((sk.key, lo, hi), ())): continue       # covered by a finished scan of an equal/stricter mode
                fam = f'{self.scope}sk|1|{sk.key}|{lo}-{hi}{tag}'; cur, tot, done, smp, fnd = self.db.scan_get(fam)
                if done: continue
                while cur < total:
                    if used_b >= batches or time.time() > self.deadline: return True
                    idx = np.arange(cur, min(cur + self.B, total)); hv = decode_holes(sk, idx) if sk.nh else np.zeros((1, 1), np.int64)
                    st = self.run_batch(sk, hv, 0, minc=lo); cur += len(idx); smp += len(idx); fnd += st; used_b += 1
                    self.db.scan_set(fam, cur, total, int(cur >= total), smp, fnd); self.db.con.commit()
                done_c.setdefault((sk.key, lo, hi), []).append((w, l))
                if fnd and hi <= 11: self.log(f'exhausted {sk.key}  (cost band {lo}-{hi}, {total} instances, {fnd} windows stored)')
        return False
...

Well, what would I do and why? In my opinion, the competitive aspect here is dead (or will be in a year at most). Every competitive player will have their bots/solutions made from these autonomous AIs, some with more involvement, some with none, and the rest will either have to end up doing the same or go on their own without caring about anything else (fair enough). I’m not saying that AIs or AI code should be banned or something… there’s no way to stop that, but I think that what is rewarded (either implicitly or explicitly) should change drastically:

I think that there should be incentives for people sharing what they know. Now we only get a glimpse of this after a contest, but now that we know that anything competitive will be AI-based, there will be not much to scratch in that regard. A way to implement this would be to allow players to write their own “hints” or details about the problem or what they did (not necessarily a full spoiler, and optional in any case). Those hints could be “locked” (set by whoever makes it, or some league-like system) so that you need to reach some level before accessing “higher knowledge. In either case I’m not the one to design a system like that… I’ll leave this to think for the rest (and if they want, for those who work at CG). My broad idea is that people could “vote” about what hints or guides were useful for them, and people who give the most useful hints are the most “rewarded” in this kind of system, level/ranking whatever might be.

This would work both ways: for people who want to learn, they can get resources and know how specific things are done. For people who solve a problem with agentic coding, it would give a reason to look and understand what the AI code actually does. I think there is loads of valuable knowledge on this platform, and most of it is kept “locked away” because it used to give people some “competitive advantage”. Does it make sense anymore? I myself fall into both categories depending on the context: wanting to know and having some knowledge of something :wink:. This would also give a starting point for people who don’t have a subscription to be able to somewhat compete or build something good.

It makes no sense for everyone to waste tons of tokens for an AI to solve the same problem over and over…

The other thing that I would change, in competitive terms (for what it’s worth), is that the ranking rewards should be much more skewed towards the top (or by how “dominant” a bot is rather than its raw rank): there is no point in having mediocre bots everywhere for the sake of having higher general ranking: specialization should be the objective. Also, by knowing (from the hints) what the rest of people do, this could incentivise others to test different/more exotic approaches to “break” the norm: that’s where breakthroughs happen in almost any field. That, at least, would have some value of its own…

Well with how long I took I rather not enter into other personal things as to why I used to like this page, or what motivated me to solve problems, or anything in that regard… but it really feels that those motivations are slowly going away. I’m indeed incentivised to use and rely excesively on AI to keep up, and I can’t put up with it. And I won’t be the one to pay for an AI subscription, I don’t doubt it has its fun, and its complexity, but if I ever got one I would rather spend it into some “bigger” project of my own, rather than to get a score in a benchmark without telling anybody anything so that I can keep my edge.

So feel free to leave your opinion, it would be good to get varied points of views: from newcomers, casuals, veterans, people who use agentic coding, people who don’t… What do you agree with? Or disagree with? Or what would you do/propose?

7 Likes

I haven’t solved a single puzzle (outside of the contributions page) since that update. Everything just seems pointless now. Language certifications and quest map certifications are temporarily removed. Classic puzzles and clash of code count for nothing, and there’s no point playing ‘AI Games’ because I’m not an AI.

if I ever got one I would rather spend it into some “bigger” project of my own

This is my main issue. Why do I need a website of puzzles to test my ability to tell an AI what to do when I can just have it do something useful instead?

I would have understood new game modes or leaderboards intended specifically for AI users, but completely switching gears and dedicating the entire website to AI leaves a sour taste in my mouth.

I was gut punched by the decision to remove clash of code from the leaderboard. I really don’t understand the logic. People (including me) were complaining about a handful of AI clashers on the CoC leaderboard, so instead of doing something about it they remove CoC from the global leaderboard. There’s still going to be AI clashers at the top of the CoC leaderboard, all this has achieved is removing the incentive for people to play clashes. There’s maybe 100 codinpoints between a person who has played a few clashes and the person at the top of the CoC leaderboard, was that 100 points making any difference to anyone at the top of the global leaderboard? Not to mention, if the website is now dedicated to AI, why is AI use in CoC a problem?

I also don’t understand why code golf is in the AI section instead of arcade. Rule of three perhaps? The best code golf solutions are hard codes anyway. What am I supposed to do? “Hey claude, make this shorter” I guess?

I know times are changing and we need to bow to our AI overlords but humans still need to know how to code, and this was the place to learn.

3 Likes

It feels a bit weird that now the LLM agents can be running free and there is no real way of knowing how many of the solutions are human and what is the work of LLMs. On the other hand it is nice that there is some kind of LLM arena where it is possible to see how good it is and to get a chance to see how LLM would approach given problem and maybe learn something from it.

I think it would be amazing to have separate leaderboards for LLMs and humans, so it is possible to ignore LLMs if those are not the solutions I do not want to be competing against. It would be even better if the solutions could be annotated with information such as what LLM was used and how much of human input it took to get the solution. This way the LLM leaderboards would bring value to those who want to choose what LLM to use and how much the prompting effort affects quality of the solution.

Obviously as this would be on voluntarily basis, and there could be some cheating, but it still would feel kinda nice.

3 Likes

Yes, of all solutions that could have been taken to “fix” Clash of Code, they really took the worse one which is just removing it from view… And it didn’t even have that big of an effect on “ranking” terms: you could be top 100-ish just playing a few clashes from time to time and that was already ~4990 points! As long as there is a leaderboard OR it is unmoderated, I doubt it would be fixed (and you can tell that many of the people at the top must have had some kind of exploit to get there in 10-20 games, not even AI…).

2 Likes

What about the learning and gaming aspects, though? They haven’t vanished. I’m still solving puzzles, and it’s still interesting to me. I’ve never aimed for the top spots on the leaderboard though.

I’m new to agentic coding and haven’t tried it yet. From what I’ve heard, it’s more about working on big projects than solving CodinGame puzzles in a single file. But I guess trying it out in the CodinGame format is a good idea and might be fun. The new quest path gives me some motivation to try Codex.

But yeah, I agree that separating leaderboards for agentic and non-agentic coding would be beneficial. Although, I guess there are pros and cons to that, as with anything.

1 Like