On agentic AI, Codingame and incentives

In my personal opinion, it seems that Codingame is choosing quite poorly the direction that it is taking with this new “agentic coding” meta. I think that big changes will be required or this page will move away from its original purpose of “learning” and “having fun” coding. As I see it, what is being done may as well be the start of Codingame’s demise (not that I want that to happen, of course, that’s why I take my time to make this topic/rant).

Recently there was already a topic about code golf specifically and some worries about this new paradigm (I posted some thoughts there, although I don’t often participate in that mode, but it seems to be the most affected only after Clash of Code) and today I noticed too (from a discord comment) the new landing page and how the page is sold as an “agentic arena” with “real benchmarks” to compare each other and all that stuff… I won’t comment much on that, although it’s really not a good view especially for newcomers (but hey! At least we get to keep our natural reserve for humans in the arcade modes). The thing is more changes are being made: the next contest to test the API stability, making easier things for people that already use agents (I don’t know how that is the priority… as if agents didn’t do enough work already that we need to make it more accessible), Clash of Code was removed from the ranking points (okay? I don’t know if that will change anything), and the API itself that I don’t know what changes (there were ways already to submit or at least test code and get an API answer, check Number Shifting puzzle). Unless Codingame wants to change its business into being some kind of AI benchmark (fair enough if it’s good for them, I don’t know their company inner workings), I think that everything is going in the wrong direction.

Before, there were mostly three incentives to be on this page, depending on the person: one was the competitive programming aspect, the learning aspect and the “having fun” aspect (which could be thought as a mix of the other 2, varying from person to person).
For the learning aspect you could solve some puzzle or read other people’s comments and post-mortems (although this has always been quite lacking, and often relies on external resources or is poorly structured). You could also make your own discoveries/inventions which was quite neat!
The competitive aspect was obvious: there are leaderboards for games, in general, categories… and it was part of the motivation to “improve” on things that you could have already solved (or mostly solved).
With these new paradigms, the competitive aspect especially (and most of the learning) is now almost over: the most efficient way of getting around, solving something or climbing the leaderboard is to just pay some subscription and throw any problem to several agents in a loop until they find the answer. And in most games… that’s the optimal way too! In terms of how many things they can do and how precise they can end up being. If you’re efficient enough, you can even remove yourself from the loop!, isn’t that fun?

Some time ago at least when I got an answer from an AI, I could check what it did, correct something and know what was happening. Now the agents send you a block of insufferable and dense code that somehow works better than anything you could have ever come up with, and I can’t touch a single line because I don’t know anything that is happening (I’ve tried looking… here you have an excerpt, now imagine 1800 lines of that):

No way I try to understand all of this...
...
    def exhaust(self, batches, cap, maxlen=7):
        """walk the systematic skeleton list (cheapest first) band by band; every (skeleton, cost band, hole-space index, scan mode) is evaluated once ever
        (cursor stored in the database, so a later level only adds the new, costlier band; a stricter mode -- smaller --wmin / smaller --bal-* -- rescans
        only what earlier modes did not already cover).  True while unfinished work remains"""
        used_b = 0; (w, l), tag = self.mode_tag(); done_c = self._scan_cache()
        legacy_cov = self.full and w >= 10 and l >= 32
        for lo, hi in self.bands():
            for sk in self.systematic(1, maxlen):
                if sk.cost_min() > hi: continue
                if sk.cost() + 6 <= lo: continue
                total = space_size(sk)
                if total > cap: continue
                if legacy_cov and self.db.scan_get(self.scope + 'sk|1|' + sk.key)[2]: continue            # finished by the previous (band-less) version
                if any(w0 <= w and l0 <= l for w0, l0 in done_c.get((sk.key, lo, hi), ())): continue       # covered by a finished scan of an equal/stricter mode
                fam = f'{self.scope}sk|1|{sk.key}|{lo}-{hi}{tag}'; cur, tot, done, smp, fnd = self.db.scan_get(fam)
                if done: continue
                while cur < total:
                    if used_b >= batches or time.time() > self.deadline: return True
                    idx = np.arange(cur, min(cur + self.B, total)); hv = decode_holes(sk, idx) if sk.nh else np.zeros((1, 1), np.int64)
                    st = self.run_batch(sk, hv, 0, minc=lo); cur += len(idx); smp += len(idx); fnd += st; used_b += 1
                    self.db.scan_set(fam, cur, total, int(cur >= total), smp, fnd); self.db.con.commit()
                done_c.setdefault((sk.key, lo, hi), []).append((w, l))
                if fnd and hi <= 11: self.log(f'exhausted {sk.key}  (cost band {lo}-{hi}, {total} instances, {fnd} windows stored)')
        return False
...

Well, what would I do and why? In my opinion, the competitive aspect here is dead (or will be in a year at most). Every competitive player will have their bots/solutions made from these autonomous AIs, some with more involvement, some with none, and the rest will either have to end up doing the same or go on their own without caring about anything else (fair enough). I’m not saying that AIs or AI code should be banned or something… there’s no way to stop that, but I think that what is rewarded (either implicitly or explicitly) should change drastically:

I think that there should be incentives for people sharing what they know. Now we only get a glimpse of this after a contest, but now that we know that anything competitive will be AI-based, there will be not much to scratch in that regard. A way to implement this would be to allow players to write their own “hints” or details about the problem or what they did (not necessarily a full spoiler, and optional in any case). Those hints could be “locked” (set by whoever makes it, or some league-like system) so that you need to reach some level before accessing “higher knowledge. In either case I’m not the one to design a system like that… I’ll leave this to think for the rest (and if they want, for those who work at CG). My broad idea is that people could “vote” about what hints or guides were useful for them, and people who give the most useful hints are the most “rewarded” in this kind of system, level/ranking whatever might be.

This would work both ways: for people who want to learn, they can get resources and know how specific things are done. For people who solve a problem with agentic coding, it would give a reason to look and understand what the AI code actually does. I think there is loads of valuable knowledge on this platform, and most of it is kept “locked away” because it used to give people some “competitive advantage”. Does it make sense anymore? I myself fall into both categories depending on the context: wanting to know and having some knowledge of something :wink:. This would also give a starting point for people who don’t have a subscription to be able to somewhat compete or build something good.

It makes no sense for everyone to waste tons of tokens for an AI to solve the same problem over and over…

The other thing that I would change, in competitive terms (for what it’s worth), is that the ranking rewards should be much more skewed towards the top (or by how “dominant” a bot is rather than its raw rank): there is no point in having mediocre bots everywhere for the sake of having higher general ranking: specialization should be the objective. Also, by knowing (from the hints) what the rest of people do, this could incentivise others to test different/more exotic approaches to “break” the norm: that’s where breakthroughs happen in almost any field. That, at least, would have some value of its own…

Well with how long I took I rather not enter into other personal things as to why I used to like this page, or what motivated me to solve problems, or anything in that regard… but it really feels that those motivations are slowly going away. I’m indeed incentivised to use and rely excesively on AI to keep up, and I can’t put up with it. And I won’t be the one to pay for an AI subscription, I don’t doubt it has its fun, and its complexity, but if I ever got one I would rather spend it into some “bigger” project of my own, rather than to get a score in a benchmark without telling anybody anything so that I can keep my edge.

So feel free to leave your opinion, it would be good to get varied points of views: from newcomers, casuals, veterans, people who use agentic coding, people who don’t… What do you agree with? Or disagree with? Or what would you do/propose?

13 Likes

I haven’t solved a single puzzle (outside of the contributions page) since that update. Everything just seems pointless now. Language certifications and quest map certifications are temporarily removed. Classic puzzles and clash of code count for nothing, and there’s no point playing ‘AI Games’ because I’m not an AI.

if I ever got one I would rather spend it into some “bigger” project of my own

This is my main issue. Why do I need a website of puzzles to test my ability to tell an AI what to do when I can just have it do something useful instead?

I would have understood new game modes or leaderboards intended specifically for AI users, but completely switching gears and dedicating the entire website to AI leaves a sour taste in my mouth.

I was gut punched by the decision to remove clash of code from the leaderboard. I really don’t understand the logic. People (including me) were complaining about a handful of AI clashers on the CoC leaderboard, so instead of doing something about it they remove CoC from the global leaderboard. There’s still going to be AI clashers at the top of the CoC leaderboard, all this has achieved is removing the incentive for people to play clashes. There’s maybe 100 codinpoints between a person who has played a few clashes and the person at the top of the CoC leaderboard, was that 100 points making any difference to anyone at the top of the global leaderboard? Not to mention, if the website is now dedicated to AI, why is AI use in CoC a problem?

I also don’t understand why code golf is in the AI section instead of arcade. Rule of three perhaps? The best code golf solutions are hard codes anyway. What am I supposed to do? “Hey claude, make this shorter” I guess?

I know times are changing and we need to bow to our AI overlords but humans still need to know how to code, and this was the place to learn.

6 Likes

It feels a bit weird that now the LLM agents can be running free and there is no real way of knowing how many of the solutions are human and what is the work of LLMs. On the other hand it is nice that there is some kind of LLM arena where it is possible to see how good it is and to get a chance to see how LLM would approach given problem and maybe learn something from it.

I think it would be amazing to have separate leaderboards for LLMs and humans, so it is possible to ignore LLMs if those are not the solutions I do not want to be competing against. It would be even better if the solutions could be annotated with information such as what LLM was used and how much of human input it took to get the solution. This way the LLM leaderboards would bring value to those who want to choose what LLM to use and how much the prompting effort affects quality of the solution.

Obviously as this would be on voluntarily basis, and there could be some cheating, but it still would feel kinda nice.

5 Likes

Yes, of all solutions that could have been taken to “fix” Clash of Code, they really took the worse one which is just removing it from view… And it didn’t even have that big of an effect on “ranking” terms: you could be top 100-ish just playing a few clashes from time to time and that was already ~4990 points! As long as there is a leaderboard OR it is unmoderated, I doubt it would be fixed (and you can tell that many of the people at the top must have had some kind of exploit to get there in 10-20 games, not even AI…).

3 Likes

What about the learning and gaming aspects, though? They haven’t vanished. I’m still solving puzzles, and it’s still interesting to me. I’ve never aimed for the top spots on the leaderboard though.

I’m new to agentic coding and haven’t tried it yet. From what I’ve heard, it’s more about working on big projects than solving CodinGame puzzles in a single file. But I guess trying it out in the CodinGame format is a good idea and might be fun. The new quest path gives me some motivation to try Codex.

But yeah, I agree that separating leaderboards for agentic and non-agentic coding would be beneficial. Although, I guess there are pros and cons to that, as with anything.

1 Like

AI is mandatory in our jobs and used every day in almost every domain.
But when it comes to gaming, we can choose whether to use AI or not :grinning_face:
I prefer to play without it for fun.
For some games, it is possible to perform well without it (Troll Farm for example)
For all the others, it is interesting to see how much better AI is.
Keep in mind : the most important is to have FUN !

Isn’t solving any of the puzzles or competing in the quarterly Winter/Spring/Summer/Fall challenges pretty pointless since the era of AI? Any recent AI solves all of this pretty much instantly and often better than I could do it myself, finding and implementing the best-fitting algorithm with a snap of a finger.

I still like doing things on my own, though, understanding the problem at hand in depth and solving it efficiently. But every kid with AI can do it faster and better.

1 Like

Dear colleagues, have you ever given serious thought to the Winter/Spring/Summer/Fall challenges themselves? What real-world scenarios do they actually mirror? When viewed from this perspective, it becomes clear why the solutions provided by the site’s users are no longer sufficient. The real-world counterparts to these challenges demand the absolute best solutions available—the kind that can only be produced by AI. And, quite possibly, “fun” is far from the most important aspect here.

Then what do Codeforces contests mirror? :sweat_smile: I think their platform is more competitive, although not about bot programming, but they still don’t allow AI usage in live contests.

Codeforces: Coding Club. Develop your programming and problem-solving skills through coding practice and challenges.

If these challenges demand the absolute best solutions, then we should’ve been forming teams to solve them before LLMs. :sweat_smile:

Since @alexcercos chose to not go into their personal reasons, allow me to share mine.

I found CodinGame in 2015 at a time when I had to put my physics study on hold due to personal problems. I had just taught myself a little bit of C++ and CodinGame gave me a way to start from the basics and slowly discover my interests within the large world of programming. I discovered more programming languages (python is now my favorite for puzzles), learned new tricks from studying other’s solutions, and found a new love in code golf. I spent hours and hours tweaking solutions until I was satisfied with stuff like readability, even if that did not give me more points. The XP, achievements and leaderboards helped me stay motivated, but it was learning all the inns and outs of programming languages and individual problems that really stuck. I can’t stress enough how rewarding it is to finally solve a problem after hours or even days of thinking about them.

Fast forward to 2021 when my life was back on track and I finally finished my Master’s degree in physics. I chose the path of least resistance and applied for a job at a small local web development firm. And thanks to CodinGame I aced my assessment.

Fast forward again to the present where I am a senior(ish) developer with 5 years of experience. As everywhere else, LLM’s have completely changed our work. It has reached a point where my bosses (with 20 years of coding experience) throw AI generated code at junior developers that even I don’t fully understand. How are these new developers supposed to learn the basics? To counteract this movement, I had a plan to use CodinGame to give a workshop on competitive programming without the help of AI.

My heart sank when I opened CodinGame today.

The future of programming is unclear, but my professional opinion is this: in a world where agents can produce code faster than any human can read and think, it is of utmost importance that the developers steering those agents know the basics of programming and I believe that CodinGame was a perfect place to learn this.

8 Likes

I’ve been partaking in coding competitions in various forms for about 20 years and I’ve learnt you’re only ever really competing with yourself, even though the yardstick others provide can be hugely motivating.

There are a ton of reasons you can come up with why you didn’t outright win a comp with thousands of other entrants. Busy RL. Started late. Never done this before. Learning a new language. Got side-tracked on some cool learning experience more fun than the contest. Or, if you’re really brave, just ‘those guys are a bit better than me right now but I’m improving’. And surprise, surprise, when you are winning the opposite sets in, the doubt: I had more time. My rivals weren’t playing serious. They probably had more fun doing something else that week. I’m a sucker for tryharding. I should find something more challenging.

Coding agent are just one more dimension to add to that. I don’t see the big deal. Don’t use them, have fun taking pride in your manual coding, and factor that in when you interpret your result. Or do use them, and also factor that in when you interpret your result. Bring your own handicap. Focus on having fun, and feeling some sense of achievement and betterment in your chosen dimension, whatever that is.

I do agree though in hoping many will continue to properly share their approaches and experiences whatever those are. It’s an effort and something I’m patchy at, but it’s my favourite part of any coding comp. The Summer Challenge was notable for lots of excruciatingly patronising (but still appreciated) Claude-speak write-ups. I really hope the proliferation of those doesn’t put people off doing manual write-ups. The guys who did a quick LLM-writeup and then still took the time to manually edit or separately write-up get my special kudos. I think the human passion that comes through in those things is as important as the coarse technical details, even though something is always better than nothing!

1 Like