Vision for Community Puzzles - 1.5 years later

[aCat]

In the current edition of my AI4Games course, which lab is mainly based on CodinGame, I decided to put more accent on puzzles - not only multiplayer games. As there were no classic puzzles satisfying my needs, I decided create my own (A* exercise, Minimax exercise, MCTS exercise). And I was hit by everything I remember inconvenient when trying that activity last time, probably a year ago. Sadly, I just realized that many issues were already pointed out and foreseen by @player_one in a 1.5 years old topic: link.

Thus, I decided to share my feelings, thoughts, and ideas of improvements. Of course I’m very interested how these things I’m pointing out are seen by the rest of the CG community. I structured the post so the sections correspond to the life cycle of a puzzle: creating, validating, and finally and hopefully - solving.

Creating

It’s probably the most straightforward one. Interface for making puzzles is sometimes a painful copy-paster but it is mainly OK (although I won’t complain on some “upload form xml” option; adding test cases between the existing ones isn’t the most user-friendly operation under the Sun).

However, there are some simple (as one might think) things that impossible to do when writing a puzzle. The most trivial and important one is inserting hyperlinks. The choice is either to make people google for the phrase (but the common reviewing point is that there should be some link), or put plain link as text, which looks ugly, unreadable, and requires copy-pasting to actually use the link.

The second point concerns image insertion. It is probably more complex to add, but really - a picture is worth a thousand words - and this will be very useful feature for the more complex puzzles. I know ASCII art is fancy, but not always the most readable.

Last, and most complex, improvement proposition repeats my idea from the previously-mentioned topic. Allow us to make interactive puzzles! This is not SPOJ, this is CodinGame. Community puzzles will always be boring because they lack graphics that is so crucial in official puzzles - is making them fun. At least, give users the power to send programs as validators. Programs that can converse with user’s programs. This is the scheme used by most official puzzles and this is the other reason it distinct such game-like tasks from simple and boring output-checking tasks. The interactivity that gives game-feel is really important here. Also - this will allow to implement more interesting puzzles and in less ‘artificial’ way. Consider my Minimax puzzle - what’s the point of alpha-beta if I have to read all the leafs? No point at all. It’s just the best way I found to make the puzzle given current limitations. Situation with MCTS puzzle is even worse.

Validating

Reviewing process sounds great - really. On paper it shines. In practice its… not working well, euphemistically speaking. However I don’t have any clear recipes, I just observe the flaws. I won’t focus on CoC puzzles because a) they are handled better as they are simpler, b) it is not an interesting case from my point of view.

One thing I know and strongly suggest to be fixed is acceptance/rejection ratio not visible right on the puzzle. Now we have to click puzzle, and then click dots to know what the values are. Which is incomprehensible and inconvenient. Also one have to click play to know who validated the puzzle, and I still have no idea how to check who refused it. (Notifications are the other thing, they do not remain with the puzzles, after month, two months and so on.) I wouldn’t mind puzzle/CoC filtering but that’s a minor suggestion.

The problem is that there are around 80 puzzles in the queue, some more than 1 year old that are mostly in a limbo state. No one is patient enough to solve them and approve them, no one is brave enough to reject them. The idea of community rejection/acceptance is good. There are even some really minor xp points to gain, but still it seems it is not motivating enough - or, alternatively, not clear enough how to make the reviews.

If I don’t like the puzzle, it doesn’t necessarily mean I’m rejecting it. I often feel like I do not have background to throw it to the bin right away. And probably multiple reviewers feel the same so the puzzle stays untouched. On the other hand, there are some puzzles I know are good, but it’s too much work to make a working solution and I have no time, so I won’t accept them. Even If I do, waiting for the remaining two approvals can take months!

It’s easier with clashes because they are smaller. But with community puzzles? I have to make my own solution that takes time equal of solving other unsolved community puzzle. Then inform the author about my propositions of improvements / found errors etc. Then wait for him to response - if he still cares, and some authors are not. An alternative, is dumping into someone’s contribution and changing it without his approval. As old versions are not stored, it does not seem as a nice behavior. Also, I should know all existing (>160) community puzzles to be able to spot a duplicate. (Just a note that existence of tags would simplify the process, as that would allow to make beam search through the list of accepted puzzles.)

So assume I solved the puzzle, I have to make a verdict. That is problematic because of the lack of clear criteria. I know everything is subjective, but really: ‘original enough + nicely themed’ make no criteria at all. What’s the global policy? Accept as many puzzles as possible? Accept the ones that teach something useful? Only hard ones? At the end all of them are threated the same, so all this thinking is pointless. Ironically, imho it will be easier to accept ‘hard puzzle’ or ‘easy puzzle’ then just ‘puzzle’. At least we would more clearly know what is expected of us (I meand the reviewers).

Solving

When opening community puzzles page, you are thrown over the bag of more than 160 puzzles. And you know that newest are first. In theory there are some user grades, but: a) they are meaningless, b) nearly all of them have 4 stars (the rest have 5 stars). Why meaningless? Given a grade, do you know the puzzle is easy/hard? No. Do you know it is teaching something important? No. You just know that some (not all) of the users that finished this puzzle liked it. Whatever that means for them. You even don’t have access to your ‘friends’ grades.

OK, but we have number of people that solved the puzzle. That gives some intuition towards easiness but: except the cases when puzzle is new, or is easy but long and boring, or the statement is vague, or is long forgotten on the bottom of the list, or whatever.

My point is, there should be some tagging system. If we are here to train ourselves and learn, what is expected is a mechanism consistent with the one of classic puzzles. Tags pointing out the topic covered. Difficulty measured by ‘experts’, hopefully influencing XP gain. Some of the community puzzles are hard, interesting and should have assigned appropriate rewards. Some has difficulty of the CoC tasks.

Tags and difficulties should also go in hand with some mechanism of filtering/sorting the puzzles. Also, given the number of puzzles, some more achievements should be very appropriate (solve 50 community puzzles, solve 100, solve 10 hard - given existing tag system, etc.)

Summary

To summarize. As there are no new classic puzzles (why?), community puzzles are the ones that support growth of the ‘practice’ subpage. And because they are not evolving, they are decaying. There are not that fancy because of no graphic, and no interactivity. There is too many of them, and too diverse in difficulty and teaching potential, to be threated as an unordered list without any sorting/filtering. Citing @player_one prediction, now we have “haphazard list of hundreds of random puzzles”. Also the reviewing process is slow, unreliable and unclear in its goals.

I’m very interesting about your feelings: authors, reviewers, solvers. And CG team - as this situation is actually not new - what plans do you have to support this half(!) of CodinGame-based activities?

[java_coffee_cup]

I cannot agree more that it needs the ability to attach images to puzzles. An image is worth a thousand words. It can simplify “wall of words” descriptions into a few nice short paragraphs and make the puzzle eye-catching to stimulate attempts.

Ranking of puzzles by number of submissions, number of members submitted solutions, success rate, etc. can automatically rank the difficulties of puzzles. Puzzles can be interactively sorted by these stats. By the way, I think Community Puzzles should be more difficult than CoC. I target to solve an average puzzle in a few hours, some may need a few days for research. Keep the 5 to 10 minute solutions to CoC.

It will be helpful for puzzle setters if there are more transparent criteria and procedure of approval. Let’s know what kinds of submission are preferred and what are discouraged. Let the community know how they can help in the screening or validation procedure. Then, you will get many more helping hands.

The community puzzle section is my most favorite content in CodinGame. It is updating frequently and with a wide variety of coverage that the frozen official puzzles cannot catch up.

[VilBoub]

I love Community Puzzles too but…
why can I see the authors’s solution when I click on “VIEW CONTRIBUTION” ?

[nicola]

Because you can fully edit his contribution (including the tests) so you should be able to edit the code too.

[TwoSteps]

Thank you @aCat for taking the time to reflect on the Community Puzzles section, and others for contributing to the discussion. I don’t know what we’ll do about it yet, but it’s definitely in our priorities (with the community contestS) for the CG platform. You all know that we’re also working on the Tech.io platform and also on our B2B platform (which enable us to live), but we’ll work on improving CodinGame too.

So please continue to give ideas, I read all comments and discuss it with the team after.

Without talking about plans, I guess the first steps would be to add filters/tags and then working on the whole validation process since we have now around 300 pending contributions.

[Unihedron]

I’m just a small potato so my voice probably doesn’t mean much, but being able to provide interactivity would be a huge bonus. Current “main puzzles” gains a lot from visualizing the process of the running code: When I’m playing The Bridge and I see the bikes fall off, I can identify the procedure and know that we screwed up, and there’s something in there I have to fix. When I’m solving Army Ants and I get “CoiGdmnae” instead of “CodinGame”, WHAT?!

It doesn’t even matter if it’s graphical anymore, I think we can all imagine a much more simple version of There Is No Spoon 1 in community puzzle form where you have to print out the nodes as a bunch of lines. You get:

00
0.

And you have to print out precisely:

0 0 1 0 0 1
1 0 -1 -1 -1 -1
0 1 -1 -1 -1 -1

Not getting a graphical output is, well, not bad, it’s not impossible to visualize it to some degree of correctness, even if it’s a nuance. Not being able to print in whatever order you want means you’re now forced to map nodes in a specific order, that’s not too bad, even if it’s annoying. However not having the interactivity and having your program terminated at exactly the line you screwed up means you’ll lose that much time trying to figure out what went wrong. It’s frustrating!

Result: "1 0 1"
Expected: "1 0 -"

(While we’re at this, this is just painful:)

Result: Nothing
Expected: Nothing

I get that it’s difficult to give the most relevant message ever, so showing you your output and compare that to the results directly does work well. But a lot of these community puzzles are really hard to understand considering the criteria can be quite convoluted (I have just finished a puzzle where the text was encoded by taking some letters in strings and shifting them forwards, different letters and then backwards, and reversed…).

I think community puzzles shouldn’t be treated as a “Plan B” citizen for those who don’t want to tackle the main puzzles, but that they should act as a complement. A lot of the community puzzles are a joy to figure out, but because it’s tied to a “get 100%” system, there’s not much to do beyond “solve it lol”, where as main puzzles could introduce elements around a mechanic, and propose new “puzzles” and “traps” to force the user to improve their algorithm to handle edge cases. This is especially well demonstrated when you look at Power of Thor 2 - You get monsters and a generous amount of attacks, the amount of attacks are no longer generous, you gotta move, you gotta spin them around. Interactivity enables intuitive puzzles delivered in a progressive way. A community puzzle that attempts to “step up the difficulty” would probably not be as successful in its delivery, considering you’re expected to reach the optimal algorithm from the start. (And you won’t be rewarded for reaching progression milestones.)

TLDR: I wish there were more interactivity on these. I don’t know if what I’m looking for is more community contests or interactive puzzles interface, I don’t know how expensive it is to maintain something like this, and I definitely can’t say for certain that I think such a feature would be successful in competing with other existing puzzles’ for the users’ time. But it’s something nice to think about. By the way, being able to create your own multi puzzles sounds fun, I wonder if that will become a priority after this first Community Contest.

[TwoSteps]

To all moderators of puzzles/coc genuinely interested in reviewing pending contributions: what do you think about these contributions-related notifications?

To others, I understand that it’s just a pain (and I agree).
We discussed before about an option to deactivate these notifications, but if noone is interested in these, I’d prefer that we just remove them.
Any idea for a kind of notification which will occur less often but will stir the curiosity towards contribution moderation?

[JBM]

I like and follow the ones for community puzzles. Moreover, I’d actually like one more for puzzle submission, and no 24-hours grace “information loss” period at all for approvals and refusals.

I’d like the CoC ones.to die in fiery hell.

[player_one]

like and follow the ones for community puzzles. … I’d like the CoC ones.to die in fiery hell.

My thoughts exactly. I completely ignore all CoC notifications and delete them on receipt. I keep around the puzzle ones as a to-do list: “Oh, someone looked at this puzzle. Maybe I’ll take a look too.”

• danBhentschel

[java_coffee_cup]

I welcome these notifications, including rejecting, accepting, or having comments added, etc. Having options to select/deselect Puzzle | CoC related posts is fine. I know my favorite can be very different from many other people.

[aCat]

I find notifications very useful, they give me hint what people thinks is unequivocal/interesting to review.
The CoC/Puzzle filter will be great though. However there are some other general issues:

1. Why I do not get notifications about puzzles I already validated/rejected? It seems weird.
2. There is no distinction between normal validation and accepting validation (same with rejection). Maybe the message should be something “XX validated (2/3) YYY puzzle”
3. Going back to list of pending puzzles, lack of information which one I validated/rejected is also annoying. One can don’t remember after a long time. (It is a special and more detailed case of one of my initial points of main page acceptance/rejection balance info.)

[JBM]

While we’re on topic with notifications: they’re implemented as JS-bound DOM event handlers instead of plain old HTML hyperlinks. So we can’t get a link out of them, or open in new window, private window, new tab.

This makes them completely unusable, for example, in an IDE tab where, no, I don’t want to leave without submitting my code, I don’t actually want to leave that tab at all, and now I clicked I can’t even have the notification stay “active” but it’s too late to repair. Why even have a notification area there if it can’t be usable?

[java_coffee_cup]

If possible, I would like Official puzzles and Community Puzzles be downloadable as pdf, and best to include test cases in it, to help preserving and printing on paper for carry-around amusement.
Having a printer-friendly version of html is also fine.
Some puzzles need a longer thinking time than coding time. Having the puzzles on paper I can do the thinking anywhere.

[nicola]

Is it possible to have a menu somewhere where we can list the contributions we have upvoted/downvoted and validated/refused?
I’d like also that the refused puzzles/CoC go in limbo (we can still see them) instead of hiding them permanently. Why not a two-step validation process? First, a proposed puzzle/CoC is in limbo then, if three moderators validate it, it’s in the the second stage. Three other moderators might return it back in limbo or validate it definitively. A puzzle can oscillate between limbo and the second stage but once a moderator accepted or refused a puzzle once, he can not refuse nor validate anymore.

[Arglanir]

Here are some enhancements that I thought of when I contribute/help contribution.
I also found this 1-year-old topic which deals with contribution sorting.
This other 1-year-old topic deals with interactive puzzles. It would be sooo good to have them.

For a contributor:

• When writing a draft, the formatting is not applied. You have to submit it as non-draft before you see the formating.
• It’s strange to have the button to test it in an IDE only on edition mode (well, it can make sense, but it’s different from when the puzzle is submitted).
• The contribution guidelines are outdated.
• Monospace strings leads to a block. It is not possible to have it inlined.
• Allow more than 1000 characters in the output. (The limit is 10000 for the input)
• Allow URLs (but I understand it may be dangerous)
• Allow images (but I understand it may be dangerous)
• For the stub generator:
• Allow writing constants (numbers, texts, maybe maps or lists)
• (I found something that made sense but was not accepted, but I can’t find it anymore)

For a moderator:

• Pending puzzles list page:

• See if you have already approved (green ✓) or refused (red ✗) a puzzle so that you know you don’t need to come back.

• Have a mark telling how much approvals/refusals have been done.

• Separate CoC from CP puzzles.

• In a puzzle page:

• Have a better indication of the type of the puzzle. The only way currently is a faint gray text over a gray background, at the top, hidden as soon as you descend in the comments, an the section “Game mode” which is present or not.

• In a puzzle IDE:

• If it is a CoC, allow switching to “Reverse” mode. Activating the Test cases in a modal window is not easy.

• It would be good to test the tests first, then the validators. This would allow testing if a validator covers something the test cases don’t.

Yeah, I know it’s Christmas soon

[TwoSteps]

Thank you @arglanir for the useful and constructive feedback. I hope you don’t mind that I moved your post to this thread which seems appropriate.

We’re currently working on our tools to create community contests. Improving the section of community contributions is part of our priorities (along with improving our emails). I can’t promise anything yet, but I’ll keep you updated.

[Arglanir]

Thank you. The current notification system is the only way to know where other moderators have approved/refused contributions, so it could stay as long as another visual information is not available.

[CyberLemonade]

Hello, I see all suggestions for improving community puzzles have been posted here. Most of my opinions have already been expressed, but I’ll still mention some of the ones I would really like:

1. Mention difficulty of a puzzle, and what we will learn.
   [Thanks a lot to Nicola, who religiously does this ]
2. Have a community suggested xp for the puzzles. Everyone who accepts must suggest a suitable xp for solving the puzzle.
   [Again, this is susceptible to misuse, so keep limits]
3. Sort puzzles in the community puzzles page (like aCat mentioned) in order of difficulty.
4. Add a discussions page for each puzzle.
   [Maybe this already exists, but I have never come across it ]

My opinion might not stand for much, but I do hope that one day, codingame will implement these changes

[TwoSteps]

We’ll soon add difficulty and tags to community puzzles.

For existing puzzles, difficulty will be based on the success rate of the puzzle and there won’t be any tags. They can still be edited by those who have the rights (level > 29).

We’re also planning on merging community puzzles with all classic puzzle (from CG), ie in the “practice” section.

[GoofyGas]

Where is your AI4Games course? It sounds interesting. I’ve bookmarked your puzzles to try them later.