I have to disagree with you on this point based on my experience as an Agile developer. In the Agile method, a task difficulty is evaluated by all team members within a few minutes. The more experienced the developers, the more accurate the evaluation because this evaluation is based on how they perform a similar task previously.
A bot is nothing more than a projection of someone’s biases and assumptions and, therefore, will not offer any more guarantees.
You’re talking about work. You don’t have 10000 people in your team at work. You don’t estimate alone, you have a dedicated meeting for that. There are a system in place to help you estimate. It’s probably tracked how much time programmers are spending on a task and in the case of over- /under- estimation don’t you have meeting to figure out what went wrong?
The bot will at least provide agency.
And in cases when a lobby of top-players are all having 0% after 15minutes you don’t need bias to figure out that puzzle is too hard.
Also, another point to my previous post:
It would be nice if people that approved/refused/commented on a contribution wouldn’t be able to get it as the random puzzle in the game of clash.
There is a moderation bot in place (for already approved contributions).
Each clash can be rated after the end. This happens for all modes (fastest, shortest, reverse) separately. When a minimum amount of votes is reached and the average vote turns out to be a negative feedback, the clash will be removed (at least in that one mode).
See the comment section of this contribution for instance. The CodinBot deactivated one mode.
Technically if you follow the rules, you should never approve a puzzle you haven’t solved yourself. So we’re talking more about an extrapolation than a estimation.
This rule is currently optional however and that’s more of a problem imo. The author’s solution should be hidden until the puzzle is fully approved, and you should have a solution passing all the tests/validators to be able to approve (CoC or not).
Oh yea, I forgot that evaluation has a ‘difficulty’ point.
The solution is extremely obvious then:
Bot filters some more saddle “too difficult”/“too easy” cases, moderators reject obvious “extremely difficult”/“extremely easy” cases.
So just change “no coc puzzle is too easy” to something like “CoC puzzles is meant to be easy, quick and fun. But if it’s too easy, it becomes too quick and then it becomes a game of waiting for another round to start.” And add something like “would take longer than X minutes to solve” to the approval checkboxes. Ideally, X would be calculated based on the approving moderator rank and number of games played but something like “would take longer than a minute for average 6000-8000rank player to solve” will work too.
There may be difficulties with that tho. Mostly for the reverse where you may sit for 15 minutes figuring out what the trick is and then when you get it puzzle feels like the easiest thing in existence. Of course, a moderator would be able to reflect on that and base his evaluation of difficultness on the actual time spend and not on their feelings of the puzzle. Maybe it needs to be written explicitly that you shouldn’t try to imagine how other players would approach the puzzle and instead look at your time, try to interpolate, and then include a lot of room for error.
That’s why I’m saying that we need different guidelines for different genres.
At least make some separate recommendations for Shortest, Fastest, and Reverse.
But it’s not that important tho.
What more important is that “solution time” of the problem includes reading the statement. What if the statement is complicated but as soon as you understand it the solution is clear. Is the puzzle too easy?
I’ve partly described that problem and the solution to it in point 3 in my post above. TLDR: we need to get rid of the complex wordy statements.
Thanks for darkhorse64 and Djoums for pointing out some obvious things that I’ve missed.
I do now agree that some expert advice may be more valuable. Mostly because I’ve started to look at the rank of commenters and oh boy, there is some interesting correlations there.
I do still think that it brings elitism in at least when it comes to difficulty. It’s may bring some subtle problems like
Yea, I agree that you should be able to approve only if you have a working solution. And that is a restriction only if the intended solution is hidden.
But there should be an option to show the solution. However, if you choose to show it you can’t approve the puzzle, your only options are to reject or to move by.
I suspect that any version of that is hard to implement and it’s easier to just trust the reviewers.
Just hiding solutions to Reverse puzzles shouldn’t breake anything tho
I agree with that has been said, especially regarding the solutions.
It should be made mandatory for moderators to provide a working solution in time (with a timer, just like a real clash) before being able to accept a contribution.
Of course the author’s solution should be hidden until that requirement has been fulfilled.
I see many contributions accepted with comments like “should be fun to solve”, “looking forward to solving it”, “seems interesting”, and it’s really unacceptable when you see how many clashes have bad test cases or incomprehensible statements.
Hello! I am relatively new to contributions. I have been using the site casually for a while but the past two weeks I have been active on clash of code reviews. Here is some of my feedback
Reviewers should do challenge
I strongly agree with the above sentiment that in order to give feedback on a potential clash of code a user must complete it in the given time window. Additionally, only those who solved it in the given time window should see the author’s solution, yes. The ability to solve the problem in the given time window is a strong signal of both difficulty and clarity.
Binary solutions are bad design
I strongly dislike when a CoC has binary output for tests. Not just true/false, but when I need to output one string or another. I dislike that competitors can submit “print(‘true’)” and get half of the test cases correct. Puzzles should be written in such a way that the output should have a larger range of possible outputs.
An example puzzle: Is the largest number in these lists the same?
1 2 5
1 5 3
5 -2 0
Instead of outputting ‘true’ the puzzle should request the largest value, 5. Instead of false, the puzzle could request one of many things. The row numbers(zero index) without the highest number
0 1 9
0 0 0
0 2 3
output “1 2”.
This example is somewhat contrived but the point is puzzle prompts should be made to reward partial credit and not incentivize guessing.
Shortest should not reward 1 character submissions
On shortest mode, all players who submit a 0% solution should tie for last. Currently, there is incentive for a player who believes they cannot do the clash to submit early. If they submit a 0% with 1 character first they will out do anyone else who does not solve the problem. CoC are low time investment, players should be rewarded for doing the best they can.
Stats on your submissions
I would love to see some statistic about my CoC contributions. These can include, % of players who answered correctly. Distribution of time to completion. Distribution of response correctness%. Distribution of shortest 100%. I am not sure what is feasible for the developers or how difficult this would be to add, so I understand if this does not happen.
Thank you for reading my post, I look forward to your thoughts.
@lopidav@esteban616 and others, thank you for the suggestions and ideas. I agree with you for most of the things:
agreed, players/moderators are biased and view this differently. That’s why I don’t think easy problems should be refused, as the guidelines recommend. On the other hand, hard CoCs shouldn’t be refused either; they can be released as easy/medium puzzles instead.
In any case, a too easy/hard CoC will probably have bad ratings and will eventually get removed by the moderation bot.
agreed, this should be avoided too. Even though programming is about taking the right tool to tackle a problem, we probably can define puzzles that don’t favor too much one language against another. I can add a guideline about that but I believe this is already discussed in contribution comments
Complexity of statement
Not sure I get your point. Same issue as in 1. People are biased and read statements differently. In CoC, we recommend not adding a story because it’s useless for most players. Again, I believe this is discussed in the comments too.
Statement should be self-sufficient (no need to search for a definition or formula)
Agreed. I can add a guideline
All players have different preferences and it’s great we have so many different subgenres. In any case, badly rated CoC/puzzles get removed by the moderation bot.
Reviewers should solve the puzzle
Yes, declaring it is not enough imo. It implies a few changes though, including the hiding of the author’s solution (do we really need it visible to all btw?) However, I don’t think it should come with a timer to check the difficulty. Overly complex for something that can be bypassed easily.
Agreed, it should be avoided
About bugs & improvements:
8. Shortest solutions at 0% should tie
Agreed, although it seems to me the impact of this issue is very low
9. Stats on contributions
Agreed. We thought about this last year. We’ve been pretty busy with the quest map, the learning vision, certifications (and also the challenge) so we decided to push back this kind of improvements.
6 I do not think author’s solution needs to be visible, and certainly not to those who have not solved it. I would prefer a time to be attached to the solution/approval. The reviewer should not be stopped at 15 minutes though.
7 I know this too is low priority, but this could be checked on submission. Confirm the range of outputs is >2.
8 & 9 I understand these are not urgent concerns but I would love to see them addressed eventually.
If you require a solution to approve, some of the tougher community puzzles might get stuck in process longer than you like. You might consider just passing with a single approval if the person has a high reputation (like the chat moderator pool, or maybe the top X in CP).
What are those “contribution comments” you’re talking about?
Here is my response to each point:
Extremely hard and extremely easy coc puzzles 100% should be refused.
The difficulty is tied to fun and you should reject non-fun puzzles.
That’s what you used to argue against Boulet’s extremely easy puzzles.
I can create extremely hard puzzles to prove my point if needed.
In classic puzzle form, it can live of course because players can choose what they play. In coc you shouldn’t be given a problem so easy that it’s ‘fastest typer.’
The problem is that we already have refusals with reasoning like ‘easy’/‘too hard’ and we will have that even if guidelines would change.
But the guidelines can give power to the contributors that read them. Do you want to give them the power to create puzzles like Boulet’s?
Yes, we can skip all of that by putting work on the bot but the bot is already working and I still get those puzzles that take 15 minutes to solve by <300rank players.
Also, do you have data from the bot on how many puzzles are filtered because of difficulty and how many clashes it took? If it takes 15 or fewer frustrated players to filter out too easy/hard puzzles then I may agree with your solution.
Ok, I don’t know what are those “contribution comments” so I’ll not answer that point yet.
I have to add tho that there are obvious language-specific coc puzzles but there is also stuff like ‘True’/‘False’ and ‘true’/‘false’. And it needs to be clear that the second type should be avoided too if possible.
Reviewers should solve the puzzle
Looks like all of the suggestions under this accumulated in “make a beta for coc with pending puzzles and ability to approve when if you solved it 100% under 15 mins.”
That would be very cool but sound hard to implement.
What is not hard to implement is hiding the solution/validator by default a showing it on mouse click
I would just hate to see great ones like Space Maze have trouble getting approved, and discourage people from making more like that. If someone like you solves and approves it, there really is no need to look for two more opinions.
That reminds me.
Maybe Contribution author name should be hidden as well.
“I like you so I’ve approved” is present enough for JBM to start satirizing it. Yet no one else is talking about it?
I guess contributions like that are often approved quickly so not a lot of reviewers see it but
I would also suggest that all new contributions be required to be a WIP for at least X days (to be determined). I see many clashes that are accepted the same day they are first published. Most people do not check new contributions everyday, therefore these clashes receive very little feedback and the moderation is not representative of the community.