Parse SQL Queries was not a difficult puzzle. I changed it to Medium because that’s typical for parsing, and also because there seems to be an unwritten understanding that it isn’t actually Easy as it was classified and as the statement itself had claimed. When a new user was having trouble, one of the discussion comments was to “try an easier puzzle”. So there’s definitely a disconnect between puzzle ratings as documented versus what the community believes. In my opinion it would be better to determine this by consensus rather than leaving it up to the author. However, in the case of Parse SQL Queries that’s a minor point, as there were much bigger issues. And from this we should conclude that automatically adjusting difficulty based on completion rates would not be entirely accurate. While it’s true that 12% is an exceedingly low success rate, Parse SQL Queries is not a Very Hard problem just because it had design issues.
It’s too early to know if my changes will have a significant impact, but I believe that the fundamental problem with the way the puzzle was written is that it was too open-ended. This is illustrated by a seemingly innocuous link to the W3Schools documentation for SQL queries, whose use would be overwhelming for someone knowledgeable of databases, let alone anyone new to them. It is not good introductory material, and it is neither appropriate reference material for this very restricted puzzle, which did not even have multiple conditions using AND and OR despite making reference to them as if it did. There were also superfluous references to row indexing, dropping tables, and numerical columns which were not explained. Concerning the latter point in particular, I was able to simplify significantly with only minor changes to one test case because the validators did not actually test certain situations. It would have been a better puzzle to also include sorting on string values and comparisons on numerical values, but because it was so lightly documented and furthermore not already sufficiently tested, I could not introduce or extend these in an already existing puzzle. In short, I spent a lot of time tidying up little details in order to better guide the user, by approaching the puzzle from the standpoint of someone trying to solve it instead of someone trying to learn SQL.
While hopefully all of these changes will be beneficial, aside from one or two I do not believe most to be critical. However, there was one very simple change that I’m sure could have had a massive impact on the perception of the puzzle and therefore determination to complete it once started. For some reason the validators had been given names which made it seem that they were testing for more advanced features such as “ternary operators”, which of course were not present in the test cases and not described in the statement. In truth they were not tested in the validators either, but since these are hidden to the user they really shouldn’t be expected to know. It’s quite possible that a lot of people quit after failing validators for reasons they could not anticipate, or gave up when reading that the last test case was gargantuan when it was actually not all that much larger and in fact simpler than the one before it. Furthermore, five tests/validators is not enough coverage even for a single aspect of the puzzle like sorting, let alone for the open-ended nature of the problem as presented. Things not tested for include different spacing, different order of WHERE and ORDER BY, different casing, multiple WHERE, multiple ORDER BY, etc etc etc. It certainly feels that a lot of motivation could have been lost when programming features where the extent is not clearly defined. Then seeing these validator names would have been enough to throw in the towel.
My experience in editing this puzzle is that it is too difficult to plan without just jumping in and going for it. There are simply too many small changes to enumerate, including things that are clearly broken like the formatting, more subjective decisions like which term to use when aiming for consistency across the puzzle, and dozens of other minuscule changes where it’s too much trouble to stop and ask, does this actually need to change for the better or would it be fine to leave slightly awkward as the author’s original intent? Of course I did hold back from rewriting most of the text, but I’m guessing I wouldn’t have uncovered many of the issues that were resolved if I had held back too much. On the one hand, I feel like I could have improved the puzzle much more if given freedom to change the problem itself, instead of being restrained by existing test cases. For instance, I could have created a new puzzle in the same vein. Yet I know that would also have taken a lot longer. And at the same time, the issues that were most pressing I can count on one hand. So I’m really at a loss for any recommendations on how to move forward with problematic puzzles like these, and I will not repeat the exercise. Except that I do think moderators should continue to make alterations when it is apparent that they are beneficial, and probably that’s a sufficient condition rather than the puzzle actually being broken in some way. In particular, mischaracterization of the difficulty level and incorrect validator names did not actually break the puzzle, but it was apparent from the comments that they would have been beneficial fixes. At least, if these came up in any of my puzzles, I hope I’d have the resolve to correct them immediately.