Optimization Validators private vs public

Puzzles
#1

I had a few exchanges in chat lately regarding this topic.

I believe that In some case, private validators are quite mandatory when the point is to devise an algorithm that is able to adapt to a kind of problem rather that to a specific set of data. Like NN image recognition contest and the like.

On the other hand, quite a lot of the optimization problems available in here are very specific cases that doesn’t have much if any direct generic applicability.

To be competitive in those optimization puzzles such as for instance CvZ, you just actually cannot compete if you don’t use the final validators as optimization target.

While Mars lander’s validators are given as test cases, CvZ validators are hidden and require a daunting amount of time to generate those out of some boring trial and error process.

I understand that you are not supposed to… However you just cannot ever be competitive if you don’t do so…

For not only you are competing from behind without much clues on the validators but you also have to do it in real time while every top ranked player likely ran their optimization algorithms for hours on the actual final validators…

And that is actually why I stopped working on optimization puzzle… Because the sheer amount of time to spend just to rebuild the validators in order to be competitive is plain boring.

I’m thinking that this keeps a lot of people from even competing.

Regarding this, in my opinion, every optimization puzzle validators should be made public.
Simply by just setting them as test case like in Mars Lander.

I would also add that regarding a “stabilization” of optimization puzzle in case of making validators public that this “stabilization” has already happened…

Beside the fact that you kinda have 2 ranking mixed into just one : The ranking for people who used final validators optimization and those who didn’t…

On a note, making those old puzzle validators public could also create some incentive for people into joining the competition :slight_smile:

#2

I don’t agree with you.

On the other thread you wrote, that you couldn’t finish in time and would have needed 1 or 2 more hours. At the same time you want to optimize offline, which requires to finish earlier in order to let the computer search for a while. You may even rent some servers, which results in an inequality between competitors, a better code might lose because it didn’t run long enough.

Moreover at the resent contest (which I admit is quite special in that matter), I saw handcrafted solutions that were almost 100 points better than my best on a given testcase, while I already broke the 10k score. So you don’t only compete vs other algorithms but also the human brain.

Instead I would like to see more validators. At Yandex Algorithms we got 100 testcases during the round (I’m fine with less here) and 500 in the recalc. They also provided a test generator.

#3

The issue with optimizations puzzles is that the validators will be discovered one day. You’ll always have someone who will find them. At this point, you know that one day the top 10 will be hardcoded solutions.

This is even worse for old puzzles (Cotr). Because validators were publics and one day codingame decided to hide them. This is outrageously unfair for new players. So everytime someone is asking for Cotr (or CvZ) validators, i just give it to them.

The second issue is that without any other feature, the optimizations puzzles are only a “spam submit of my random code until it is good enough”.

My opinion on this is that if we can’t prevent someone to find the validators someday, just give them. Or one day the puzzle will be unfair.

If we want to keep them hidden, codingame will have to code something. Random validators every weeks (with a full rerun of all codes) maybe. Or just generate 1000 random validators for each submit (but i will not prevent spam submit with a random code, of course).

2 Likes
#4

The other post you are referring to, is about a contest.

This post is about puzzles. Those are two different things.

There is just no time limit on puzzles and that’s a part of my argument in this post, I suspect you didn’t read it completely.

Now if you want to argue about public or private validators on contests that is another problem.

I would say that there also are some argument for validators to be public on some contest as well.

If in case of something like image recognition it wouldn’t make any sense, in some case it could be a good option in my opinion.

For hidden validators might suits some algorithm better than some other and it’s often not fully based on the overall algorithm performance but rather on the specifics of those validators.

The option of having more validators / more runs is a problem of computing power requirement on CG’s end, having a lot more validators would require a lot more computing power or at least a lot more time to complete a rerun process.

Now from a pure balance perspective it would be better indeed.

Regarding the lack of time on my previous contest, well that’s something entirely different related to the amount of time I chose to invested in the contest and to my time management.

It’s a completely different kind of issue which is unrelated to my point in the current post.

2 Likes