Timeouts with C++ in CSB (might be more general)

#22

So according to the graph is better to submit at 10pm :smiley:

I think Compilation errors might be discarded because I never saw any compiler error in these (no line error, or warning, just an empty error or the mount proc one). It’s the same tested coded that compiles fine on CG IDE, on my local GCC and Visual Studio. It also goes fine in the rest of matches except when the dreaded proc error happens.
I saw the same behavior than Agade, some submits have crazy high failures at start, with up to 5 crashes in the first 15 matches, all of them at turn 0. It’s annoying as hell, lately I just give up trying to have a clean submit without any mount error in first 20 matches.
I can’t infer about the execution part, but having 60ms in it and tested with a lot of code guards for possible overflow/errors I don’t get assert errors. I’d say it’s in that umount/mount phase.

Also I can’t know if it’s a problem from compiled langs only, in multis I usually am C++/C# masterrace

#23

Code failed: your program was terminated before reaching the main entry point for your language
(possible reasons: segfault on static initializer, too much memory used, etc.)
/usr/bin/stdbuf: failed to run command ‘/tmp/Answer’: Permission denied

That doesn’t sound good

Code failed: your program was terminated before reaching the main entry point for your language
(possible reasons: segfault on static initializer, too much memory used, etc.)
/usr/bin/stdbuf: failed to run command ‘/tmp/Answer’: No such file or directory

No file?

In Hypersonic, but I don’t think that matters.

Code failed: your program was terminated before reaching the main entry point for your language
(possible reasons: segfault on static initializer, too much memory used, etc.)
“/tmp/Answer”: not in executable format: File truncated
No executable file specified.
Use the “file” or “exec-file” command.

#24

Some news regarding the “mount proc”:

  • We did some fixes in the code. Please @Marchete and others, could you check if we improved anything or not.
  • As we trace these errors now, I can tell you that the numbers of these errors is very variable. We can have a full day without one and then all of a sudden we can have many over a short period of time.
  • So if you still have the errors, please let me know which code you used on which game as I now believe this is tied to particular code (although I cannot understand why it would be).

Regarding the “/tmp/Answer’: No such file or directory” or ““/tmp/Answer”: not in executable format: File truncated” issues, we are investigating. This can be due to two things: either compilation takes too long and our system kills the compilation process or the compilation takes too much memory and again our system kills the compilation process. Again, being investigated. We see it happening for C++ and Swift programs.

Regarding the “’/tmp/Answer’: Permission denied”, no idea. Probably a bug in our compilation caching system but no clue as to why for now. Anyway let’s fix the above problems first.

#25

And we moved the allowed compile time to 20s for all languages. Should remove the “No such file or directory” error as we confirmed that the 10s limit was the issue there.

#26

It seems that these changes are working. Before them I struggle to have 10 first matches without a timeout. Now I have tested 3 resubmits and I haven’t seen any in the first matches.
About the compilation time, in C++ you can have code that precalculates stuff at compile time, or maybe the compiler does it by itself because it’s optimal, but in local GCC barely takes 2 seconds to compile my code. I have no idea how to control memory usage at compile time.
If there is still some corner case of the problem (maybe some weird race condition) it could me enough to just make a single retry of the process if some error is detected during mount phase.

#27

I am glad we fixed things.

Regarding compile time that’s ok, some code were near the 10s limit and sometimes it would take just over 10s. With a 20s limit it seems to be ok for all languages (except Scala for which we still grant more time).
We ruled out the memory problem at compile time.

With the next release when we detect a “mount” problem or any other internal problem, the game will be flagged as “in error” and retried 5 minutes later (plus, we get an alert). This “error flag” mechanism is not new but it was not applied to our C system code that would do the jailing, mounting, etc. So it means that in case of an issue, either it is a transient issue that will get solved by itself or we will be spammed with alerts and forced to look into it.

We’ll also put a probe into the “/usr/bin/stdbuf: failed to run command ‘/tmp/Answer’: Permission denied” issue to try to understand it and fix it.

I think we can finally go back to the initial timeout issue (spiking issue). A more complex problem… (next culprit in sight: systemd which consumes CPU all over the place when mounting partitions)