So according to the graph is better to submit at 10pm
I think Compilation errors might be discarded because I never saw any compiler error in these (no line error, or warning, just an empty error or the mount proc one). Itâs the same tested coded that compiles fine on CG IDE, on my local GCC and Visual Studio. It also goes fine in the rest of matches except when the dreaded proc error happens.
I saw the same behavior than Agade, some submits have crazy high failures at start, with up to 5 crashes in the first 15 matches, all of them at turn 0. Itâs annoying as hell, lately I just give up trying to have a clean submit without any mount error in first 20 matches.
I canât infer about the execution part, but having 60ms in it and tested with a lot of code guards for possible overflow/errors I donât get assert errors. Iâd say itâs in that umount/mount phase.
Also I canât know if itâs a problem from compiled langs only, in multis I usually am C++/C# masterrace
Code failed: your program was terminated before reaching the main entry point for your language
(possible reasons: segfault on static initializer, too much memory used, etc.)
/usr/bin/stdbuf: failed to run command â/tmp/Answerâ: Permission denied
That doesnât sound good
Code failed: your program was terminated before reaching the main entry point for your language
(possible reasons: segfault on static initializer, too much memory used, etc.)
/usr/bin/stdbuf: failed to run command â/tmp/Answerâ: No such file or directory
No file?
In Hypersonic, but I donât think that matters.
Code failed: your program was terminated before reaching the main entry point for your language
(possible reasons: segfault on static initializer, too much memory used, etc.)
â/tmp/Answerâ: not in executable format: File truncated
No executable file specified.
Use the âfileâ or âexec-fileâ command.
We did some fixes in the code. Please @Marchete and others, could you check if we improved anything or not.
As we trace these errors now, I can tell you that the numbers of these errors is very variable. We can have a full day without one and then all of a sudden we can have many over a short period of time.
So if you still have the errors, please let me know which code you used on which game as I now believe this is tied to particular code (although I cannot understand why it would be).
Regarding the â/tmp/Answerâ: No such file or directoryâ or ââ/tmp/Answerâ: not in executable format: File truncatedâ issues, we are investigating. This can be due to two things: either compilation takes too long and our system kills the compilation process or the compilation takes too much memory and again our system kills the compilation process. Again, being investigated. We see it happening for C++ and Swift programs.
Regarding the ââ/tmp/Answerâ: Permission deniedâ, no idea. Probably a bug in our compilation caching system but no clue as to why for now. Anyway letâs fix the above problems first.
And we moved the allowed compile time to 20s for all languages. Should remove the âNo such file or directoryâ error as we confirmed that the 10s limit was the issue there.
It seems that these changes are working. Before them I struggle to have 10 first matches without a timeout. Now I have tested 3 resubmits and I havenât seen any in the first matches.
About the compilation time, in C++ you can have code that precalculates stuff at compile time, or maybe the compiler does it by itself because itâs optimal, but in local GCC barely takes 2 seconds to compile my code. I have no idea how to control memory usage at compile time.
If there is still some corner case of the problem (maybe some weird race condition) it could me enough to just make a single retry of the process if some error is detected during mount phase.
Regarding compile time thatâs ok, some code were near the 10s limit and sometimes it would take just over 10s. With a 20s limit it seems to be ok for all languages (except Scala for which we still grant more time).
We ruled out the memory problem at compile time.
With the next release when we detect a âmountâ problem or any other internal problem, the game will be flagged as âin errorâ and retried 5 minutes later (plus, we get an alert). This âerror flagâ mechanism is not new but it was not applied to our C system code that would do the jailing, mounting, etc. So it means that in case of an issue, either it is a transient issue that will get solved by itself or we will be spammed with alerts and forced to look into it.
Weâll also put a probe into the â/usr/bin/stdbuf: failed to run command â/tmp/Answerâ: Permission deniedâ issue to try to understand it and fix it.
I think we can finally go back to the initial timeout issue (spiking issue). A more complex problem⌠(next culprit in sight: systemd which consumes CPU all over the place when mounting partitions)
@_CG_XorMode I am currently experiencing an abnormally high amount of CSB games crashing on the very first turn with this error message: /usr/bin/stdbuf: Resource temporarily unavailable
When trying to see if Python3 was the culprit, I obtained a similar one using C++:
Code failed: your program was terminated before reaching the main entry point for your language
(possible reasons: segfault on static initializer, too much memory used, etc.)
/usr/bin/stdbuf: Resource temporarily unavailable
This is a very common occurrence, as my last submission had 13 of its 44 losses (~30%) directly caused by a game crashing on the first turn. An additional 2 losses were very suspicious timeouts, which could indicate the spiking issue is back as well, and possibly another symptom. I can also reproduce the crash on first turn maybe once every 10 games in IDE.
So far I have only obtained a few clues from my investigation. This crash seems to happen way before any of my code gets executed, as I have been unable to retrieve any logs when it happens. However, a key piece of information I later discovered is 100% of those occurrences happen as player 1. Not only I have been unable to reproduce this crash on my bot as player 2, but I have been able to reproduce the crash on another opponent as player 1 (see replay).
This seems to be a recent regression, as I have not had this problem at all until today. I will happily provide any more information or assistance to track down and fix this issue. This is severely impacting my progression at the moment. Thanks!
EDIT: Revamped the post to match my investigation progress.
Can confirm the /usr/bin/stdbuf: Resource temporarily unavailable error. This happened several times to myself yesterday (C++, BR2048), also had some suspicious timeout spiking of ~70ms(!) sporadically over the past week or so both in CSB and BR, I havenât checked any other multis.
I aldready had this message but in a different context.
Create a executable file (a .sh file should be enought) then create a program where you spawn some process executing your file. Do it in a multithread program. Some of your process will fail to spawn with the following message : myfile.sh: Resource temporarily unavailable.
I canât be sure, but i assume that codingame try to spawn multiple child process at the same time with the command stdbuf. Or maybe the file is locked by something else.
Thank you for reporting this and sorry this has been happening. It seems to be linked to an issue we had with our code machines yesterday. Weâve since rebooted them manually so you should not be having the same issues today.
If this is indeed the case, weâll regularly reboot machines to avoid this kind of problem from now on.
I did not try yesterday, but there is still some on C4L /usr/bin/stdbuf: Resource temporarily unavailable (the only multi I tried), i donât know either if the frequency is higher or not.
So i guess itâs still occur on multi as well.
EDIT: weâve further investigated and discovered more machines affected with the issue. We rebooted them and will add a warning notification if it happens again so we can take action. Weâll also look into the root cause: zombies (donât ask me more )
Thanks!
)