"Killed" on C++ memory reservations

#1

I’m having some problems when trying to send some bots (Hypersonic).
I tried both sending the bot in plain C++ or precompiled.
In the first case I’m seeing: g++: “fatal error: Killed signal terminated program cc1plus”, randomly

With the precompiled the server kills the bot, and writes on standard error:
“Killed”


Sometimes it just dies before starting, so I don’t see neither KIlled nor any of my cerr

I never write “Killed” on cerr, that comes from out of my code.
It happens randomly. May be because I try to reserve a bit chunk of contiguous memory ( new array[256MB or more]). But I also tried a reserve with noexcept to avoid error when no memory is available but the “Killed” message keeps showing, killing my bot without any warning.

I don’t know what to look at.

#2

I’m pretty sure it’s the contiguous allocation that failed because the system could not supply it. I only ever had the “Killed” message when trying to allocate too much memory in a single chunk.

1 Like
#3

I’m also having issues (on oware and yavalath), seeing the cc1plus error and also getting the message below. I’ve not seen Killed, but I do the majority of allocations as static ones which may explain the different messages (it runs 90% of the time, so I don’t think I am hitting any intended limits).

'Code failed: your program was terminated before reaching the main entry point for your language
(possible reasons: segfault on static initializer, too much memory used, etc.)

1 Like
#4

Seems like that’s also what would happen when the executable loader cannot allocate the address space for your program, which includes the size of static variables.

#5

Yesterday I read some people in chat with cc1plus problems too.

I tried to reduce more the memory to about 156MB, and split it in smaller chunks (4.5MB each reservation and 1 bigger of 9MB)… and I keep seeing the “Killed” thing.
I tested a precompiled bot (to reduce compiler issues), and I got 9 timeouts/crashes on 80 tests this morning. Most of them don’t show any log or error, just a first turn timeout. I can’t see my “INIT CACHE” text, the first thing the bot does. The rest of crashes are “Killed” when reserving 32x4.5MB memory, I see the debug texts just until that point.
“new nothrow” shouldn’t fail on a memory assigment (http://www.cplusplus.com/reference/new/nothrow/) it just returns a null pointer so the programmer can act accordly. But the server is directly killing the process without warnings.

I precisely changed from big static arrays to dynamic memory to avoid limits on static vars. I hardly think my stack vars take more than 300KB… (5x100x4bytes and 13124*4bytes are the biggest ones)

#6

I haven’t made the post yet, but we changed from C++ 8.3.0 to C++ 9.2.1 at the beginning of the week. Could it be related?

3 Likes
#7

Almost all my codes don’t compile anymore. Ouch … :smiley:

After investigation, it appears that the following code will not compile :

#pragma GCC target("arch=core-avx2","tune=core-avx2")
// ...
#include <iostream>

I have to include <iostream> before this pragma to be able to compile. A little weird.

2 Likes
#8

To me very much related, I have some batches from 09 Nov and days before, and all of them have 0 timeouts.
Then I started doing some tests from 12 Nov onwards, and all of them have timeouts. I assumed my code was failing, that’s why I changed to dynamic memory.

#9

Do you have any idea what in the update from 8.3 to 9.2 could cause these errors? Any information could be useful.

Thanks and sorry :confused:

#10

Is it possible for the setup of the worker to somehow have less (contiguous) available memory since the update?

#11

In the multi I was testing (HS) I think I’m seeing two different errors:

  • g++ Compiler related errors at turn 0: cc1plus errors, bots that timeouts without any message, and some other weird messages (something about unmap destructors).
  • Memory related errors at turn 0: I’m having it also with vectors, filling with zeroes a 64 x vector.resize(2.4MB). These errors have the message “Killed”, and now it seems it’s always dying at the same specific point (like on the 46th clear of the array).

I installed locally the compiler g+±9 (Ubuntu 9.2.1-17ubuntu1~16.04) 9.2.1 20191102, and I run continuously 300+ matches without any error (yet I ignore timeouts or memory limits).

Then I used that precompiled code and I sent it as C++, and it showed both problems. But if I send the very same binary packed as a C# code I’m only seeing the second type of problems (memory issues). If I reduce the memory usage of vectors to 99MB it no longer crashes as “Killed”. Compiled binaries just run fine, it’s not an incompatibility of g++ 9.2 but the compiling process + something else.

Maybe the new compiling process takes more memory (check ulimit -a of the compiling user) or time and that is breaking the process. Or there is just one rogue server that is broken and makes that randomness.

1 Like
#12

@Marchete let me ask you a question to see if I understood correctly: what would happen if you compiled using g++8.3, without reducing your usage of vectors and then send the binary packed as C# code. Would you still get the memory issue or not? If not, that would mean that g++ 9 changed something in a bad way, no?

#13

Unfortunately no, it’s not related to g++ 9 in my example:
This is a minimal example bot:


Play with this C++ bot on Hypersonic, on a 4vs4, use the same bot on all players. After a couple of attempts (<7) eventually one of them will crash at turn == 0 while setting the arrays to zero (I write a counter on cerr). /EDIT: All bots auto suicide at turn 3, to make tests faster. What I was looking for are timeouts and errors at first turn. Ignore these timeouts as they are normal.

But the bad thing is that it also happens on a precompiled code packed as C#:
I used clang+±6.0, a completely different compiler, then packed to a C# and it shows the same behaviour, player 4 dying:


In this case I see the “Killed” text:

INIT CACHE 142187KB CHUNK3906 SIZE:749952 524288

ABCDEF0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,Killed
2 Likes
#14

The timeout problem at turn 0 also happens without any pragma.
I can’t reproduce it with the code converted to real C# (not packed binary). I don’t know, maybe it’s my C++ code but it seems “safe” for C++ coding standards, no defines,pointers,casting or weird memory tricks. Maybe it’s a library of the 9.2.0 version that both clang and gcc use? I don’t know.

1 Like
#15

It looks like there is something wrong with the runtime

#16

For the sake of the experiment, we rebooted all servers. Let me know if the issue is still happening. If still happening and happening randomly, please paste a few replay links exhibiting the faulty behavior (we can track which server executed a given replay). Thanks.

2 Likes
#17

Just tried a submit on yavalath and have had 0 timeouts in the first 50 games (which would have been very unlikely yesterday).

Supson is still losing every game on turn 1 (I’m assuming he has a different issue where the update has broken his code).

Edit: Apparently I posted too soon. https://www.codingame.com/share-replay/422018599 (terminated before reaching main entry point of program). I still think it’s much improved from yesterday.

1 Like
#18

I have a lot examples in uTTT. it never happened 3 days ago.






and i can continue with the list

2 Likes
#19


Always the red player. Both times crashing when doing the resize of the vector.

May be some weird corner case. If it only happens to me there is no big deal, but I’m puzzled for the randomness of timeouts.
In my defense I must say the ol’ classic “but it worked on my computer!”

#20

Same for me. Cant to reproduce on my computer and it did not happen to same code 3 days ago.

1 Like