[JAVA] JVM memory issues

[Neumann]

Hi,

Today, when I tried to implement a new algorithm (MCTS) for the ongoing contest, I faced weird timeout issues.
My code was taking up to 50ms to call a simple function that just does a 6*12 array copy.

I then decided not to lose too much time and tried something else. I quickly ran into similar issues. But this time, I figured out where it came from.

In this second attemp, I use a static List of custom objects. These objects contains an integer, and a List of another custom object that contains 4 integer, up to a total of 5 integers, plus the List structure.
I clear then fill this list every turn, with about 20k of those objects.

That’s not a lot, but apparently it is enough to make the JVM freeze.

The FAQ says that 768Mb of RAM are allocated to our code, but when I call Runtime.getRuntime().totalMemory() and ***Runtime.getRuntime().maxMemory()***, I can see that 30Mb is allocated, and that it can lazily allocate up to 494Mb when necessary.

I’m not a JVM expert, but when I print Runtime.getRuntime().freeMemory() every turn, I can see the GC cycles, and I can see that sometimes, there is close to no memory left.

My theory :
It seems that at each turn, the GC free less memory than my code needs, and 95% of the time, after about 10 turns, the JVM is trying to reallocate some more (maxMemoty = 494Mb). But this reallocation produces a ~50ms freeze, making my code timeout.

The algorithm I was talking at the beginning of my message was also using a relatively big amount of memory, but again not that much.

Now, a few questions :

• Is my theory relevant ?
• If so, are you aware of that issue ?
• Can you change the JVM settings ?

I know Java isn’t the best language for such ressource-heavy algorithms, and I am used to deal with CPU limitations; but this is much more annoying.
It makes impossible for Java developper to use a lot of useful algorithms.

I’m looking forward to your answers
Thanks for reading.

tl;dr; The JVM seems to allocate only 30Mb of RAM. If more than this is used, the program crashes, and you can start crying. I’m learning C++ for the next contest, since it’s the only language in which you can really write efficient algorithms.

[Aries]

When one says that C++ is better to manage memory, one usually refers to variables auto(matically) allocated and released on the stack, that Java only uses for primitive types. Regarding dynamic variables allocated on the heap, the situation is basically as bad in Java as in C++.

To handle your situation, the classic solution would be the same in C++ as in Java: manage yourself a memory pool of the objects you are using. In fact, Java latest GC could almost free you of this by being smart enough (but doing it by hand remains faster, at least more responsive, I guess).

Since growing the totalMemory (up to the maxMemory) could induce freezes, using a pool will allow you to pay for it only once at init time (I think I’ve read somewhere that bots have additional time the first turn).

Note: freeMemory is a bit misleading in Java, since it is totalMemory - usedMemory and not maxMemory - usedMemory. The memory still available is in fact maxMemory - (totalMemory - freeMemory).

[Neumann]

I know that it is the same in C++, but there are two differences :

• In C++ you can manually manage the memory as you wish, you cannot do this in Java. When the GC isn’t synchronised with your program turns, it can lead to annoying behavior and crashes.
• Even if I could manage the memory myself, I still would face the same issue. My MCTS node objects weight 6080 bits (could have been optimised tho), that means that I cannot store more than 5000 of those alone in 30Mb without having a freeze.

30Mb is too short, even with a lot of reuse and optimization.

[Neumann]

I just figured out a way of forcing the JVM to allocate the 494Mb in the first turn, when freezes does not matter :

if (turn == 0) {
	FOR_ALLOC = new Board[2000000];
	for (int i = 0; i < 2000000; i++) {
		FOR_ALLOC[i] = new Board();
	}
	FOR_ALLOC = null;
	System.gc();
}

I then have totalMemory = 464Mb. My memory-greedy algorithm that used to freeze after 10 turns now runs like a charm (EDIT2 : false hope guys … It still crashes like crazy, but in a different way. Nevermind.).

EDIT The FAQ says 768Mb btw, you might want to update it

[selslack]

I didn’t dig this deep. But I can confirm that my Java code crashes sometimes due to timeouts even if I have 85ms limit in my loop and each iteration takes less than 0.1 ms.

@Neumann thx for the hack, will use it.

P.S> Don’t know if its relevant or not – CG’s servers or JVM’s are ~2.5x times slower than my local 3GHz Haswell core. But I just accepted it.

[egaetan]

JVM do a full GC ( stop the world) before reallocating additional memory, so yes force the initial allocation in the warm up tour is a good solution

[Neumann]

Little update :

Even though the JVM says that 464Mb of memory is available, I still encounter the same crashes, in the same way that before :

Free Memory:343
Free Memory:330
Free Memory:319
Timeout: the program did not provide 1 input lines in due time... Neumann will no longer be active in this game.

95% of all of my objects are pre-allocated during the first turn. If I cut the number of objects created, the program stops to crash. There is definitely an issue.

[selslack]

I didn’t notice any crashes during the previous contests despite I used MCTS for teh zombies.

Maybe CG made some changes recently.

[ElanVB]

I am not sure if this is the same issue but I am getting the following error:

“ERROR: ld.so: object ‘/usr/lib/coreutils/libstdbuf.so’ from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread”

I see this happening in my IDE and I see that I am randomly loosing some battles without my program doing anything so I assume it has to do with this…

Any ideas?

[tyrcho]

java.lang.OutOfMemoryError: unable to create new native thread"

I have exactly this same error in maybe 1% of games, at round 1. I’m using scala which also runs on the JVM.

I saw it in CG IDE and in my replays.

[selslack]

Just got this error:

ERROR: ld.so: object ‘/usr/lib/coreutils/libstdbuf.so’ from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread

Never had it before in any costest.

[Aries]

I’m afraid the trainee who changed the Docker service configuration just before the contest is actually having crash courses in brasse sicilienne.

[int33h]

Same here, had too many timeouts and OutOfMemory issues with java. A few times I was very angry seeing my algorithm winning a good number of rounds just to crash a few runs later

I had a 85ms timeout for the algorithm and also was printing the number of milliseconds it takes to read input and solve the problem and it was all quite random.
Sometimes input can take 60ms and sometimes 10ms and the timeout is arbitrary. I had code that executed more than 140ms and it was OK so it’s not quite clear whether the statement “your program must output smth in 100ms” also refers to the time reading the input or not.

First round can take up to 400ms of time just to read data though it seems like everyone is saying that first run has no timeout penalty or it’s quite large.

Anyway, in case of Java, stop the world garbage collector can really play against you in such contests and I hope guys at CG can tune it a little bit more. Didn’t think about the trick with initializing a huge amount of memory at the beginning - will try this next time.

[Medici]

Same problem for golang (another GC language)

I solved it by allocating big space at first and manual control it then.

[PrinceKurt]

I’m having the same problem with one of the puzzles that I am coding in Java. The error will appear about every 10th or so run. The biggest problem is it randomly appears on one of the test cases for every submit, making it so I can never get the 100%.

[Jahz]

Had the same, and indeed never seen before the contest.
On top, I had sometimes weird timeouts as reported by int33h, even taking a good 15 ms margin on the 100 available.
Or is System.currentTimeMillis() meant to not be reliable ?

Would be good to have an official statement from CG, it’s a bit frightening to see Neumann as one of the very best Java dev here stating he will stop Java and learn c++ for the next contest…

[ThierryC]

I’ve also seen pause (that i suspect due to the GC too) in my java code during the smash the code contest. Are you using the default -Xms param or using a custom one ? perhap’s increasing -Xms to 100 or 200 MB would prevent GC from kicking in too early. Could it be a user setting ?

I’ve also seen the other issue reported here : ERROR: ld.so: object ‘/usr/lib/coreutils/libstdbuf.so’ from LD_PRELOAD
cannot be preloaded (wrong ELF class: ELFCLASS64): ignored

Also could you include in the code snippet that read the inputs, the init of the timeout ? Because there were a lot of questions around where to put it

[Jahz]

Now that’s “funny”, I just got the error as well on a single player standard puzzle (Skynet strikes back), which had never failed before like this…

ERROR: ld.so: object ‘/usr/lib/coreutils/libstdbuf.so’ from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.lang.ref.Finalizer.(Finalizer.java:226)

Happened twice and then worked as usual without changing anything…
Please CG, any idea ?

[selslack]

@MaximeC @Nonofr Guys, please make some statement here. Java is unplayable right now.

[_CG_SaiksyApo]

I played in Java the whole last hour and can’t reproduce it.

So, we restarted some code machine just in case.

Feel free to update the post if you experience any problems (Puzzle name and your code in Java via pastebin.)