Mars Lander - Puzzle discussion

Puzzles
338

ezgif.com-optimize(3)

After 1 month trial, I finally pass Level3 with reinforcement learning approach!
my steps:

  • write a simulator, modified base on “gymnasium” LunarLander env
    mainly learn Box2d to simulate physical movement and detect collision
    design action space: since the space is large, I reduce the action to 3x2 (+15/0/-15 for rotate with clip of ±45degree, +1/-1 for power with clip of 0-4)

  • train a policy with help of “stable baseline3” PPO model
    tried DQN first, then switch to PPO
    need to carefully design the input and the reward function
    input: in final version, I give a 13 dimension vector as input [distance in 6 directions, h/v speed, rotate, power, relative h/v distance to the middle of landing zone, whether the lander can see the LandingZone]. normalize with modified sigmoid function to get all input in range [-1,1].
    reward: lots of pains in reward function design, +++ for success landing, + for approaching (i use the angle leftOfLandingZone-rocket-rightOfLandingZone as a indicator), — for collision, - for too high speed, - for high power, – for wrong direction
    training: since the space is very large, I train gradually (at beginning, the rocket start very near to the LandingZone, to get a good Q-value/Policy for that area, then gradually move the start far away and eventually the actual start point). I use random map between the 2 tests and random starting point (with randomness on position(with in the chosen area)/speed/rotation…) in “make_vec_env” to train.
    network arch: first I used a 4x200 nodes policy network and a same form of vf network, but then find out the parameters are too big for the 100k code limit.
    in final version is 2x100 nodes policy network and 4x200 vf network.

  • transfer the parameters (tensor or numpy array to base64 string) and use in codingame.
    thanks to the help from @ jacek1 and @ FrancoisB about this part :))) (the Unicode method is genius, but since I reduce my network, I won’t need that magic this time)
    manually calculate the “prediction” part to get output from neural network

And that’s it, enjoy the game!

3 Likes