I do agree that it's quite mathematical. I wouldn't say that it's for mathematician only but clearly you need to have a good background.
It took me a few minutes to rewrite the algorithm so that it becomes very simple to understand. I thought that it would be not so difficult to solve but I was wrong: I spent more than a week-end to figure out what the formula means mathematically (but it would have taken me few minutes 20 years ago). Either you have good mathematical knowledges and it's rather simple, or you don't have the necessary background and you may be stuck for a long time, no matter how clever you are... And google does not help a lot at this stage. This is a bit frustrating to be blocked not because you are not good but because you don't remind what this formula is.
Once you have understood which mathematical problem you have to solve then it's not so difficult but you are far from having finished. But google is your friend : a lot of litterature exists on the web but articles are written by mathematicians so not so easy to read and understand for non-math guys. There are not many algorithms but many derivatives and it's hard to translate the descriptions into efficient algorithms. I spent a few evenings trying to understand all the maths and trying to find ready-to-go implementations (there are some but not very friendly and not necessarily optimized for this particular case) and I wanted to do it myself thus I decided to try to write my own from scratch. I needed several attempts before I managed to implement the algorithms properly. I also discovered a very nice and powerful free cloud-based service (cloud Sage) which helped me a lot to check if my maths were correct.
I did everything in C, not real C++, and it took me ~1000 lines of code because I implemented everything on my own, trying to document things, not using the existing C++ classes, mostly for performance reasons and to track what was done at each step. By chance, I had very few bugs. If I had to redo it now that I understand it better, I would try directly in C++.
I did everything locally, outside of Codingame, using tests that I wrote myself (easy as we know the encoding formula) then putting the values from the Nintendo tests and comparing with the expected results. When I put everything under Codingame, I realized that i was expecting and writing the 32 bit integers in the wrong order. Easy to fix. I passed all tests and I submitted. Succeeded on the first submit. In total I probably spent nearly two weeks (a few hours every evening).
What I noticed is that when you take the encoded messages provided by Nintendo in the tests, you will obtain very few possibilities for the decoded messages (ie. the ones which correspond to the same encoded message). Same thing for the validators. This is a chance because some encoded messages correspond to many different decoded messages and you would spend a huge CPU time to evaluate and build the many possible combinations. I pass the 256 bit test in 12ms (on my PC using gcc with all possible optimizations on but keeping a single thread) but it may take much more time with some other messages, really a matter of chance that this one leads to only two possibilities (this is inline with the constraints on N2 given by Nintendo, meaning that they don't choose the messages to encode randomly but select ones which are simple to decode).