I've spent a few months on this problem (on and off again), and I haven't been able to come up with a solution that's much better than brute force. So I want to walk through my logic a bit (if that's allowed, I'm not sure). The encryption is interesting. Obviously it helps to simplify the encryption code a bit.
And once you do, you find certain patterns. You're comparing two 32 bit numbers, and at some point you're comparing each bit in one number with each bit in another number. To find the value in the encrypted bit, you take the sum of the comparisons (for the last bit in your encrypted solution, you'd see if the last bit of both numbers is 1, and if they are, than the encrypted solution is 1, for example). In the second to last bit, you compare the last bit of one to the second to last bit of the other to either get 1 or 0. In the third to last bit, you compare the third to last bit with 0, then the second to last bit with the other second to last bit, and the sum is either a 1 or a 0. And so on and so forth.
And then in the middle of our four byte numbers (the 16th bit or so), you have these massive combinations of bit comparisons. But after awhile, we stop getting useful information from our encrypted numbers. We just say that bit = sum of a bunch of comparisons, the outcomes of which are unknown. So my solution was to... make systematic guesses as to their outcomes? Until I came up with something that encodes to the original? This doesn't have performance much better than brute force.
I don't even know what to google. Is this even the right track? Others in the forum have said they used google for an algorithm, that once they understood the "mathematical name" of the problem, it was easy to find. But I'm not even sure what to google. "Broken encryption algorithm?" "Fancy math?"