Tricky question. I'm not up on all the deep technical stuff, but I know that when your turn ends, your time stops until the opponent does his stuff. If you don't stop timing yourself at that point, you'll capture the length of the opponent's turn too. That sounds like what's happening here; 80ms is how long it took the opponent to do his turn, minus the time it took you to read your inputs.
In the past I've had some success in not starting my timer until after I read the first turn input. I know it throws off the precise timing by a little bit, but at least that way I know it's my own time.