Theory mode forces you to play the fastest line to mate, if you play a move that is not one of the moves that leads most quickly to mate (and there may be more than one), then you will be asked to try again. This will continue until you find one of the fastest mating moves, or you play a move which loses or draws, at which point the problem will end and you will be marked incorrect. This process will continue until you have either failed, or reached the end of the problem. The main purpose of theory mode is to be shown the optimal mating line, and as such is useful for endgames where you may not know how to mate at all.