'That is why' introduces an explanation or reason for a past event or situation. Therefore, it must be followed by a past tense construction. 'I was late' uses the past tense correctly, while 'I am late' is present tense, 'I would late' is grammatically incorrect, and 'I have late' misuses the present perfect tense.