The structure 'I'd rather you' is followed by a subject and a past simple verb to express a present or future preference. Therefore, the negative past simple 'didn't speak' is the correct choice. Other options like 'not speak' or 'didn't speaks' are grammatically incorrect in this pattern.