The sentence uses 'hardly...when' to show one action immediately followed another. The first action (finishing lunch) occurred before the past moment of knocking, so past perfect 'had' is required. 'Was' is simple past and incorrect for this sequence. 'Did' and 'would' do not fit the grammatical structure.