To use the REWRITE statement in COBOL to update an existing record in a file, the file must be opened in I-O (Input-Output) mode. Opening in INPUT mode only allows reading, while OUTPUT mode only allows writing new records, making I-O the only valid mode.