Custom dialog boxes can be created in Xcelsius by grouping multiple components together and treating them as a single visual unit. This allows building complex, reusable UI elements like custom input forms, message boxes, or configuration panels by combining basic components.