In Pega, custom RuleSet names must start with a letter and can contain only alphanumeric characters. They are limited to a maximum of 32 characters, making the first option correct. RuleSets cannot contain spaces or special characters, and cannot begin with reserved prefixes like 'Pega'.