Custom JSP tags are defined in tag libraries, which are XML files with a .tld (Tag Library Descriptor) extension. These tag library descriptors contain the tag definitions, including tag names, attributes, and handler classes. The tag library is then referenced in the JSP page or web.xml configuration.