Docbook is a collection of standards and tools for technical publishing. A Docbook file is composed of SGML(3) tags and is also dependant of a Document Type Definition(2). Docbook has defined standard DTD that you can find in the docbook tools.
DTD : Document Type Definition. The DTD defines the vocabulary of content elements that an author can use and how they relate to each other. For example, a book element can contain a title element, any number of para elements for paragraphs, and any number of chapter elements.
SGML : Standard Generalized Markup Language, XML : Extensible Markup Language
XSL (Extensible Stylesheet Language) is a language of description used for the transformation of the sgml file into formatted ouput: XSL processors like xsltproc, saxon or xalan, ... do that.
Catalog : In XML, it provides a mapping from generic addresses to specific local directories on a given machine. A catalog can be used to locate the DocBook DTD, system entity files, and stylesheet files during processing.