XML_Beautifier_Tokenizer | --phpDocumentor_XML_Beautifier_Tokenizer
Located in File: /phpDocumentor/Converters/XML/DocBook/peardoc2/Tokenizer.php
tokens are word separators. In English, a space or punctuation are examples of tokens. In PHP, a token can be a ;, a parenthesis, or even the word "function"
The parse() method is a do...while() loop that retrieves tokens one by one from the $_event_stack, and uses the token event array set up by the class constructor to call event handlers.
The event handlers each process the tokens passed to them, and use the _addoutput() method to append the processed tokens to the $_line variable. The word parser calls newLineNum() every time a line is reached.
In addition, the event handlers use special linking functions _link() and its cousins (_classlink(), etc.) to create in-code hyperlinks to the documentation for source code elements that are in the source code.