Difference between revisions of "Scanner/Tokenizer"
From Free Pascal wiki
Jump to navigationJump to search| Line 8: | Line 8: | ||
variables of the compiler, and that all illegal characters found in the | variables of the compiler, and that all illegal characters found in the | ||
input stream cause an error. | input stream cause an error. | ||
| + | |||
| + | == Architecture == | ||
| + | |||
| + | The general architecture of the scanner is shown in the following figure: | ||
| + | |||
| + | Todo: Add Image | ||
| + | |||
| + | Several types can be read from the input stream, a string, handled by | ||
| + | <tt>readstring</tt>, a numeric value, handled by <tt>readnumeric</tt>, comments , compiler | ||
| + | and preprocessor directives. | ||
Next chapter: [[The parse tree]] | Next chapter: [[The parse tree]] | ||
Revision as of 05:16, 14 December 2003
back to contents FPC internals
Scanner/Tokenizer
The scanner and tokenizer is used to construct an input stream of tokens which will be fed to the parser. It is in this stage that the preprocessing is done, that all read compiler directives change the internal state variables of the compiler, and that all illegal characters found in the input stream cause an error.
Architecture
The general architecture of the scanner is shown in the following figure:
Todo: Add Image
Several types can be read from the input stream, a string, handled by readstring, a numeric value, handled by readnumeric, comments , compiler and preprocessor directives.
Next chapter: The parse tree
