Inserting preprocessor module between the loader and the lexical analyzer

860 14. BUILDING THE XTREMESCRIPT COMPILER FRAMEWORK

Figure 14.3

Storing source code in

Although virtually every plain text file in the world is stored in the ASCII format, the specific method for denoting line breaks often changes significantly from one platform to the next. To make things easier to manage internally, and to aid in portability, the loader will be responsible for ensuring that the in-memory version of the source code uses a consistent representation for line breaks and newlines.

Team-Fly®

Figure 14.4

The preprocessor

The Lexical Analyzer Module

As you know well by now, the lexer is responsible for converting the raw source code into a more usable format for the parser. This particular module doesn’t do anything on its own, however; although its convention to treat the lexer as its own conceptual step that takes place independ-ently, before the parser, it actually operates in parallel with it. The parser is responsible for invok-ing the lexer on a regular basis to return the next token in the stream, so the lexer doesn’t actual-ly execute until the parser explicitly calls it.