Lexical analysis, Part 2: Build an application

How to use the StreamTokenizer object to implement an interactive calculator

By Chuck Mcmanis, JavaWorld.com, 02/01/97

To review last month's article briefly, there are two lexical-analyzer classes that are included with the standard Java distribution: StringTokenizer and StreamTokenizer. These analyzers convert their input into discrete tokens that a parser can use to understand a given input. The parser implements a grammar, which is defined as one or more goal states reached by seeing various sequences of tokens. When a parser's goal state is reached, it executes some action. When the parser detects that there are no possible goal states given the current sequence of tokens, it defines this as an error state. When a parser reaches an error state, it executes a recovery action, which gets the parser back to a point at which it can begin parsing again. Typically, this is implemented by consuming tokens until the parser is back to a valid starting point.

Last month I showed you some methods that used a StringTokenizer to parse some input parameters. This month I'll show you an application that uses a StreamTokenizer object to parse an input stream and implement an interactive calculator.

Building an application

Our example is an interactive calculator that is similar to the Unix bc(1) command. As you'll see, it pushes the StreamTokenizer class right to the edge of its utility as a lexical analyzer. Thus, it serves as a good demonstration of where the line between "simple" and "complex" analyzers can be drawn. This example is a Java application and therefore runs best from the command line.

As a quick summary of its abilities, the calculator accepts expressions in the form

[variable name] "=" expression 

The variable name is optional and can be any string of characters in the default word range. (You can use the exerciser applet from last month's article to refresh your memory on these characters.) If the variable name is omitted, the value of the expression simply is printed. If the variable name is present, the value of the expression is assigned to the variable. Once variables have been assigned to, they can be used in later expressions. Thus, they fill the role of "memories" on a modern hand-held calculator.

The expression is composed of operands in the form of numeric constants (double-precision, floating-point constants) or variable names, operators, and parentheses for grouping particular computations. The legal operators are addition (+), subtraction (-), multiplication (*), division (/), bitwise AND (&), bitwise OR (|), bitwise XOR (#), exponentiation (^), and unary negation with either minus (-) for the twos complement result or bang (!) for the ones complement result.

In addition to these statements, our calculator application also can take one of four commands: "dump," "clear," "help," and "quit." The dump command prints out all of the variables that are currently defined as well as their values. The clear command erases all of the currently-defined variables. The help command prints out a few lines of help text to get the user started. The quit command causes the application to exit.