Lexical analysis, Part 2: Build an application

How to use the StreamTokenizer object to implement an interactive calculator

By Chuck Mcmanis, JavaWorld.com, 02/01/97

Page 2 of 6

The entire example application consists of two parsers -- one for commands and statements, and one for expressions.

Building a command parser

The command parser is implemented in the application class for the example STExample.java. (See the Resources section for a pointer to the code.) The main method for that class is defined below. I'll walk through the pieces for you.

 
 1    public static void main(String args[]) throws IOException {
 2        Hashtable variables = new Hashtable();
 3        StreamTokenizer st = new StreamTokenizer(System.in);
 4        st.eolIsSignificant(true);
 5        st.lowerCaseMode(true);
 6        st.ordinaryChar('/');
 7        st.ordinaryChar('-');

In the code above the first thing I do is allocate a java.util.Hashtable class to hold the variables. After that I allocate a StreamTokenizer and adjust it slightly from its defaults. The rationale for the changes are as follows:

• eolIsSignificant is set to true so that the tokenizer will return an indication of end of line. I use the end of the line as the point where the expression ends.
  
  
• lowerCaseMode is set to true so that the variable names will always be returned in lowercase. This way, variable names are not case-sensitive.
  
  
• The slash character (/) is set to be an ordinary character so that it will not be used to indicate the start of a comment, and can be used instead as the division operator.
  
  
• The minus character (-) is set to be an ordinary character so that the string "3-3" will segment into three tokens -- "3", "-", and "3" -- rather than just "3" and "-3." (Remember, number parsing is set to "on" by default.)
  
  

Once the tokenizer is set up, the command parser runs in an infinite loop (until it recognizes the "quit" command at which point it exits). This is shown below.

 
 8        while (true) {
 9            Expression res;
10            int c = StreamTokenizer.TT_EOL;
11            String varName = null;
12
13            System.out.println("Enter an expression...");
14            try {
15                while (true) {
16                    c = st.nextToken();
17                    if (c == StreamTokenizer.TT_EOF) {
18                        System.exit(1);
19                    } else if (c == StreamTokenizer.TT_EOL) {
20                        continue;
21                    } else if (c == StreamTokenizer.TT_WORD) {
22                        if (st.sval.compareTo("dump") == 0) {
23                            dumpVariables(variables);
24                            continue;
25                        } else if (st.sval.compareTo("clear") == 0) {
26                            variables = new Hashtable();
27                            continue;
28                        } else if (st.sval.compareTo("quit") == 0) {
29                            System.exit(0);
30                        } else if (st.sval.compareTo("exit") == 0) {
31                            System.exit(0);
32                        } else if (st.sval.compareTo("help") == 0) {
33                            help();
34                            continue;
35                        }
36                        varName = st.sval;
37                        c = st.nextToken();
38                    }
39                    break;
40                }
41                if (c != '=') {
42                    throw new SyntaxError("missing initial '=' sign.");
43                }

As you can see in line 16, the first token is called by invoking nextToken on the StreamTokenizer object. This returns a value indicating the kind of token that was scanned. The return value either will be one of the defined constants in the StreamTokenizer class or it will be a character value. The "meta" tokens (those that are not simply character values) are defined as follows: