Monitor your Web server in realtime, Part 2

Chart a real Web server's performance in realtime using Percollator

Give the current Percollator applet a try. The Panel defaults to the current file. Load it by depressing the "Load URL" button. A URL is generated and the corresponding file on the server is read. The data is stored in a class and the graphing panel is displayed. Every ten seconds the graph panel changes to another performance graph. Once a cycle is complete, you can select the Statistics panel and see the current average, minimum and maximum for the parameters that were just graphed. At the bottom of the Statistics panel is a button whereby you can mail the Statistics data as an applet.

This is the second installment on building a Web Server performance-monitoring tool in Java. In this article we'll cover the following topics:

  • How to send a mail message directly from an applet to a MIME-capable mail reader that can display executable content.
  • How to store data samples in a hierarchical calendar.
  • Setting alarms and delivering alerts.
  • Working with datafiles on a remote server that are being updated.
  • Graphing the states of the rules using colors and a modified Gantt chart.
  • General user Interface improvements and improved object-oriented design.

The current HTTP operations per second

You need a Java-enabled browser to view this applet.

The current percent of user time in terms of cpu percentage

You need a Java-enabled browser to view this applet.

Sending mail messages directly from an applet

The latest version of the Netscape mailer supports MIME mail messages, which support a content type allowing messages to be decoded and associated with some type of processing program. This is a very powerful feature that will facilitate the delivery of executable content directly in a mail message. The Percollator applet periodically receives updates from the Percollator.se tool, which generates log entries on the server being monitored. You can see how this works in this example applet and source code. A user-interface panel is provided to the user so that upper and lower thresholds can be set. A number of options were considered for implementing this functionality:

  • Write a cgi script and send the data from the applet as a post. This approach has the usual problems of additional dependencies on the server, redundant transmissions, and the use of yet another scripting language.
  • Write a server program that listens for the data, This approach requires more code, the definition of a protocol, and installing the server program on all the servers.
  • Use the browser APIs for sending mail messages. This approach requires different versions for each browser.
  • Use SMTP directly. SMTP was designed to support a pretty simple interface on a known port.
  • Use IMAP directly. IMAP is a little more involved than SMTP.

I decided to use the direct interface to SMTP. In order to send a mail message to a designated recipient you can follow the following steps:

  • Contact the local SMTP server on port 25. To verify that this will work on your system try the following telnet localhost 25. You should see something like this coming back. Type "quit" to exit.

  • Send the following strings:

        MAIL FROM: senders email address
        RCPT TO: recipient
        DATA 
                To: "Some String"
                Mime-Version: 1.0
                Content-type: text/html
                Subject: A Subject line
                <HTML>
                <HEAD>
                </HEAD>
                <BODY>
                <A HREF=\"$url\">form</A>.
                </BODY>
                </HTML>
            .
    

    The first three messages inform SMTP of the message sender, destination, and the actual message body as a MIME compliant message. Notice the "." to indicate the end of the message.

The approach taken above an be taken with many network services. The following applet should allow you to send yourself a copy of this program as a mail message. That is this program will appear in your mail inbox and if your mailer is capable of associating a MIME message that says it is HTML with a program (in the future -- applet) that can process it, you should be able to send another mail message from your mailer. Sorry about the recursive MC Escher quality of this but its use in the Percollator applet will actually make sense. The standard output contains a wealth of diagnostic information for you so that you may become familiar with the interaction with a SMTP mail server program.

How to store data samples in a hierarchical calendar?

Adrian Cockcroft has written a very powerful package to implement a hierarchical calendar. Adrian has put this software in the public domain and I have provided some documentation enhancements and will be using it in the Percollator applet to keep track of performance readings for specific days. The documentation for the should be pretty self-explanatory. The next version will alow the saving of historical data for a specific day with storage on a server.

Setting alarms and delivering alerts

I have begun modifying the alerts panel to support the setting of upper and lower thresholds. These settings will be used by a background thread that compares newly arrived log entries to the thresholds and will then send a small applet containing a summary of the current conditions with the threshold values indicated in red. Take a look at the alerts menu in the Percollator applet.

Working with datafiles on a remote server that are being updated.

The Percollator applet has been modified to periodically rescan the server for new data in the log file. Ideally this should be modified to be a socket-based connection. There may be some issues in implementation depending on the caching policy that is being used. The current design reads the log file and then closes it and re-reads it at a later time.

General user-interface improvements and improved object-oriented design

Many of the original methods were turned into separate classes. A DataSet class has been created. This class contains a copy of the original input dataset and can be saved as an object in the calendar class. This class will also be used on the server to parse the input and deliver it via a socket -- greatly improving performance and allowing the Percollator applet to be event driven. This functionality will be added in the last and final installment. Various panels were made into separate classes so that the applet program is smaller and easier to understand. Common functions such as a mail panel were developed and the Statistics panel was made reusable for the the mailing of a Statistical Summary applet.

Saving files on a server

It is very desirable to be able to save files on a server. The various browsers impose quite a few restrictions on this. There are many products due out that will support the saving of state. There are a number of choices you have for saving data on the server. During the coming months we will explore all of the following in greater detail:

  • CORBA approach
  • Iona
  • JOE
    • Database
    • CGI Scripts
    • Socket Programs
    • RMI

This month we will look into the socket approach. The following example writes 100 doubles to a server on a known port and retrieves the numbers from the file using a URL. I have not enabled the server side of this example due to possibility that someone would abuse the server. In deploying this application I had to severely limit the ability of the users to read and write on the disk. Here is the source code so that your implementation will not be restricted. A java main program WriteFileFromSocket resides on the server and saves files to a fixed directory. The applet class libraries simplifies the programming required. The collection of applets on this page would be several thousand lines of code without the use of class libraries for networking, graphing, and user-interface management.

Open issues write to us

The following items have given me quite a bit of trouble and I will try to get more definitive answers for the next installment.

  • When using showStatus in a compute-bound loop status messages are queued by the browser and displayed all at once. I inserted a yield call and also changed the applets priority to be the lowest possible in an attempt to get more accurate update to no avail.
  • When using the applet viewer the length of a file opened as a URL is not reported correctly.
  • In compute-bound loops, repaint methods preceded or succeeded by yield() method calls do not cause the repaint to occur in a timely fashion. The solution to this was to grab the graphics context for the component and call the paintall method.
  • The author tag does not seem to work with the javadoc utility.
  • It is not clear how one would display an HTML page when in the context of an applet you have some HTML in a string or stringbuffer that you would like to display.
  • The performance of the read loop for loading 50 to 60 kilobytes is not very fast. It has not been determined if the problem is with the StringTokenizer or with the String class.

Datafile generation

The data for this applet is generated by Percollator.se described in the companion article. The format and meanings of this data are repeated below and are very cursory. The datafile must have the format indicated below. Future releases may allow you specify the titles and types in the title as a user option. You can do this as an exercise using the Vector and has classes.

timestamp            unix time
locltime             local time of day hh:mm:ss
DNnsrkcmdi           rules for Disk,Net,nfs,swap,ram,kmem,cpu,mutex,dnlc,inode
wbgARB               rule status white,blue,green,Amber,Red,Black
usr                  usr CPU time for the interval
sys                  system CPU time for the interval
pdb                  peak disk busy - utilization of the busiest of the disks
mdb                  mean disk busy - utilization averaged over all the disks
Ipkt/s Opkt/s Coll%  ethernet stats per second and collision%
tcpIn/s tcpOut/s     system total TCP traffic bytes/s
DupAck/s  Retran%    more TCP stuff - ignore it
httpop/s             access log entries per sec over the reporting interval
httpb/s              Access log value of bytes per second
http/p5s             peak 5 second interval access rate
get/s post/s cgi/s   Requests per second    
to4KB/s....to256KB/s http transfer in that range
ov256KB/s            http transfer over that range

The applet brings up a card panel that allows the user to select various functions. This first installment partially implements the Active, Load and Display Graphs cards. These three cards allow the user to load a particular dataset, have it display in the active window and display summary graphs respectively. The recommended method for using this applet requires that datafiles be available on the server you loaded the applet from -- if you are using the Netscape browser. The Netscape browser security policy raises a security exception if you attempt to load a URL from a server that is different than the one that the applet was loaded from. The applet has a default mode of generating a name string of the format:

percol-YYYY-MM-DD -- so January 22, 1996 is percol-1996-01-22

In order to get around this I have installed some local files in the current directory and the webmaster may have been able to get performance data collection on the JavaWorld server running.

How do I install this at my local site

Three components are required to get Percollator running on your own system.

The data used by the Percollator applet is generated by the tool described by Adrian Cockcroft in Monitoring Web Server Performance. Download and install this tool on your Web server.

The Percollator applet architecture is based on reusable component class libraries. Good, tested reusable component libraries help developers build applets quickly and with less errors. For example, the Percollator applet uses a reusable component class for drawing the charts (class PlotMixin) and a reusable component class for reading data in a URL (class FileMixin). Without them, the Java source program would be five to ten times larger. These and other reusable component class libraries are available from Inductive Solutions: example applets using these and other interesting class libraries are viewable at Inductive Solutions.

The Percollator applet architecture is based on reusable component class libraries. Good, tested, reusable component libraries help developers build applets quickly and with fewer errors. For example, the Percollator applet uses a reusable component class for drawing the charts (class PlotMixin) and a reusable component class for reading data in a URL (class FileMixin). Without them, the Java source program would be five to ten times larger. These and other reusable component class libraries are available from Inductive Solutions.

Next month's article

  • An ORB based server will be implemented and used to store data for users. Those of you not wishing to procure a server such as the one from Iona can use the write to server programs described above.
  • General heisenbug removal as reported by my faithful readers
  • Management of alerts
  • Statistics on a bunch of datasets

One final item. A coworker of mine came across this interesting representation of Web access stats in 3D. If someone ports POV(described in reference) to java for the next article I will include it.