Wizard API updated!
Tim Boudreau has released a new version of the Swing Wizard library (version 0.997) that fixes the WizardException bug reported in JavaWorld's recent Open Source Java Project profile. The article's examples have been reworked to test out the new, improved WizardException. Thanks, Tim, for this helpful fix!
Open Source Java Projects: The Wizard API

Newsletter sign-up

Sign up for our technology specific newsletters.

Enterprise Java
View all newsletters
Email Address:
JavaWorld.com > Java Standard Edition >

Results of first-ever JVM server benchmark revealed

New benchmarking tool used to rank Java virtual machine server performance

By John Neffenger, JavaWorld.com, 12/01/97

In the computer industry, there are three kinds of lies: lies, damn lies, and benchmarks.


--The Jargon File



In the world of Java, it seems that benchmarks are the only part of that trio in short supply this year.

If you've tried to keep current on the state of server-side Java, you have undoubtedly seen a lot of confusing information. You've read press releases from vendors claiming that their Java implementations are enormously faster than their competitors', but getting caught with some questionable optimization techniques. You've even read peculiar claims that developers who use Java are lazy.

Who's a developer to believe? As always, believe the machine.

That's what we had to do here at Volano when customers of our chat server software began asking us which Java virtual machine (JVM) to use on their server machines. We ended up with a set of test cases and benchmarks that gave us a good indication of the stability, capacity, and performance of a particular Java implementation for our Java server products. Along the way we discovered results that were often surprising, certainly interesting, and almost always encouraging.

One of the tools we used to perform the tests was VolanoMark, a Java server benchmark with a special emphasis on long-lasting network connections and threads. The VolanoMark benchmark creates 10 groups of 20 connections in the VolanoMark server for a total of 200 concurrent connections. Each client then takes turns broadcasting messages to its group. At the end of the test, the throughput is reported as the average number of messages transferred by the server per second. That average throughput is the VolanoMark score.

In this article, I'll share with you the results of our tests. Since the VolanoMark benchmark requires Java 1.1, I have also included the source code to a single-threaded Java client/server test case we used to compare the performance of Java 1.0 to Java 1.1 (see Resources).

Benchmarks of a different breed

The tests we performed are unique for several reasons. First, they're on the server side. Java applet benchmarks, such as the CaffeineMark by Pendragon Software, have done a lot to encourage Java vendors to increase the general speed of their virtual machines. However, we quickly found that it gave us little indication of whether a Java virtual machine was acceptable for running a Java server application. Some commercial Java virtual machines that posted excellent CaffeineMark scores were incapable of satisfying even the minimal requirements of our Java server products. (Several virtual machines that we tested could not establish or maintain more than a few dozen simultaneous network connections. Our benchmark calls for JVMs to support a minimum of 200 concurrent connections. In a chat server that would represent 200 people, or ten rooms of 20 people.) Yet performance on the server is far more critical than applet performance since the demands of hundreds or thousands of clients are concentrated into a single machine.

In addition, all tests were conducted on the same hardware platform -- a 200-MHz Intel Pentium Pro with 128 MB of memory. All Java code was compiled using JavaSoft's JDK 1.1.4 compiler. The result is a direct comparison of the vendors' software implementations that lets you judge your Java bang for the buck.

1 | 2 |  Next >
Resources
  • The 1.0 release of VolanoMark, a benchmark for evaluating a Java Virtual Machine for use in a server environment -- especially one characterized by long lasting connections. http://www.volano.com/mark.html
  • The source code to a simple Java TCP/IP client which bounces messages back and forth between a Java echo server. http://www.volano.com/Client.java
  • The source code to a simple Java TCP/IP echo server. http://www.volano.com/Server.java
  • Java Development Kit, the latest 1.1 version of the Java reference release from JavaSoft. http://java.sun.com/products/jdk/1.1/
  • The latest Java Virtual Machine implementation specially optimized for the Solaris operating system. http://www.sun.com/solaris/java/
  • Java-Linux Porting Project -- the port of Sun's Java Development Kit from The Blackdown Organization. http://www.blackdown.org/java-linux.html
  • IBM Centre for Java Technology. IBM's port of Sun's Java Development Kit to IBM's various operating systems. http://ncc.hursley.ibm.com/javainfo/download/index.html
  • Microsoft's port of Sun's Java Development Kit to Windows 95 and Windows NT 4.0. http://www.microsoft.com/java/
  • CaffeineMark Benchmark test suite from Pendragon Software. The CaffeineMark applet used to judge general Java performance in a Web browser environment. http://www.webfayre.com/
  • Java Ports -- A list of Java virtual machines for many different hardware platforms. http://java.sun.com/cgi-bin/java-ports.cgi
  • An insightful article from JavaWorld's sister publication, SunWorld discusses the benchmark results. http://www.sun.com/sunworldonline/swol-12-1997/swol-12-volano.html
  • How much of an impact do benchmark scores have in determining the Java implementation you choose? Speak your mind in our latest reader poll. http://nigeria.wpi.com/cgi-bin/gwpoll/gwpoll/ballot.html