Newsletter sign-up

View all newsletters

Sign up for our technology specific newsletters.

Enterprise Java

Email Address:

Results of first-ever JVM server benchmark revealed

New benchmarking tool used to rank Java virtual machine server performance

By John Neffenger, JavaWorld.com, 12/01/97

In the computer industry, there are three kinds of lies: lies, damn lies, and benchmarks.

--The Jargon File

In the world of Java, it seems that benchmarks are the only part of that trio in short supply this year.

If you've tried to keep current on the state of server-side Java, you have undoubtedly seen a lot of confusing information. You've read press releases from vendors claiming that their Java implementations are enormously faster than their competitors', but getting caught with some questionable optimization techniques. You've even read peculiar claims that developers who use Java are lazy.

Who's a developer to believe? As always, believe the machine.

That's what we had to do here at Volano when customers of our chat server software began asking us which Java virtual machine (JVM) to use on their server machines. We ended up with a set of test cases and benchmarks that gave us a good indication of the stability, capacity, and performance of a particular Java implementation for our Java server products. Along the way we discovered results that were often surprising, certainly interesting, and almost always encouraging.

One of the tools we used to perform the tests was VolanoMark, a Java server benchmark with a special emphasis on long-lasting network connections and threads. The VolanoMark benchmark creates 10 groups of 20 connections in the VolanoMark server for a total of 200 concurrent connections. Each client then takes turns broadcasting messages to its group. At the end of the test, the throughput is reported as the average number of messages transferred by the server per second. That average throughput is the VolanoMark score.

In this article, I'll share with you the results of our tests. Since the VolanoMark benchmark requires Java 1.1, I have also included the source code to a single-threaded Java client/server test case we used to compare the performance of Java 1.0 to Java 1.1 (see Resources).

Benchmarks of a different breed

The tests we performed are unique for several reasons. First, they're on the server side. Java applet benchmarks, such as the CaffeineMark by Pendragon Software, have done a lot to encourage Java vendors to increase the general speed of their virtual machines. However, we quickly found that it gave us little indication of whether a Java virtual machine was acceptable for running a Java server application. Some commercial Java virtual machines that posted excellent CaffeineMark scores were incapable of satisfying even the minimal requirements of our Java server products. (Several virtual machines that we tested could not establish or maintain more than a few dozen simultaneous network connections. Our benchmark calls for JVMs to support a minimum of 200 concurrent connections. In a chat server that would represent 200 people, or ten rooms of 20 people.) Yet performance on the server is far more critical than applet performance since the demands of hundreds or thousands of clients are concentrated into a single machine.

 

About Us | Advertise | Contact Us | Terms of Service/Privacy

Copyright, 2006-2008 Network World, Inc. All rights reserved.