Results of first-ever JVM server benchmark revealed

New benchmarking tool used to rank Java virtual machine server performance

In the computer industry, there are three kinds of lies: lies, damn lies, and benchmarks.

--The Jargon File

In the world of Java, it seems that benchmarks are the only part of that trio in short supply this year.

If you've tried to keep current on the state of server-side Java, you have undoubtedly seen a lot of confusing information. You've read press releases from vendors claiming that their Java implementations are enormously faster than their competitors', but getting caught with some questionable optimization techniques. You've even read peculiar claims that developers who use Java are lazy.

Who's a developer to believe? As always, believe the machine.

That's what we had to do here at Volano when customers of our chat server software began asking us which Java virtual machine (JVM) to use on their server machines. We ended up with a set of test cases and benchmarks that gave us a good indication of the stability, capacity, and performance of a particular Java implementation for our Java server products. Along the way we discovered results that were often surprising, certainly interesting, and almost always encouraging.

One of the tools we used to perform the tests was VolanoMark, a Java server benchmark with a special emphasis on long-lasting network connections and threads. The VolanoMark benchmark creates 10 groups of 20 connections in the VolanoMark server for a total of 200 concurrent connections. Each client then takes turns broadcasting messages to its group. At the end of the test, the throughput is reported as the average number of messages transferred by the server per second. That average throughput is the VolanoMark score.

In this article, I'll share with you the results of our tests. Since the VolanoMark benchmark requires Java 1.1, I have also included the source code to a single-threaded Java client/server test case we used to compare the performance of Java 1.0 to Java 1.1 (see Resources).

Benchmarks of a different breed

The tests we performed are unique for several reasons. First, they're on the server side. Java applet benchmarks, such as the CaffeineMark by Pendragon Software, have done a lot to encourage Java vendors to increase the general speed of their virtual machines. However, we quickly found that it gave us little indication of whether a Java virtual machine was acceptable for running a Java server application. Some commercial Java virtual machines that posted excellent CaffeineMark scores were incapable of satisfying even the minimal requirements of our Java server products. (Several virtual machines that we tested could not establish or maintain more than a few dozen simultaneous network connections. Our benchmark calls for JVMs to support a minimum of 200 concurrent connections. In a chat server that would represent 200 people, or ten rooms of 20 people.) Yet performance on the server is far more critical than applet performance since the demands of hundreds or thousands of clients are concentrated into a single machine.

In addition, all tests were conducted on the same hardware platform -- a 200-MHz Intel Pentium Pro with 128 MB of memory. All Java code was compiled using JavaSoft's JDK 1.1.4 compiler. The result is a direct comparison of the vendors' software implementations that lets you judge your Java bang for the buck.

Finally, VolanoMark complements other benchmarks commonly cited for Java server performance, such as Web server benchmarks run against a Java Web server. Specifically, Web servers are characterized by a high number of very short-lived connections. Such environments are important for high-traffic Web sites and transaction systems. VolanoMark is characterized by a high number of long-lasting connections that demand sub-second response time. Together, both benchmarks can provide a broad perspective on two important characteristics of Internet servers written in Java.

We have provided a more flexible version of VolanoMark -- one that allows complete manipulation of all test parameters, such as the size of the broadcast groups, the total number of connections, and the pacing of the messages -- to the major Java vendors, including Apple Computer, BSDI, Hewlett-Packard, IBM, Microsoft, Novell, Santa Cruz Operation, Silicon Graphics, Softway Pty. Ltd., Sun Microsystems, and SuperCede. Several of the vendors have used VolanoMark over the past year to find and fix important bugs in their Java implementations.

Although I have lots to say about the stability and scalability of these Java virtual machines, this article is about performance on a single processor. I'll save these other two very important subjects for a future report.

The criteria

The test results are broken down into three comparisons:

  1. A direct performance comparison of the more popular Java virtual machines on an Intel hardware platform.
  2. A look at how fast a Java licensee's release is compared to the reference release from JavaSoft.
  3. A comparison of today's server-side Java performance with that of a year ago.

To compare these results with the Java virtual machine you use, you can download VolanoMark 1.0 from Volano's Web site (see Resources). That page is linked to an HTML form at JavaWorld where you can submit your results and system configuration for posting in the next issue of JavaWorld.

My goal is to inspire you to get answers to your nagging Java questions, not from the marketing departments of the Java vendors, but directly from the machine itself.

Sun vs. Microsoft

Which Java vendor? That's one of the most important questions for programmers developing a Java server application -- not in choosing a compiler or development environment, but in recommending the best Java environment for their customers. With dozens of different Java implementations available even for the same operating system, how are you to choose? We wrote VolanoMark for just that purpose.

The table below shows the VolanoMark 1.0 scores for the Java Virtual Machine implementations provided by each of the operating system vendors.

VolanoMark 1.0

Platform

Score

Windows NT

756

Solaris

334

OS/2 Warp

239

Linux

223

Bigger numbers are faster.

200 MHz Intel Pentium Pro with 128 MB of RAM

  • Windows NT: Microsoft Windows NT Workstation 4.00.1381 Service Pack 3 with Microsoft SDK for Java 2.0 Build 4.79.2252, includes a JIT compiler and native thread support
  • Solaris: Sun Solaris 2.6 with Sun's built-in Java 1.1.3 virtual machine, includes a JIT compiler and native thread support
  • OS/2 Warp: IBM OS/2 Warp 4 Build 20.40 with IBM JDK 1.1.4 Build o114-19971114, includes a JIT compiler and native thread support
  • Linux: Red Hat Linux 4.2 (Linux Kernel 2.0.30) with Sun's JDK 1.1.3 Version "root:08/30/97-09:42" ported by Randy Chapman and Steve Byrne, includes neither a JIT compiler nor native thread support

You've got to love competition! Microsoft's SDK 2.0 for Java is fast -- not just faster than the others, not just "fast enough," but just plain fast.

Sun's built-in Java support in Solaris 2.6 is a fast and stable port of JavaSoft's JDK 1.1.3. Sun has added a just-in-time compiler (JIT) and is one of the few Unix vendors with native thread support for Java.

OS/2 Warp was holding back in the sense that I was unable to get above roughly 25 percent processor usage during the test. Based on the results of some single-threaded network tests, I suspect the delays are in the threading support rather than in the TCP/IP stack's processing of VolanoMark's small 60 character messages. In addition, I used the TCP/IP stack provided in OS/2 Warp 4 and not the new TCP/IP Version 4.1 release.

The Java Linux port is hampered by its lack of a JIT and native thread support. I would expect its score to increase dramatically with these two additions.

Who's missing from this test? With more than 100 Java licensees, almost everyone is missing. You can find a list of other Java virtual machines for your hardware from the list on Sun's Web site (see Resources). Most of these JVMs were excluded because they run on different hardware than our Intel machines, and we wanted a direct software comparison. Others were excluded because of the limited success we have had in getting them to complete the test over the past year, such as Kaffe, the FreeBSD port, and SuperCede. Other platforms don't even offer a Java virtual machine -- most notably BSDI's BSD/OS.

Any Java virtual machine should be able to run server applications like VolanoMark. Java's "Write Once, Run Anywhere" capability arrived months ago for browsers, with our networked client applets running on 23 operating systems and 15 hardware platforms the last time we checked. I hope Sun can encourage the same kind of broad Java support on the server as Web browser vendors did for the client. In particular, the lack of good Java server support in BSD/OS, FreeBSD, OpenBSD, and NetBSD represents a large chunk of Internet servers to be excluded from Sun's "run anywhere" claim.

Java vs. Java

It's no wonder people think Java is slow. If you download and run the Java Development Kit (JDK) from JavaSoft, you can draw no other conclusion. But the JDK is simply a reference release, meant to be taken by the Java licensees and "souped up" for real use. In particular, the JDK does not have a JIT, and on Solaris it does not use native threads.

Microsoft took the JDK and added a JIT compiler to create SDK 2.0 for Java, increasing the performance by 500 percent. Sun took the JDK, added a JIT compiler and native threads, and got a 40 percent performance improvement.

The tables below compare the latest Java machines from Sun and Microsoft with the JDK 1.1.4 reference release from JavaSoft. Unlike the first table above, none of the following results are meant to declare a "winner." We're simply comparing the slow non-optimized reference release from JavaSoft with the latest optimized release from the respective operating system vendor to show what improvements have been made. Dozens of other Java licensees have optimized versions for these operating systems, so if you're interested in posting a result not shown here, we encourage you to download and run VolanoMark, and post your results to http://www.javaworld.com/javaworld/jw-12-1997/jw-12-submit.volano.html.

VolanoMark 1.0 on Windows NT

Vendor

Score

JavaSoft

149

Microsoft

756

Bigger numbers are faster.

Microsoft Windows NT Workstation 4.00.1381 Service Pack 3

  • JavaSoft: JavaSoft JDK 1.1.4 reference release
  • Microsoft: Microsoft SDK for Java 2.0 Build 4.79.2252 including a JIT compiler
VolanoMark 1.0 on Solaris

Vendor

Score

JavaSoft

240

Sun

334

Bigger numbers are faster.

Sun Solaris 2.6

  • JavaSoft: JavaSoft JDK 1.1.4 reference release
  • Sun: Solaris 2.6 built-in Java 1.1.3 virtual machine including a JIT compiler and native thread support

1997 vs. 1996

You can better appreciate where Java is today when you see how far it had to come. The picture for Java servers at the end of 1996 was pretty grim, with few JIT compilers and immature native thread support on Unix systems. Today, that picture is looking much better, as you can see from the tables below.

For the 1996 result, these tests ran the single-threaded Java client/server programs on Java 1.0.2. For the 1997 result, the tests ran on the operating system port of either Java 1.1.3 or 1.1.4. The source code for the test programs is provided in the Resources section of this article.

Java Client/Server on Windows NT

Year

Score

1996

260

1997

1070

Bigger numbers are faster.

Microsoft Windows NT Workstation 4.00.1381 Service Pack 3

  • 1996: JavaSoft JDK 1.0.2 reference release
  • 1997: Microsoft SDK for Java 2.0 Build 4.79.2252
Java Client/Server on Solaris

Year

Score

1996

JDK 1.0.2 not supported on Solaris 2.6

1997

695

Bigger numbers are faster.

Sun Solaris 2.6

  • 1996 = JavaSoft JDK 1.0.2 reference release
  • 1997 = Solaris 2.6 built-in Java 1.1.3 virtual machine
Java Client/Server on Linux

Year

Score

1996

245

1997

521

Bigger numbers are faster.

Red Hat Linux 4.2 (Linux Kernel 2.0.30)

  • 1996: Sun JDK 1.0.2 Version "chapman:10/12/12-23:12" ported by Randy Chapman
  • 1997: Sun JDK 1.1.3 Version "root:08/30/97-09:42" ported by Randy Chapman and Steve Byrne
Java Client/Server on OS/2 Warp

Year

Score

1996

597

1997

858

Bigger numbers are faster.

IBM OS/2 Warp 4 Build 20.40

  • 1996: IBM JDK 1.0.2 Build o102-19970207
  • 1997: IBM JDK 1.1.4 Build o114-19971114

Conclusion

As a result of the improvements made by vendors this year, Java performance is no longer an issue for Volano. We can confidently recommend to our customers any of the Java virtual machines evaluated in this article.

These test results summarize our search for the best Java environments for our highly networked, highly threaded multiuser Java applications. These applications and the VolanoMark benchmark are characterized by large numbers of long-lasting threads and socket connections. Other applications, such as Web servers, database servers, or transaction processors, could have radically different results, with a surprisingly different ranking of the Java vendors.

I encourage you to find out for yourself whether Java is ready for your server application.

John Neffenger founded Volano in June 1996 and is its chief technology officer. Prior to his role at Volano, he was a software developer at IBM Corp., working on Taligent's CommonPoint Application Development Toolkit for OS/2 Warp. Before his assignment at Taligent, John worked in Palo Alto, CA, and Rome, Italy, on IBM's implementation of the Open Systems Interconnection protocol stack. John has a BA in Mathematics from Northwestern University. When not in front of his computer, John can be found in Marin County on his Kestrel 200SC.
1 2 Page 1