Which Java VM scales best?

A JavaWorld Exclusive! Results of new VolanoMark 2.0 server benchmark show how 12 virtual machines stack up

I find it ironic that the one company with a not-so-secret objective to "kill cross-platform Java" (according to an internal Microsoft document; see Resources below) also is the Java vendor that kept our 100% Pure Java company (Volano) in business for the past year and a half. Without the impressive performance of Microsoft's Java virtual machine, we would not have acquired the many VolanoChat customers who run our chat server on high-traffic Web sites. My article in the December issue of JavaWorld summarized our experience and reported the performance scores of four Java virtual machines using our Java server benchmark, VolanoMark 1.0.

While writing that article in November, I suggested to a Microsoft product manager that Sun might eventually trade places with Microsoft as the Java performance leader. "It'll never happen," was his quick reply. Although it might still be hard to argue with his response, most of the Java VM vendors have significantly narrowed Microsoft's lead.

Mostly in response to the December article, I have received more than 450 e-mail messages from 19 Java vendors trying to solve the technical problems they encountered while running VolanoMark and various Java server applications that the benchmark represents. Their efforts are now starting to reduce the great disparity that exists between Java virtual machines commonly used on the Internet today.

What is VolanoMark?

VolanoMark is a 100% Pure Java server benchmark characterized by long-lasting network connections and high thread counts. In this context, long-lasting means that the connections last several minutes or longer, rather than just a few seconds. The VolanoMark benchmark creates client connections in groups of 20 and measures how long it takes for the clients to take turns broadcasting their messages to the group. At the end of the test, it reports a score as the average number of messages transferred by the server per second. Its results have accurately predicted the real-world Java performance and scalability limits of our VolanoChat product line for almost two years now.

VolanoMark 2.0, which adds scalability measurements, is an update to the tool and is the first benchmark to be submitted to the SPEC Open System Group for use in its server-side Java benchmark suite. There are other Java benchmark programs under consideration for the SPEC suite that have different characteristics, such as the high connection-turnover rates of a Java Web server or the transaction-processing requirements of a Java database.

When trying to determine whether a product meets your performance and scalability requirements, the best approach is to write your own benchmark and obtain your own results. The next best approach is to run a benchmark program yourself on your own system. VolanoMark 1.0, for example, is available as a free download from Volano's Web site. (See the Resources section below.) In any case, when you read benchmark reports you should make sure you understand the characteristics of the test and the configuration of the system. Without information about the hardware configuration, the operating-system settings, and the Java virtual machine heap and stack options, benchmark scores are meaningless. Fortunately, SPEC will provide a set of well-defined rules for running its server-side Java benchmark suite when it becomes available early next year.

The benchmark tests

The tests presented in this report look at the scalability of 12 Java virtual machines on eight operating systems, using a common Intel hardware platform when possible.

The connection scalability tests are divided into two groups of Java virtual machines (VMs):

  • Java VMs available now as final production releases
  • Java VMs scheduled for release later this year

The processor scalability tests compare Microsoft's virtual machine on Windows NT with Sun's virtual machine on Solaris Intel Edition and Solaris SPARC Edition. Note that these processor scalability tests ran on different hardware than all the other tests.

The final tests show a direct comparison of all the Java virtual machines, using the local loopback interface with the client and server application running on the same machine.

Except for the processor scalability results, all tests ran the VolanoMark server on a 200-megahertz (MHz) Intel Pentium Pro processor with a 256-kilobyte (KB) L2 cache and 256 megabytes (MB) of RAM. The client connected to the server machine via a 10-megabit-per-second (Mbps) Ethernet network. For the VolanoMark network test, the client side ran on a 200-MHz Intel Pentium Pro processor with a 256-KB L2 cache and 128 MB of RAM using Microsoft's SDK for Java version 3.0 Pre-Release 1 (jview version 5.00.2750). The Apple JVM tests ran on a Apple Power Mac G3.

We ran the VolanoMark server and client applications with an increased initial heap size of 8 MB, an increased maximum heap size of 64 MB, and a decreased native stack size of 32 KB (using java -ms8m -mx64m -ss32k), except as follows:

  • The Microsoft SDK has no heap and stack command options available.
  • The Apple MRJ failed to run with the increased heap sizes.
  • The Novell JDK failed to run with a stack size less than its 48-KB default.
  • We did not decrease the 4-gigabyte (GB) default maximum heap size of the TowerJ-generated executable.

With the exceptions stated above, the commands were entered as follows, starting with two connections and working up to 900 connections:

Server
java -ms8m -mx64m -ss32k COM.volano.Main
Client
jview COM.volano.Mark -rooms 1 -users 2 -count 10000
jview COM.volano.Mark -rooms 5
  ...
jview COM.volano.Mark -rooms 45

See the COM.volano.Mark command synopsis for a complete description of all options.

To give you an idea of the heap usage, consider that the VolanoMark server uses 6,136 KB of heap memory when it runs under JavaSoft JDK 1.1.6 with 200 connections (-rooms 10) and with the garbage collector disabled.

All connection scalability tests were performed without restarting the server between tests, except for the JavaSoft JDK 1.1.6 and Apple MRJ 2.0 JVMs, which required a restart in order to reach 900 connections. Restarting these JVMs may have caused a variation in their scores due to the effects of the garbage collector, but not enough to alter the general picture of the results.

In this article, the SunSoft JDK refers to what Sun calls the Production Release for Solaris. The JavaSoft JDK refers to Sun's JDK 1.1 Win32 Release. See the Resources section for links to the download locations of each JVM.

Current Java VMs: Free and available now

Connection scalability of Java virtual machines that are freely available now. Bigger numbers are better. All JVMs ran the same test on the same Intel hardware platform except for Apple MRJ 2.0, which ran on a Power Mac G3.

These seven Java virtual machines are the ones we recommend to our customers for running the VolanoChat product live on production Web sites. The chart shows the VolanoMark 2.0 scores for seven Java VMs as the number of connections are increased. The number of connections are shown across the bottom, with the throughput in messages per second shown on the left. The scores are shown at up to 900 concurrent connections. Failed tests in the table are indicated by "---" (no score shown).

Microsoft SDK 2.02 still stands alone as the only fast and scalable Java virtual machine. Our customers with the highest Web site traffic currently have no other viable choice for a JVM. I had to restart the JavaSoft JDK 1.1.6 virtual machine five times between tests in order to reach 900 concurrent connections, and none of our large customers have succeeded in running our VolanoChat product with JavaSoft's JVM on Windows NT. IBM's recent updates to its operating system, TCP/IP stack, and JVM have given it a huge 50 percent performance improvement over the VolanoMark 2.0 tests I ran without the updates (not shown here), but I could not get more than 400 concurrent connections with IBM's JVM.

We had been waiting for more than a year for a stable Solaris Java VM and Sun finally delivered.

The SunSoft JDK 1.1.5 VM for Solaris may be a bit laggard in its performance, but its reliability is rock solid. We had been waiting for more than a year for a stable Solaris Java virtual machine with native threads and a just-in-time compiler, and Sun finally delivered on April 6, 1998. For me this was the biggest Java news of this year, but it went completely unnoticed (perhaps because its robust performance was not immediately apparent to the trade press that covered its release). After working with Java virtual machines on the server since early 1996, I had begun to lower my expectations of quality -- but the SunSoft JDK 1.1.5 sets a new standard for stability. No other JVM comes close. We ran our VolanoChat demonstration server on this JVM for more than 32 days non-stop -- that's roughly 40 messages per second, 24 hours per day for 32 days, without any down time and without even going over its initial 8-MB heap size! I think it would have run forever, but I had to bring it down in order to upgrade the VolanoChat server. (In fact, we haven't rebooted the Solaris 2.6 operating system since we first set it up on our Internet Web and chat server machine on November 5, 1997.)

Number of connections
Java Virtual Machine2100200300400500600700800900
Microsoft SDK 2.02175125252090176715561365124211261030946
JavaSoft JDK 1.1.6142020821997179715781381123510991011951
IBM JDK 1.1.612352009184318531603---------------
SunSoft JDK 1.1.5902122111341051888725322178145102
Apple MRJ 2.01191090950781877767------------
Linux JDK 1.1.6836688429---------------------
FreeBSD JDK 1.1.5516443382---------------------

Apple's Java virtual machine performed better than I had expected, although I had to restart the server between each test. The Linux and FreeBSD virtual machines do quite well considering that neither of them have a just-in-time compiler or native threads. For both Linux and FreeBSD, I was unable to get 300 concurrent connections or more, even after I increased their per-process file descriptor limits to more than 1000. In addition, the Linux and FreeBSD VMs have problems with the new Java 1.1 socket timeout feature, although I have been unable to isolate the cause. With the socket timeout feature enabled in the VolanoMark test, the Linux JVM runs extremely slowly and the FreeBSD JVM crashes with a core dump. Disabling the feature allowed them both to run the tests successfully.

Microsoft SDK 2.02
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 3)
  • Microsoft Internet Explorer 4.01 SP1 (Version 4.72.3110.8)
  • Microsoft SDK for Java Version 2.02
  • Microsoft virtual machine for Java (jview version 4.79.2405)
  • Ran clspack -auto after installation (\WINNT\Java\Classes\classes.zip is 7,702,544 bytes)
  • Set foreground application performance boost = None.
  • No heap or stack command options are available.
JavaSoft JDK 1.1.6
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 3)
  • JavaSoft JDK Version "JDK1.1.6N"
  • JIT Update for JDK 1.1.6 Early Access 2 (Symantec JIT Version x3.00.050)
  • Set foreground application performance boost = None.
IBM JDK 1.1.6
  • IBM OS/2 Warp 4.00 FixPak 7 (Revision 9.031)
  • IBM TCP/IP Version 4.1 (SOCKETS.SYS: 5.3003, AFOS2.SYS: 5.3000, AFINET.SYS: 5.3002)
  • IBM JDK Version "JDK 1.1.6 IBM build o116-19980605 (JIT: javax)"
  • Set THREADS=4095 in CONFIG.SYS.
SunSoft JDK 1.1.5
  • Sun Solaris 2.6 Desktop Intel Platform Edition with Maintenance Update 1
  • Solaris patch number 105491-04, "Dynamic linker patch (Intel)"
  • SunSoft JDK Version "Solaris_JDK_1.1.5_02"
  • Increased file descriptor limit to 4096.
Apple MRJ 2.0
  • Apple Power Macintosh G3 with 266-MHz PowerPC and 96 MB of RAM
  • Apple Mac OS 8.1
  • Apple Mac OS Runtime for Java 2.0 (with default heap and stack settings)
Linux JDK 1.1.6
  • Red Hat Linux 4.2 (Linux Kernel 2.0.30)
  • Linux JDK Version "Linux_JDK_1.1.6_v2"
  • Increased file descriptor limit to 1024.
  • Had to disable the use of the Java 1.1 socket timeout feature by setting the VolanoMark property server.timeout=0.
FreeBSD JDK 1.1.5
  • FreeBSD Version 2.2.6-RELEASE
  • FreeBSD JDK Version "jdk1.1.5-Freebsd:02/25/98"
  • Increased file descriptor limit to 4096.
  • Had to disable the use of the Java 1.1 socket timeout feature by setting the VolanoMark property server.timeout=0.

Looking ahead: Java VMs arriving later this year

Connection scalability of Java virtual machines that are scheduled to be available later this year. Bigger numbers are better. All JVMs ran the same test on the same Intel hardware platform.

These Java virtual machines should be available at various times throughout the year. Check with the Java vendors themselves for specific availability dates. Keep in mind that none of these JVMs are yet in their final release, so performance characteristics are likely to change.

One quick look at the chart and you'll see that the results are starting to get boring. That's a good thing. It means that Java developers can start to expect the same kind of performance, scalability, and reliability in Java virtual machines regardless of the platform or Java vendor. It used to be the joke that Java was "write once, run only on Solaris and NT." That's no longer true, thanks to Tower Technology and Novell.

I was surprised

to see Tower and

Novell come out

of nowhere with

such great VolanoMark

performance and

scalability.

1 2 Page
Join the discussion
Be the first to comment on this article. Our Commenting Policies
See more