Which Java VM scales best?

A JavaWorld Exclusive! Results of new VolanoMark 2.0 server benchmark show how 12 virtual machines stack up

1 2 Page 2
Page 2 of 2

It has taken anywhere from four months to as long as a year from the time Java vendors first run VolanoMark until the time they run it well, so I was surprised to see Tower and Novell come out of nowhere with such great VolanoMark performance and scalability. To be fair, they had the advantage of coming to market when issues such as server-side Java performance had already become important.

Tower is working to determine the cause of its failure at 400 concurrent connections on Linux, but it is likely a limitation in the operating system itself. TowerJ runs on several other platforms with no such limitation. Apart from this, all of the Java virtual machines ran without any errors at all. In fact, I found the JavaSoft JDK 1.2 Beta 3 virtual machine to be more stable than the currently released JavaSoft JDK 1.1.6.

Number of connections
Java Virtual Machine2100200300400500600700800900
Tower TowerJ 2.1.21648270322841909------------------------
Novell JDK 1.1.5145326002259193016301444129811481058967
Microsoft SDK 3.0 P1177125942213187316291447129111281059956
JavaSoft JDK 1.2 Beta 314712459205917781553137012471130992939
SunSoft JDK 1.2 Dev 3986153814731425139112171074964856764

The limitation in Windows NT 4.0 of 2048 threads per process causes the VolanoMark server to get a java.lang.OutOfMemoryError when trying to get 1000 concurrent connections on Windows NT. The Solaris JVM uses a different threading model, and I have run the test myself with no problems up to 2000 concurrent connections. Sun told me that they have run the test with as many as 2400 concurrent connections, and they are working to go beyond that limit as well. Novell's JVM seems to be limited only by the amount of real memory on the system, although I have not tested to see what the actual limit might be with 256 MB of RAM.

Tower TowerJ 2.1.2
  • Red Hat Linux 4.2 (Linux Kernel 2.0.30)
  • TowerJ Version 2.1.2.0 x86-linux
  • Used command option -b-heap-min 8388608 to set initial heap size to 8 MB.
Novell JDK 1.1.5
  • Novell NetWare Prototype 5.00 Beta 3.0 (Build 1158, 19 June 1998)
  • Novell Java Version 1.1.5a (18 June 1998)
  • Symantec Java! JustInTime Compiler Version 3.00.040(x) for JDK 1.1.x
  • Set Maximum Packet Receive Buffers = 1000 (default is 500).
  • Used default native stack size of 48 KB since the test failed with 32 KB.
  • Set the -as NetWare JDK command option. According to Novell, the -as platform-specific command option provides an important load-balancing function that will be enabled by default in the final version of the Novell Java VM for NetWare 5.
Microsoft SDK 3.0 P1
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 3)
  • Microsoft Internet Explorer 4.01 SP1 (Version 4.72.3110.8)
  • Microsoft SDK for Java Version 3.0 Pre-Release 1
  • Microsoft virtual machine for Java (jview version 5.00.2750)
  • Ran clspack -auto after installation (\WINNT\Java\Classes\classes.zip is 7,012,092 bytes)
  • Set foreground application performance boost = None.
  • No heap or stack command options are available.
JavaSoft JDK 1.2 Beta 3
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 3)
  • JavaSoft JDK Version "JDK-1.2beta3-N"
  • Symantec Java! JustInTime Compiler Version 3.00.023(x) for JDK 1.2
  • Set foreground application performance boost = None.
  • Used -Xms8m -Xmx64m -Xss32k for non-standard command options.
  • Set -Djava.compiler=symcjit to enable the Symantec just-in-time compiler.
SunSoft JDK 1.2 Dev 3
  • Sun Solaris 2.6 Desktop Intel Platform Edition with Maintenance Update 1
  • Solaris patch number 105491-04, "Dynamic linker patch (Intel)"
  • SunSoft JDK Version "Solaris_JDK_1.2_01_dev03"
  • Used -Xms8m -Xmx64m -Xss32k for non-standard command options.

The hardware solution

Processor scalability of Microsoft and SunSoft virtual machines.

If you tried to buy better performance last year by adding processors, you might have been surprised by the results! Java performance early last year was strictly a software problem, and additional processors just made the problem much worse.

As you can see, those problems are fixed. The SunSoft JVM for Solaris scales well on both the Intel and SPARC platforms. The Microsoft JVM scales less well, especially on the fourth processor. Again, the virtual machines tested here are not in their final release, so these processor scalability results might change.

The highest VolanoMark result I have seen was a score of 5,642 messages per second on a Sun Ultra-Enterprise 4000/5000 with four 336-MHz UltraSPARC processors and 4 GB of RAM. I was afraid to find out what that machine might cost, though.

Microsoft SDK 3.0 P1 Intel
  • Dell PowerEdge 6100/200
  • 4 x 200 MHz Intel Pentium Pro with 512 KB of L2 cache and 128 MB of RAM
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 3)
  • Microsoft SDK for Java Version 3.0 Pre-Release 1
  • Microsoft virtual machine for Java (jview version 5.00.2750)
  • Set foreground application performance boost = None.
SunSoft JDK 1.2 Dev 3 Intel
  • Dell PowerEdge 6100/200
  • 4 x 200-MHz Intel Pentium Pro with 512 KB of L2 cache and 128 MB of RAM
  • Sun Solaris 2.6 Server Intel Platform Edition
  • SunSoft JDK Version "Solaris_JDK_1.2_01_dev03"
SunSoft JDK 1.2 Dev 3 SPARC
  • Sun Ultra Enterprise 450
  • 4 x 248-MHz UltraSPARC-II with 512 MB of RAM
  • Sun Solaris 2.6 Server SPARC Platform Edition
  • SunSoft JDK Version "Solaris_JDK_1.2_01_dev03"
Number of Processors
Java Virtual Machine12 (Scale)3 (Scale)4 (Scale)
Microsoft SDK 3.0 P1 Intel13841827 (32%)2142 (17%)2165 ( 1%)
SunSoft JDK 1.2 Dev 3 Intel9021648 (83%)2208 (34%)2645 (20%)
SunSoft JDK 1.2 Dev 3 SPARC14042390 (70%)3195 (34%)3889 (22%)

The Race

Direct performance comparison (except that the Apple MRJ ran on different hardware and the TowerJ compiler is not free).

I wanted to emphasize the scalability and not the raw performance of the Java virtual machines in this follow-up article, but I couldn't help but line them all up and compare them side-by-side again. All Java virtual machines shown below ran the same VolanoMark local loopback test on the same 200-MHz Intel Pentium Pro machine with 256 MB of RAM -- except for the Apple MRJ 2.0, which ran on a Power Mac G3. (For complete details of each test environment, see the lists above.) Apart from the Apple JVM, this is a direct, apples-to-apples comparison, with each vendor's JVM and operating system sandwiched between identical hardware and Java application code.

Where possible, the actual commands used to run the VolanoMark server and its client test driver were:

Server
java -ms8m -mx64m -ss32k COM.volano.Main
Client
java -ms8m -mx64m -ss32k COM.volano.Mark -count 100

except for the Microsoft SDK, Apple MRJ, and Novell JDK, as noted above in the description of the benchmark tests.

To give you an idea of the heap usage, the server uses 26,432 KB of heap memory in this test (40 percent of the 64-MB limit) when the garbage collector is disabled using JavaSoft JDK 1.1.6.

Tower shows that, for pure speed on the server side, a Java-to-native compiler is the way to go. The TowerJ compiler generates a native executable from pure Java code that runs almost 25 percent faster than the fastest Java interpreter, even when the interpreter has a just-in-time compiler. However -- unlike all the other listed vendors' offerings -- Tower's technology is not free, with prices ranging from 9 to ,999.

The rest of the virtual machines cover a full order of magnitude in the range of performance scores.

Microsoft's bold "it'll never happen" claim still holds true, and Microsoft comes out on top again among all the Java virtual machine interpreters. In a surprise, Novell comes in right below Microsoft with a beta release of its NetWare 5 Java virtual machine. JavaSoft's JDK 1.2 should be an excellent alternative for those unwilling or unable to use Microsoft's JVM because of its lack of Java Native Method Interface (JNI) and Remote Method Invocation (RMI) capabilities. And IBM has delivered excellent performance in its updated JDK 1.1.6 for OS/2 Warp.

Java virtual machineScoresAverage (best 2 of 3)
Tower TowerJ 2.1.21715, 1761, 17491755
Microsoft SDK 3.0 P11398, 1408, 14141411
Novell JDK 1.1.51319, 1325, 13201323
JavaSoft JDK 1.21260, 1234, 12601260
IBM JDK 1.1.61217, 1214, 12071216
JavaSoft JDK 1.1.61119, 1117, 11111118
Microsoft SDK 2.021109, 1108, 11091109
SunSoft JDK 1.2 Dev 3839, 837, 838839
SunSoft JDK 1.1.5546, 548, 546547
Apple MRJ 2.0319, 319, 323321
Linux JDK 1.1.6230, 234, 233234
FreeBSD JDK 1.1.5175, 175, 174175
It took me just one afternoon to install and verify VolanoChat on an operating system I had never even seen before!

Conclusion

If you've been reading lately about the death of Java's "write once, run anywhere" promise, don't you believe it. It took me just one afternoon to install and verify our VolanoChat product on an operating system I had never even seen before! Thanks to Novell's good work, we get another 4 million potential customers for free. And our applet runs on a wide variety of platforms -- a feat that was inconceivable just three years ago, before Java was invented.

We have customers doing just fine running VolanoChat servers on all of the currently released Java virtual machines discussed here. But with the current batch of VMs, customers are forced to make a choice between rock-solid stability (Sun) and blinding speed (Microsoft). According to my tests, though (and the latest VM release schedules), we will have at least five Java virtual machines by December that deliver the speed, stability, and scalability that the Java platform deserves.

John Neffenger is the founder and chief technology officer of Volano LLC. He is the author of VolanoChat, a Web-based chat solution written in 100% Pure Java on both the client and server side. Volano has sold more than 500 server licenses in 33 countries around the world, with its largest customer averaging 42,427 hours of active connections to 76,680 chat visitors per day. Prior to his role at Volano, John was a software developer at IBM Corp., working on Taligent's CommonPoint Application Development Toolkit for OS/2 Warp. Before his assignment at Taligent, John worked in Palo Alto, CA, and Rome, Italy, on IBM's Open Systems Interconnection protocol stack. John has a BA in mathematics from Northwestern University and secretly wishes that FreeBSD had the best Java virtual machine.

Learn more about this topic

  • This article's predecessor, "Results of first-ever JVM server benchmark revealed" (JavaWorld,, December 1997) http://www.javaworld.com/volanomark.html
  • Technical details about the VolanoMark benchmark http://www.volano.net/guide/mark.html
  • The SPEC Open System Group is considering VolanoMark 2.0 for use in its forthcoming server-side Java benchmark suite http://www.spec.org/osg/
  • The Volano LLC Home Page http://www.volano.com/
  • Apple MRJ 2.0 http://devworld.apple.com/java/index.html
  • FreeBSD JDK 1.1.5 http://www.freebsd.org/java/
  • IBM JDK 1.1.6 http://service.boulder.ibm.com/asd-bin/doc/en_us/catalog.htm
  • JavaSoft JDK 1.1.6 http://java.sun.com/products/jdk/1.1/download-jdk-windows.html
  • JavaSoft JDK 1.2 http://java.sun.com/products/jdk/1.2/
  • Linux JDK 1.1.6 http://www.blackdown.org/~sbb/
  • Microsoft SDK 2.02 http://www.microsoft.com/java/sdk/20/default.htm
  • Microsoft SDK 3.0 P1 http://www.microsoft.com/java/sdk/30p1/default.htm
  • Novell JDK 1.1.5 http://www.novell.com/netware5/index.html
  • SunSoft JDK 1.1.5 http://www.sun.com/solaris/java/
  • SunSoft JDK 1.2 Dev 3 http://developer.java.sun.com/developer/earlyAccess/jdk1.2/index.html
  • Tower Technology TowerJ 2.1.2 http://www.twr.com/
  • The Memorandum of the United States in Support of Motion for Preliminary Injunction (5/18/98) shows why some Java developers have reservations about Microsoft's Java virtual machine http://www.usdoj.gov/atr/cases/f1700/1762.htm
1 2 Page 2
Page 2 of 2