The Volano Report: Which Java platform is fastest, most scalable? A JavaWorld exclusive!

Results of new VolanoMark 2.1 server benchmarks reveal how 10 virtual machines stack up on 6 OSs

1 2 Page 2
Page 2 of 2
  1. Failed with "java.lang.OutOfMemoryError". To work around the problem, see below.
  2. Failed with "java.net.SocketException: accept: Invalid argument".
  3. Failed on the first try at 1,800 and 2,100 connections with "java.net.SocketException: Socket read failed: 10055" and "java.net.SocketException: Connection shutdown". Restarted the server and it worked on the second try.
  4. Took too long to complete at 1,800 connections (thrashing).
  5. Failed with "java.net.SocketException: Invalid argument".
  6. Failed with client dead session timeouts.

PULLQUOTE: BLUE RIGHT Just enter one command

to quadruple the number

of threads each

Java virtual machine

can handle. :END_PULLQUOTE

The stack reserve

As IBM has demonstrated, both Sun and Microsoft have artificially crippled the connection scalability of their Java virtual machines on Windows NT by shipping them with the default stack reserve of one MB per thread. Microsoft long ago changed the stack reserve of its own Internet Explorer 4.0 Web browser and Internet Information Server 4.0 Web server, following the "changed standard for all Windows NT system executables." (See the Resources section below.) Why it didn't apply the same change to its Java VM, or why Sun failed to pick up the change for its Windows NT port, is unclear to me. In any case, it's now time for them to follow IBM's lead and make this simple, but important, change.

You can make the change yourself if you buy the Microsoft Visual C++ development environment and use the Microsoft Binary File Editor. Just enter "editbin /stack:262144 java.exe" or "editbin /stack:262144 jview.exe" on the command line to quadruple the number of threads each Java virtual machine can handle.

After modifying the size of the stack reserve on the Microsoft and Sun Java virtual machines on Windows, I obtained the results shown below:

Java platform3006009001200150018002100Limit
Sun JDK 1.2 Windows
2381181214861215928732622~ 3600
Microsoft SDK 3.1 Windows
2409192315771200544a524459~ 3800

Table 3. VolanoMark 2.1.2 network scalability test of the Sun and Microsoft Java virtual machines for Windows NT after reducing the stack reserve from 1 MB to 256 KB per thread. Throughput is reported in messages per second based on the number of concurrent connections. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware. The Limit is the maximum number of simultaneous VolanoMark connections possible (shown as an estimate when preceded by ~).

Notes:

  1. Failed on the first try at 1,500 connections with "java.net.SocketException: Connection reset by peer" on client side. Restarted the server and it worked on the second try.

Environments

All tests ran identical copies of VolanoMark 2.1.2 on identical hardware -- a 200-MHz Intel Pentium Pro processor with a 256-KB L2 cache and 256 MB of RAM.

IBM JDK 1.1.7 OS/2preview operating system
  • 20 November 1998 — IBM OS/2 Warp Developer Kit, Java Edition, Version 1.1.7
  • IBM OS/2 Warp Server for e-business Preview (Internal revision 14.020F_UNI, planned for release in 2nd Quarter 1999)
  • java full version "JDK 1.1.7A IBM build o117-19981120 (JIT enabled: javax)"
  • Installed from javaintk.exe (13,659,042 bytes) and javainuf.exe (17,947,952 bytes).
  • Uses native threads and javax just-in-time compiler.
  • Modified the CONFIG.SYS file to increase the thread limit from 1,024 to 4,095 (THREADS=4095) and the initial swap file size from 2 MB to 32 MB (SWAPPATH=C:\OS2\SYSTEM 2048 32768).
Tower TowerJ 2.2.7 Linux
  • 19 October 1998 — Tower Technology TowerJ Compiler 2.2.7.0
  • Red Hat Linux Intel 5.2 (Linux Kernel 2.0.36)
  • TowerJ Compiler "version 2.2.7.0 x86-linux"
  • Installed from TowerJ_2_2_7_0_x86_linux.class (8,633,718 bytes) and TowerJ_2_2_7_0_x86_linux_glibc.tar.gz (4,663,423 bytes).
  • Uses green threads and native pre-compilation.
  • Used command option -b-heap-min 8388608 to set initial heap size to 8 MB, and built executable with TowerJ project files Main.tj and Mark.tj.
IBM JDK 1.1.7 Windowspart of SockPerf package
  • 28 January 1999 — IBM Win32 Java Virtual Machine Version 1.1.7
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
  • java full version "JDK 1.1.7 IBM build n117p-19990128 (JIT enabled: ibmjitc)"
  • Installed from ibm117gm.exe (10,132,439 bytes).
  • Uses native threads and ibmjitc just-in-time compiler.
Microsoft SDK 3.1 Windows
  • 20 January 1999 — Microsoft SDK 3.1 Windows Build 3165
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
  • jview version 5.00.3165
  • Installed from MSJavx86.exe (6,601,968 bytes) and SDK-Java.exe (10,675,448 bytes).
  • Uses native threads and just-in-time compiler.
  • Heap and stack command line options are not available.
Sun JDK 1.2 Windows
  • 01 December 1998 — Sun JDK 1.2 Windows 95/98/NT Production Release (Final)
  • Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 4)
  • java full version "JDK-1.2-V"
  • Installed from jdk12-win32.exe (20,521,166 bytes).
  • Uses native threads and symcjit just-in-time compiler.
  • The heap options must be preceded by the capital letter "X", and the stack options are not available.
Novell JDK 1.1.5 NetWare
  • 25 January 1999 — Novell JVM for NetWare
  • Novell NetWare Version 5.00
  • java full version "1.1.5" (JVM dated 01/08/99)
  • Installed from jvm.exe (18,107,065 bytes).
  • Uses native threads and symcjit just-in-time compiler (Symantec Java! JustInTime Compiler Version 3.00.040(x) for JDK 1.1.x).
  • Set Maximum Packet Receive Buffers = 1000 (default is 500).
  • Modified the java.cfg file to enable the just-in-time compiler (JAVA_COMPILER=symcjit).
Sun JDK 1.2 Solarisearly access Java VM
  • 22 December 1998 — Sun JDK 1.2 Solaris Production Release (Early Access)
  • Sun Solaris 7 Desktop Intel Platform Edition
  • java full version "Solaris_JDK_1.2_01_dev06_fcsV"
  • Installed from Solaris_JDK_1.2_01_dev06.i386.tar.Z (26,411,384 bytes).
  • Uses native threads and sunwjit just-in-time compiler.
  • The heap and stack options must be preceded by the capital letter "X".
Blackdown JDK 1.1.7 Linux
  • 04 November 1998 — Blackdown JDK 1.1.7 Linux
  • Red Hat Linux Intel 5.2 (Linux Kernel 2.0.36)
  • java full version "Linux_JDK_1.1.7_v1a_green_threads"
  • Installed from jdk_1.1.7-v1a-glibc-x86.tar.gz (12,219,873 bytes) and tya12v3.tgz (124,041 bytes).
  • Uses green threads and tya (TYA 1.2v3) just-in-time compiler.
JDK 1.1.7 FreeBSD
  • 21 December 1998 — JDK 1.1.7 for FreeBSD
  • FreeBSD 2.2.8-RELEASE
  • java full version "jdk1.1.7-FreeBSD:1998/12/21"
  • Installed from jdk1.1.7.V98-12-21.tar.gz (12,920,606 bytes).
  • Uses green threads and no just-in-time compiler.
Transvirtual Kaffe 1.00 Linuxbeta Java VM
  • 08 December 1998 — Transvirtual Technologies Kaffe OpenVM 1.0 Beta 3
  • Red Hat Linux Intel 5.2 (Linux Kernel 2.0.36)
  • Kaffe Virtual Machine "Engine: Just-in-time Version: 1.00 Java Version: 1.1"
  • Installed from kaffe-1.0.b3-3.i386.rpm (1,192,063 bytes).
  • Uses green threads and kaffe.jit just-in-time compiler.
  • Heap options require a space before the size (-ms 8m -mx 64m).

Conclusion

If I had my way, there would be only one chart in this report, but I guess that would make for a very short article. I believe the best single test for determining the best Java server platform is to compare the performance at a single high connection count. With this test, we have a clear winner:

Java platformScore at 2,100 connections

Sun JDK 1.2 Solaris

Sun JDK 1.2 Windows

IBM JDK 1.1.7 Windows

Microsoft SDK 3.1 Windows

1183
622
498
459

459

Figure 3. VolanoMark 2.1.2 network scalability test at 2,100 concurrent connections, measuring throughput in messages per second. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware. Only Java VMs that passed the test are shown; the Sun and Microsoft Java VMs for Windows required a reduction in their stack reserve in order to pass the test.

After seeing Microsoft's Java VM lead the pack in Java server performance and scalability for so long, I had started to think the other Java vendors would never catch up. It seems I greatly underestimated IBM's and Sun's ability to meet the challenge!

John Neffenger is the founder and chief technology officer of Volano LLC. He is the author of VolanoChat, a Web-based chat solution written in 100% Pure Java on both the client and server side. Volano has sold more than 500 server licenses in 33 countries around the world, with its largest customer averaging more than 69,000 hours of active connections to more than 134,000 chat visitors per day. Prior to his role at Volano, John was a software developer at IBM Corp., working on Taligent's CommonPoint Application Development Toolkit for OS/2 Warp. Before his assignment at Taligent, John worked in Palo Alto, CA, and Rome, Italy, on IBM's Open Systems Interconnection protocol stack. John has a BA in Mathematics from Northwestern University.

Learn more about this topic

  • This article's predecessor, "Which Java VM scales best?" (JavaWorld, August 1998) http://www.javaworld.com/jw-08-1998/jw-08-volanomark.html
  • VolanoMark 2.1 download location http://www.volano.com/mark.html
  • Technical details about the VolanoMark 2.1 benchmark http://www.volano.com/guide21/mark.html
  • The Volano LLC Home Page http://www.volano.com/
  • Vote for Java Bug ID 4075058, "java.ioAdd support for non-blocking I/O" http://developer.java.sun.com/developer/bugParade/bugs/4075058.html
  • Microsoft Knowledge Base Article ID Q189888, "PRBStack Size Now 256 K in Internet Information Server" 4.0 http://support.microsoft.com/support/kb/articles/q189/8/88.asp
  • The SPEC Open System Group is considering VolanoMark 2.1 for use in its forthcoming server-side Java benchmark suite http://www.spec.org/osg/
  • IBM JDK 1.1.7 OS/2 http://service.boulder.ibm.com/asd-bin/doc/en_us/catalog.htm
  • Tower TowerJ 2.2.7 Linux http://www.twr.com/
  • TowerJ's pricing structure http://www.twr.com/pricing/licensepricing.html
  • IBM JDK 1.1.7 Windows http://www.alphaworks.ibm.com/formula/SockPerf/
  • Microsoft SDK 3.1 Windows http://www.microsoft.com/java/download.htm
  • Sun JDK 1.2 Windows http://java.sun.com/products/jdk/1.2/
  • Novell JDK 1.1.5 NetWare http://developer.novell.com/java/
  • Sun JDK 1.2 Solaris http://java.sun.com/products/jdk/1.2/
  • Blackdown JDK 1.1.7 Linux http://www.blackdown.org/java-linux.html
  • JDK 1.1.7 FreeBSD http://www.freebsd.org/java/
  • Transvirtual Kaffe 1.00 Linux http://www.transvirtual.com/
  • Kaffe holds the best promise of providing Java support for some non-Intel Linux systems, such as those from Cobalt Networks http://www.cobaltmicro.com/
1 2 Page 2
Page 2 of 2