Tower Technology's TowerJ 3.1.4 for Linux dominates 15 other VMs in both performance and scalability, according to the latest VolanoMark 2.1.2 benchmarks published on October 3 by Volano LLC.
Moreover, compared to previous VolanoMark results published on May 31 (see Resources for a link to this report), raw server-side VM performance on the benchmark improved for all vendors and platforms. This trend should continue as vendors compete to produce the top VM.
Even with the increased performance, however, the network scalability tests -- with two exceptions -- show disappointing results, with some VMs incapable of handling anything but the simplest concurrent connection levels.
The VolanoMark 2.1.2 pure Java benchmark measures both raw server performance and server network scalability performance. Both tests pit 16 VMs on seven operating systems against each other on the same 200 MHz Intel Pentium Pro hardware, thus achieving a good apples-to-apples comparison.
For both tests, the VolanoMark benchmark creates client connections in groups of 20 and measures how long it takes for the clients to take turns broadcasting their messages to the group. At the end of the tests, it reports a score as the average number of messages transferred by the server per second.
The raw performance test runs with the server and all of the simulated clients on the same machine, communicating over a local loopback connection, running at a static 200 concurrent socket connections.
In contrast, the network scalability test moves the simulated clients onto another, more powerful machine and communicates with the server over a 10 Mbps Ethernet connection. The client side runs four times for each VM, simulating 1,000, 2,000, 3,000, and 4,000 concurrent socket connections. Only those VMs that successfully scale up to 4,000 connections pass the test.
In both the raw performance and network scalability tests, the higher the score, the better the result. (See Resources for more on the hardware and test methodology.)
Performance: Speedier than a fast bullet
VM bragging rights rest on speed, speed, speed, and in this test round, vendors do not disappoint. In terms of raw speed, the 16 tested VMs showed good results across all vendors and platforms -- a big plus for the Java community.
TowerJ above the rest
For raw performance, Tower Technology's TowerJ 3.1.4 for Linux finished first, with an average score of 2,309, as shown in Figure 1 below. TowerJ 3.1.4, however, is the only static compiler in the test group. As such, it takes class files and converts them into C source code, which it then compiles into a native executable program (with version 3, TowerJ can also dynamically load Java class files in their original byte code form). It's also the only contestant in the group not available for free. "TowerJ is a native compiler that costs thousands of dollars," says Volano CTO John Neffenger. "It's expected that they'd be on top."
|Tower TowerJ 3.1.4 Linux||2,309|
|IBM JDK 1.1.8 Windows NT||2,279|
|IBM JDK 1.1.8 OS/2||2,136|
|Microsoft VM 3229 Windows 2000||1,970|
|IBM JDK 1.1.8 Linux||1,770|
|Sun HotSpot 1.0.1 Windows NT||1,601|
|Sun JDK 1.3 Beta Windows NT||1,550|
|Sun JDK 1.2.2 Windows NT||1,485|
|Microsoft VM 3186 Windows NT||1,478|
|Sun JDK 1.2.1_04 Solaris||1,358|
|Novell JDK 1.1.7 NetWare||1,229|
|Sun JDK 1.2.1_03 Solaris||1,051|
|Sun JDK 1.2 Linux||915|
|Transvirtual Kaffe 1.0b4 Linux||389|
|Blackdown JDK 1.1.7 Linux||285|
|JDK 1.1.8 FreeBSD||173|
Neffenger adds that, among the standard pure Java VMs, IBM's various entries closely follow TowerJ. Indeed, IBM VMs held an impressive three of the top five spots in the raw performance tests. Big Blue's JDK 1.1.8 for Windows finished second with a 2,279 score, just 30 points behind TowerJ -- a 1.30 percent difference. IBM also locked up third place, with the IBM 1.1.8 for OS/2 VM scoring 2,136.
The latest entrant to the VolanoMark race -- Microsoft's forthcoming VM 3229 for Windows 2000 -- finished fourth with a 1,970 score, just ahead of IBM's fifth-place IBM 1.1.8 for Linux VM, which scored 1,770.
As for Sun, its top performing VM -- the highly marketed HotSpot 1.0.1 for Windows -- placed sixth, with 1,601, followed by the Sun JDK 1.3 Beta for Windows NT (1,550) and Sun's JDK 1.2.2 for Windows NT (1,485).
Table 1 below further details the performance tests by listing each entrant's Java platform, operating system, four test results, and final average score.
|Java Platform||Operating System||Results||Score|
|Tower TowerJ 3.1.4 Linux||Red Hat Linux 6.0 Intel||2,303, 2,313, 2,313, 2,300||2,309|
|IBM JDK 1.1.8 Windows NT||Windows NT Workstation 4.0||2,282, 2,276, 2,280, 2,282||2,279|
|IBM JDK 1.1.8 OS/2||OS/2 Warp Server for ebusiness||2,145, 2,141, 2,118, 2,150||2,136|
|Microsoft VM 3229 Windows 2000||Windows 2000 Server Release Candidate 2||1,948, 1,965, 1,976, 1,968||1,970|
|IBM JDK 1.1.8 Linux||Red Hat Linux 6.0 Intel||1,242, 1,772, 1,787, 1,750||1,770|
|Sun HotSpot 1.0.1 Windows NT||Windows NT Workstation 4.0||1,558, 1,611, 1,604, 1,589||1,601|
|Sun JDK 1.3 Beta Windows NT||Windows NT Workstation 4.0||1,538, 1,550, 1,556, 1,545||1,550|
|Sun JDK 1.2.2 Windows NT||Windows NT Workstation 4.0||1,496, 1,481, 1,489, 1,485||1,485|
|Microsoft VM 3186 Windows NT||Windows NT Workstation 4.0||1,469, 1,484, 1,478, 1,473||1,478|
|Sun JDK 1.2.1_04 Solaris||Solaris 7 Desktop Intel Platform Edition||1,371, 1,344, 1,374, 1,355||1,358|
|Novell JDK 1.1.7 NetWare||NetWare 5||1,227, 1,232, 1,229, 1,226||1,229|
|Sun JDK 1.2.1_03 Solaris||Solaris 7 Desktop Intel Platform Edition||1,118, 1,055, 1,050, 1,049||1,051|
|Sun JDK 1.2 Linux||Red Hat Linux 6.0 Intel||915, 914, 912, 919||915|
|Transvirtual Kaffe 1.0b4 Linux||Red Hat Linux 6.0 Intel||397, 391, 390, 386||389|
|Blackdown JDK 1.1.7 Linux||Red Hat Linux 6.0 Intel||284, 285, 286, 283||285|
|JDK 1.1.8 FreeBSD||FreeBSD 3.2-RELEASE||169, 173, 173, 172||173|
Improvements all around
Aside from these individual VolanoMark performances, the tested VMs as a group continue to show significant performance improvements over previous versions, when compared to VolanoMark tests reported on May 31 (see Resources for a link to this report). This improvement applies to all vendors and platforms in the test group. For example, this round's leader -- TowerJ version 3.1.4 for Linux -- performed approximately 21 percent faster than version 3.0.1's May 31 score.
"We're three years into having good virtual machines, and [VM vendors] are still coming out with these great performance improvements," states Neffenger.
The good news about raw performance aside, VolanoMark also measures network scalability, where the various VMs show mixed results.
Stability and scalability: Still a ways to go
The network scalability results shown in Figure 2 below demonstrate that most vendors' VMs still need some work, at least in terms of long-lasting stability at high concurrent connections. Indeed, the focus for many Java developers has shifted from raw VM performance to stability and scalability.
"The performance issue for us [Volano LLC] was solved in August and September 1997 when Microsoft and Sun gave us really fast VMs. Since then, the problem has been handling thousands of connections to a single Java application and being able to keep it running for months, or even years," says Neffenger.
There can be only two
As in the raw performance tests, TowerJ 3.1.4 for Linux won the scalability tests, followed by Sun's JDK 1.2.1_03 for Solaris. Notably, among the 16 VMs tested, these two were the only VMs able to finish the test by successfully scaling up to 4,000 concurrent connections without producing any errors -- a milestone that tripped up other promising contestants.
On the downside, Table 2 below shows that the remaining 14 VMs were incapable of finishing the network scalability test, each producing some sort of error before reaching 4,000 concurrent connections. IBM's JDK 1.1.8 for Windows NT, for example, led all comers in network scalability performance up to 3,000 concurrent connections, but then failed before it reached 4,000.
|Tower TowerJ 3.1.4 Linux||2,486||1,357||739||421||No errors!|
|IBM JDK 1.1.8 Windows NT||2,623||1,826||1,081||----|
|IBM JDK 1.1.8 OS/2||1,521||----||----||----|
|Microsoft VM 3229 Windows 2000||1,408||708||406||----|
|IBM JDK 1.1.8 Linux||----||----||----||----||Process hangs in hard run at 500/1,000 connections and dumps core when killed|
|Sun HotSpot 1.0.1 Windows NT||----||----||----||----|
|Sun JDK 1.3 Beta Windows NT||1,474||856||523||----|
|Sun JDK 1.2.2 Windows NT||----||----||----||----|
|Microsoft VM 3186 Windows NT||1468||770||----||----|
|Sun JDK 1.2.1_04 Solaris||1,875||1,040||659||----|
|Novell JDK 1.1.7 NetWare||1,575||----||----||----||Process hangs with disk thrashing at 1,960/2,000 connections|
|Sun JDK 1.2.1_03 Solaris||1,794||1,058||613||354||No errors!|
|Sun JDK 1.2 Linux||----||----||----||----||Tests fail with |
|Transvirtual Kaffe 1.0b4 Linux||612||----||----||----||Dumps core at just over 1,000 connections|
|Blackdown JDK 1.1.7 Linux||165||----||----||----||Dumps core at just over 1,000 connections|
|JDK 1.1.8 FreeBSD||37||----||----||----||Client begins failing at 1,017 connections with |
In fact, four VMs -- IBM's JDK 1.1.8 for Linux, Sun's HotSpot 1.0.1 for Windows NT, Sun's SDK 1.2.2 for Windows NT, and Sun's JDK 1.2 for Linux -- were not even able to handle 1,000 concurrent connections, the first rung on the VolanoMark scalability ladder.
Moreover, Table 2 shows that, with the exception of TowerJ 3.1.4 for Linux, VMs for Linux seem particularly susceptible to poor scalability results, a problem Neffenger blames on an incompatibility between Java's I/O model and the Linux model.
"I'm disappointed in the network scalability on Linux. IBM on Linux couldn't even do the test. The solution is to do a different threading model for Java on Linux, similar to the TowerJ solution, or better, as Sun did on Solaris," he states.
Threads and sockets: The network scalability dilemma
The scalability problem revealed by the VolanoMark tests comes down to a conflict between how Java handles threads and sockets and how the underlying operating systems handle them, says Neffenger. Java, for instance, requires one or two dedicated threads in order to read and write data from a socket connection. On the operating systems side, the Windows platform limits the number of threads handled in a single process, while Unix-based systems limit the number of sockets you can give to a single process (because of the file descriptor limits). If the VM vendor isn't careful, this incongruity can result in runtime errors at increasing concurrent connection levels.
Therefore, according to Neffenger, success in the VolanoMark network scalability test depends upon how the VM vendors mapped Java's view of threads and sockets onto the underlying operating system's view.
For example, with the IBM JDK 1.1.8 for Linux VM, IBM mapped the Java threads one-to-one to the Linux threads. "For long-lasting, dedicated connections to a pure Java server, that particular model is a disaster," says Neffenger.
With TowerJ 3.1.4 for Linux, on the other hand, Tower Technology mapped all of the Java threads onto one single process in Linux. This many-to-one model, as the network scalability results attest, seems to work.
Neffenger adds that, with the Sun JDK 1.2.1_03 for Solaris VM, Sun split the difference by following a many-to-many model. "They've taken a selection of Java threads and mapped them onto one Solaris thread, and, vice versa, they've taken one Java thread and mapped it onto different Solaris threads. Sun claims that's the best model." (For more a detailed look at the threads and sockets dilemma, see Resources.)
Whatever the issues, two successful, error-free VMs out of 16 does not bode well for network scalability, and as Java's focus shifts to server-side applications, the Java community can only hope this will change.
Whatever technical solutions vendors need to bring scalability to their VMs, TowerJ 3.1.4's results demonstrate that you can combine both high raw performance and long-lasting stability at high concurrent connection rates. The other VM entrants ought to look at the whole picture with their next development efforts -- and Tower Technologies ought to make TowerJ available for free.