The need for speed, stability

Raw VM performance improves across the board in the latest VolanoMark 2.1.2 server-side benchmarks, but network scalability remains a thorny issue

Tower Technology's TowerJ 3.1.4 for Linux dominates 15 other VMs in both performance and scalability, according to the latest VolanoMark 2.1.2 benchmarks published on October 3 by Volano LLC.

Moreover, compared to previous VolanoMark results published on May 31 (see Resources for a link to this report), raw server-side VM performance on the benchmark improved for all vendors and platforms. This trend should continue as vendors compete to produce the top VM.

Even with the increased performance, however, the network scalability tests -- with two exceptions -- show disappointing results, with some VMs incapable of handling anything but the simplest concurrent connection levels.

VolanoMark explained

The VolanoMark 2.1.2 pure Java benchmark measures both raw server performance and server network scalability performance. Both tests pit 16 VMs on seven operating systems against each other on the same 200 MHz Intel Pentium Pro hardware, thus achieving a good apples-to-apples comparison.

For both tests, the VolanoMark benchmark creates client connections in groups of 20 and measures how long it takes for the clients to take turns broadcasting their messages to the group. At the end of the tests, it reports a score as the average number of messages transferred by the server per second.

The raw performance test runs with the server and all of the simulated clients on the same machine, communicating over a local loopback connection, running at a static 200 concurrent socket connections.

In contrast, the network scalability test moves the simulated clients onto another, more powerful machine and communicates with the server over a 10 Mbps Ethernet connection. The client side runs four times for each VM, simulating 1,000, 2,000, 3,000, and 4,000 concurrent socket connections. Only those VMs that successfully scale up to 4,000 connections pass the test.

In both the raw performance and network scalability tests, the higher the score, the better the result. (See Resources for more on the hardware and test methodology.)

Performance: Speedier than a fast bullet

VM bragging rights rest on speed, speed, speed, and in this test round, vendors do not disappoint. In terms of raw speed, the 16 tested VMs showed good results across all vendors and platforms -- a big plus for the Java community.

TowerJ above the rest

For raw performance, Tower Technology's TowerJ 3.1.4 for Linux finished first, with an average score of 2,309, as shown in Figure 1 below. TowerJ 3.1.4, however, is the only static compiler in the test group. As such, it takes class files and converts them into C source code, which it then compiles into a native executable program (with version 3, TowerJ can also dynamically load Java class files in their original byte code form). It's also the only contestant in the group not available for free. "TowerJ is a native compiler that costs thousands of dollars," says Volano CTO John Neffenger. "It's expected that they'd be on top."

Figure 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second

Source: Volano LLC

Java PlatformScore
Tower TowerJ 3.1.4 Linux
2309
2,309
IBM JDK 1.1.8 Windows NT
2279
2,279
IBM JDK 1.1.8 OS/2
2136
2,136
Microsoft VM 3229 Windows 2000
1970
1,970
IBM JDK 1.1.8 Linux
1770
1,770
Sun HotSpot 1.0.1 Windows NT
1601
1,601
Sun JDK 1.3 Beta Windows NT
1550
1,550
Sun JDK 1.2.2 Windows NT
1485
1,485
Microsoft VM 3186 Windows NT
1478
1,478
Sun JDK 1.2.1_04 Solaris
1358
1,358
Novell JDK 1.1.7 NetWare
1229
1,229
Sun JDK 1.2.1_03 Solaris
1051
1,051
Sun JDK 1.2 Linux
915
915
Transvirtual Kaffe 1.0b4 Linux
389
389
Blackdown JDK 1.1.7 Linux
285
285
JDK 1.1.8 FreeBSD
173
173

Neffenger adds that, among the standard pure Java VMs, IBM's various entries closely follow TowerJ. Indeed, IBM VMs held an impressive three of the top five spots in the raw performance tests. Big Blue's JDK 1.1.8 for Windows finished second with a 2,279 score, just 30 points behind TowerJ -- a 1.30 percent difference. IBM also locked up third place, with the IBM 1.1.8 for OS/2 VM scoring 2,136.

The latest entrant to the VolanoMark race -- Microsoft's forthcoming VM 3229 for Windows 2000 -- finished fourth with a 1,970 score, just ahead of IBM's fifth-place IBM 1.1.8 for Linux VM, which scored 1,770.

As for Sun, its top performing VM -- the highly marketed HotSpot 1.0.1 for Windows -- placed sixth, with 1,601, followed by the Sun JDK 1.3 Beta for Windows NT (1,550) and Sun's JDK 1.2.2 for Windows NT (1,485).

Table 1 below further details the performance tests by listing each entrant's Java platform, operating system, four test results, and final average score.

Table 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second. The final score is the average of the last three results. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Source: Volano LLC

Java PlatformOperating SystemResultsScore
Tower TowerJ 3.1.4 Linux
Red Hat Linux 6.0 Intel2,303, 2,313, 2,313, 2,3002,309
IBM JDK 1.1.8 Windows NT
Windows NT Workstation 4.02,282, 2,276, 2,280, 2,2822,279
IBM JDK 1.1.8 OS/2
OS/2 Warp Server for ebusiness2,145, 2,141, 2,118, 2,1502,136
Microsoft VM 3229 Windows 2000
Windows 2000 Server Release Candidate 21,948, 1,965, 1,976, 1,9681,970
IBM JDK 1.1.8 Linux
Red Hat Linux 6.0 Intel1,242, 1,772, 1,787, 1,7501,770
Sun HotSpot 1.0.1 Windows NT
Windows NT Workstation 4.01,558, 1,611, 1,604, 1,5891,601
Sun JDK 1.3 Beta Windows NT
Windows NT Workstation 4.01,538, 1,550, 1,556, 1,5451,550
Sun JDK 1.2.2 Windows NT
Windows NT Workstation 4.01,496, 1,481, 1,489, 1,4851,485
Microsoft VM 3186 Windows NT
Windows NT Workstation 4.01,469, 1,484, 1,478, 1,4731,478
Sun JDK 1.2.1_04 Solaris
Solaris 7 Desktop Intel Platform Edition1,371, 1,344, 1,374, 1,3551,358
Novell JDK 1.1.7 NetWare
NetWare 51,227, 1,232, 1,229, 1,2261,229
Sun JDK 1.2.1_03 Solaris
Solaris 7 Desktop Intel Platform Edition1,118, 1,055, 1,050, 1,0491,051
Sun JDK 1.2 Linux
Red Hat Linux 6.0 Intel915, 914, 912, 919915
Transvirtual Kaffe 1.0b4 Linux
Red Hat Linux 6.0 Intel397, 391, 390, 386389
Blackdown JDK 1.1.7 Linux
Red Hat Linux 6.0 Intel284, 285, 286, 283285
JDK 1.1.8 FreeBSD
FreeBSD 3.2-RELEASE169, 173, 173, 172173

Improvements all around

Aside from these individual VolanoMark performances, the tested VMs as a group continue to show significant performance improvements over previous versions, when compared to VolanoMark tests reported on May 31 (see Resources for a link to this report). This improvement applies to all vendors and platforms in the test group. For example, this round's leader -- TowerJ version 3.1.4 for Linux -- performed approximately 21 percent faster than version 3.0.1's May 31 score.

"We're three years into having good virtual machines, and [VM vendors] are still coming out with these great performance improvements," states Neffenger.

The good news about raw performance aside, VolanoMark also measures network scalability, where the various VMs show mixed results.

Stability and scalability: Still a ways to go

The network scalability results shown in Figure 2 below demonstrate that most vendors' VMs still need some work, at least in terms of long-lasting stability at high concurrent connections. Indeed, the focus for many Java developers has shifted from raw VM performance to stability and scalability.

"The performance issue for us [Volano LLC] was solved in August and September 1997 when Microsoft and Sun gave us really fast VMs. Since then, the problem has been handling thousands of connections to a single Java application and being able to keep it running for months, or even years," says Neffenger.

Figure 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections

Source: Volano LLC

2486
2623
1521
1408
1474
1468
1875
1575
1794
612
165
37
1357
1826
708
856
770
1040
1058
739
1081
406
523
659
613
421
354
1,0002,0003,0004,000

There can be only two

As in the raw performance tests, TowerJ 3.1.4 for Linux won the scalability tests, followed by Sun's JDK 1.2.1_03 for Solaris. Notably, among the 16 VMs tested, these two were the only VMs able to finish the test by successfully scaling up to 4,000 concurrent connections without producing any errors -- a milestone that tripped up other promising contestants.

Problem children

On the downside, Table 2 below shows that the remaining 14 VMs were incapable of finishing the network scalability test, each producing some sort of error before reaching 4,000 concurrent connections. IBM's JDK 1.1.8 for Windows NT, for example, led all comers in network scalability performance up to 3,000 concurrent connections, but then failed before it reached 4,000.

Table 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Source: Volano LLC

Java Platform1,0002,0003,0004,000Notes
Tower TowerJ 3.1.4 Linux
2,4861,357739421No errors!
IBM JDK 1.1.8 Windows NT
2,6231,8261,081----java.net.SocketException: Socket read failed: 10107 at 3,553/4,000 connections
IBM JDK 1.1.8 OS/2
1,521------------java.lang.OutOfMemoryError at 1,098/2,000 connections
Microsoft VM 3229 Windows 2000
1,408708406----java.lang.OutOfMemoryError at 3,785/4,000 connections
IBM JDK 1.1.8 Linux
----------------Process hangs in hard run at 500/1,000 connections and dumps core when killed
Sun HotSpot 1.0.1 Windows NT
----------------java.lang.OutOfMemoryError at 902/1,000 connections
Sun JDK 1.3 Beta Windows NT
1,474856523----java.lang.OutOfMemoryError at 3,272/4,000 connections
Sun JDK 1.2.2 Windows NT
----------------java.lang.OutOfMemoryError at 926/1,000 connections
Microsoft VM 3186 Windows NT
1468770--------java.net.SocketException: No available buffer space on 3,000 connection test
Sun JDK 1.2.1_04 Solaris
1,8751,040659----Bus error - core dumped on 4,000 connection test
Novell JDK 1.1.7 NetWare
1,575------------Process hangs with disk thrashing at 1,960/2,000 connections
Sun JDK 1.2.1_03 Solaris
1,7941,058613354No errors!
Sun JDK 1.2 Linux
----------------Tests fail with java.net.SocketException: Interrupted system call
Transvirtual Kaffe 1.0b4 Linux
612------------Dumps core at just over 1,000 connections
Blackdown JDK 1.1.7 Linux
165------------Dumps core at just over 1,000 connections
JDK 1.1.8 FreeBSD
37------------Client begins failing at 1,017 connections with java.net.NoRouteToHostException: Connection timed out

In fact, four VMs -- IBM's JDK 1.1.8 for Linux, Sun's HotSpot 1.0.1 for Windows NT, Sun's SDK 1.2.2 for Windows NT, and Sun's JDK 1.2 for Linux -- were not even able to handle 1,000 concurrent connections, the first rung on the VolanoMark scalability ladder.

Moreover, Table 2 shows that, with the exception of TowerJ 3.1.4 for Linux, VMs for Linux seem particularly susceptible to poor scalability results, a problem Neffenger blames on an incompatibility between Java's I/O model and the Linux model.

"I'm disappointed in the network scalability on Linux. IBM on Linux couldn't even do the test. The solution is to do a different threading model for Java on Linux, similar to the TowerJ solution, or better, as Sun did on Solaris," he states.

Threads and sockets: The network scalability dilemma

The scalability problem revealed by the VolanoMark tests comes down to a conflict between how Java handles threads and sockets and how the underlying operating systems handle them, says Neffenger. Java, for instance, requires one or two dedicated threads in order to read and write data from a socket connection. On the operating systems side, the Windows platform limits the number of threads handled in a single process, while Unix-based systems limit the number of sockets you can give to a single process (because of the file descriptor limits). If the VM vendor isn't careful, this incongruity can result in runtime errors at increasing concurrent connection levels.

Therefore, according to Neffenger, success in the VolanoMark network scalability test depends upon how the VM vendors mapped Java's view of threads and sockets onto the underlying operating system's view.

For example, with the IBM JDK 1.1.8 for Linux VM, IBM mapped the Java threads one-to-one to the Linux threads. "For long-lasting, dedicated connections to a pure Java server, that particular model is a disaster," says Neffenger.

With TowerJ 3.1.4 for Linux, on the other hand, Tower Technology mapped all of the Java threads onto one single process in Linux. This many-to-one model, as the network scalability results attest, seems to work.

Neffenger adds that, with the Sun JDK 1.2.1_03 for Solaris VM, Sun split the difference by following a many-to-many model. "They've taken a selection of Java threads and mapped them onto one Solaris thread, and, vice versa, they've taken one Java thread and mapped it onto different Solaris threads. Sun claims that's the best model." (For more a detailed look at the threads and sockets dilemma, see Resources.)

Whatever the issues, two successful, error-free VMs out of 16 does not bode well for network scalability, and as Java's focus shifts to server-side applications, the Java community can only hope this will change.

Conclusion

Whatever technical solutions vendors need to bring scalability to their VMs, TowerJ 3.1.4's results demonstrate that you can combine both high raw performance and long-lasting stability at high concurrent connection rates. The other VM entrants ought to look at the whole picture with their next development efforts -- and Tower Technologies ought to make TowerJ available for free.

Related:
1 2 Page 1