The Volano Report: Which Java platform is fastest, most scalable? A JavaWorld exclusive!

Results of new VolanoMark 2.1 server benchmarks reveal how 10 virtual machines stack up on 6 OSs

Speed and scalability are the crucial elements for the successful deployment of a Java server platform, as any Java software vendor can attest. But most administrators possess neither the resources nor the time to test the various server-side Java platforms on the market. JavaWorld therefore turned to Volano and the VolanoMark 2.1 benchmark for answers.

The tests presented in this report look at the performance and network scalability of ten Java virtual machines on six operating systems using a common Intel hardware platform running VolanoMark 2.1. VolanoMark attempts to answer two questions about Java virtual machines used in network server environments -- especially those environments involving a large number of active socket connections:

  • Is it fast?
  • Does it scale?

VolanoMark isn't a Java virtual machine test. It's a Java platform test. It allows you to take one set of Java class files and use them to evaluate an entire Java system, including the Java virtual machine, the operating system, the hardware, and the network. Each of the tests in this article was executed on identical Intel hardware, providing an answer to the question our customers ask most -- Which operating system and Java virtual machine will provide the best Java server platform for my Intel box?

Unlike Java support for applets on the client side (where two virtual machines -- those provided by Microsoft and Netscape in their respective Web browsers -- dominate), the leaders in Java on the server side change monthly. Nowhere is that more apparent than for Intel-based systems, which provide dozens of Java VM alternatives. Volano customers can and do switch overnight from one Java virtual machine to another -- or even from one operating system to another -- in order to gain the best performance or network scalability from their Java applications.

What is VolanoMark?

VolanoMark is a 100% Pure Java server benchmark characterized by long-lasting network connections and high thread counts. In this context, long-lasting means the connections last several minutes or longer, rather than just a few seconds. The VolanoMark benchmark creates client connections in groups of 20 and measures how long it takes for the clients to take turns broadcasting their messages to the group. At the end of the test, it reports a score as the average number of messages transferred by the server per second. Its results have accurately predicted the real-world Java performance and scalability of our VolanoChat product line for more than two years.

Furthermore, VolanoMark seems to hit Java platforms where it hurts the most, not only creating a large number of active socket connections and threads but forcing the system to constantly switch among them all. The resulting scores are dominated by several factors:

  • Whether the Java virtual machine uses a just-in-time compiler (JIT)
  • Whether the Java virtual machine uses native threads
  • How many threads per process are supported by the operating system
  • How many network connections per process are supported by the operating system
  • How the Java virtual machine maps Java threads onto operating-system threads

VolanoMark 2.1 is available as a free download from Volano's Web site. (See the Resources section below.)

Is it fast?

Each time I run a set of VolanoMark tests against the latest batch of Java virtual machines I find some surprises, and this time is no exception. IBM now dominates VolanoMark performance, whether on Windows NT 4.0 or on its new OS/2 Warp Server for e-business. IBM's Java virtual machine for Windows NT is a remarkable 20 percent faster than the alternatives from Microsoft and Sun. On OS/2 Warp Server it even beats Tower Technology's TowerJ static compiler, showing that it is possible for normal Java virtual machines to surpass native performance, even if that's not yet happening on the same operating system.

PULLQUOTE: BLUE RIGHT Each time I run

a set of VolanoMark

tests I find surprises,

and this time is

no exception. :END_PULLQUOTE

The performance of Tower Technology's latest TowerJ 2.2.7 is still superb, as should be expected from a static compiler. TowerJ takes Java class files and converts them into C source code, which it then compiles into a native executable program. TowerJ 3.0, due for release in March, can dynamically load Java class files in both their original form and in their native TowerJ-compiled form. But unlike all the other Java virtual machines shown here, TowerJ isn't free. (See Resources for the pricing structure.)

Microsoft's and Sun's Java VMs on Windows both fall right in the middle of the pack, with Novell's NetWare Java VM right behind them. Sun's JDK 1.2 [aka the Java 2 platform --editor] on Solaris executes at about half the speed of the top performers, but once you discover its network scalability scores below, you won't care!

The Blackdown Java-Linux port now has a good just-in-time compiler called TYA, which gives it roughly a 50 percent performance improvement over its score without TYA. The latest release of the FreeBSD Java VM is very stable but suffers in performance from its lack of a just-in-time compiler.

Transvirtual's Kaffe OpenVM holds a lot of promise and even convinced a panel of 21 judges that it's the Best Virtual Machine. Unfortunately, that's not a conclusion I can share based on my own tests. For more than a year, I have been unable to run VolanoMark with Kaffe on several different operating systems. I'm still hopeful, however, since Kaffe holds the best promise of providing Java support for some non-Intel Linux systems, such as those from Cobalt Networks.

Run rules

The performance tests were executed with the following commands on a local loopback connection, using the heap size options shown below where possible:

Server:
java -ms8m -mx64m COM.volano.Main
Client:
java -ms8m -mx64m COM.volano.Mark -count 100

See the COM.volano.Mark command synopsis for a complete description of all options.

The operating system was rebooted before each set of tests for a particular Java virtual machine, and the first test result was discarded. The server side was restarted before each run of the client benchmark. The final score is the average of the best two out of three subsequent results.

Results

Scores are the throughput of the server in messages per second. Bigger numbers are faster. See the "Environments" section for details on the hardware platform, operating systems, and Java virtual machine environments.

Java platformScore

IBM JDK 1.1.7 OS/2

Tower TowerJ 2.2.7 Linux

IBM JDK 1.1.7 Windows

Microsoft SDK 3.1 Windows

Sun JDK 1.2 Windows

Novell JDK 1.1.5 NetWare

Sun JDK 1.2 Solaris

Blackdown JDK 1.1.7 Linux

JDK 1.1.7 FreeBSD

Transvirtual Kaffe 1.0 Linux

1857
1817
1660
1412
1344
1218
912
448
166
0

Figure 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second.

Java platformOperating systemResultsScore
IBM JDK 1.1.7 OS/2
OS/2 Warp Server for e-business1832, 1867, 18471857
Tower TowerJ 2.2.7 Linux
Red Hat Linux 5.2 Intel1820, 1801, 18141817
IBM JDK 1.1.7 Windows
Windows NT Workstation 4.01652, 1664, 16561660
Microsoft SDK 3.1 Windows
Windows NT Workstation 4.01413, 1407, 14111412
Sun JDK 1.2 Windows
Windows NT Workstation 4.01344, 1342, 13441344
Novell JDK 1.1.5 NetWare
NetWare 51216, 1216, 1220a1218
Sun JDK 1.2 Solaris
Solaris 7 Desktop Intel Platform Edition915, 909, 908912
Blackdown JDK 1.1.7 Linux
Red Hat Linux 5.2 Intel447, 447, 449448
JDK 1.1.7 FreeBSD
FreeBSD 2.2.8-RELEASE166, 166, 165166
Transvirtual Kaffe 1.0 Linux
Red Hat Linux 5.2 IntelSocketExceptionb---

Table 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second. The final score is the average of the best two out of three results. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Notes:

  1. Received "java.net.SocketException: Bad file number" 36 times on the third run, but the test completed successfully.
  2. Failed with "java.net.SocketException: Unimplemented socket option" when the server called Socket.setSoTimeout. Although the latest Kaffe build supports socket timeouts, it still fails to run VolanoMark.

Does it scale?

The fundamental problems with Java performance for Volano's products were solved back in the fall of 1997 with the release of Sun's JDK 1.1.3 on Solaris 2.6 and Microsoft's Internet Explorer 3.0 Java VM for Windows. Granted, there's no such thing as too fast, but even FreeBSD and Linux have Java support fast enough for all but the most demanding Java server applications.

Much more interesting is what happens to all that speed when we increase the number of simultaneous connections to the Java application. In my previous VolanoMark article for JavaWorld, I tested a dozen Java platforms for their performance in handling 100 to 900 concurrent connections. This time I pushed that number up to 2,100 connections. As you can see from the table below, there's still plenty of room for improvement.

Only two out of the ten Java platforms even survived the test, and only one survived without errors. Sun's JDK 1.2 on Solaris 7, still an early access release, finally shows it's possible to handle large numbers of connections to a single Java application without destroying performance (and without any errors). In fact, Sun's performance at 2,100 connections is only 23 percent below its performance at 300 connections!

PULLQUOTE: BLUE LEFT Only two out of the

ten Java platforms

even survived the test,

and only one survived

without errors. :END_PULLQUOTE

IBM also provided some surprises, breaking past VolanoMark's 1,000-connection barrier on Windows NT, which has held back Sun and Microsoft for the past two years. I've been asking IBM how it did that ever since I first saw the results, and the company agreed to disclose the not-so-secret secret (see sidebar). Microsoft and Sun, take notes.

IBM's JDK 1.1.7 on OS/2 Warp Server for e-business died after 600 concurrent connections because of a bug in its uniprocessor kernel that prevented the Java VM from using high virtual memory. IBM says the problem will be fixed in the final release of OS/2 Warp Server for e-business, allowing for at least 1,100 concurrent connections to a single Java application process.

TowerJ and the Blackdown Java-Linux port were unable to break past the Linux kernel's default limit of 256 file descriptors per process. For my previous VolanoMark article, I rebuilt the Linux kernel to allow up to 1,024 file descriptors per process only to discover that neither TowerJ 2.1.2 nor Blackdown JDK 1.1.6 took advantage of the change. The new Linux version 2.2 kernel should provide some relief in this area.

The FreeBSD Java port, along with the FreeBSD operating system, can easily handle up to 2,048 file descriptors per process, but its relatively poor performance makes attempts at anything greater than 600 connections unbearable. A good just-in-time compiler would solve that problem.

Novell's NetWare 5 Java VM, on the other hand, seems to be limited only by the amount of real memory on the system, although I was unable to obtain results above 1,500 concurrent connections with the 256 MB of RAM on my test system.

Run rules

The network scalability test was executed with the following commands over an isolated 10-Mbps Ethernet connection with a 10-Mbps hub, using the heap and stack size options shown below where possible:

Server:
java -ms8m -mx128m -ss32k COM.volano.Main
Client:

java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 15

java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 30

...

java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 105

See the COM.volano.Mark command synopsis for a complete description of all options.

The client test driver was executed under the IBM JDK 1.1.7 Java virtual machine using Windows NT 4.0 on a 200-MHz Intel Pentium Pro with a 256-KB L2 cache and 256 MB of RAM. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs except where noted.

Results

Scores are the throughput of the server in messages per second based on the total number of concurrent connections. Bigger numbers are faster. See the "Environments" section for details on the hardware platform, operating systems, and Java virtual machine environments.

2192
0
2535
2368
2340
2487
1528
0
237
1907
0
2181
1943
1831
2014
1461
0
135
0
0
1753
1554
1482
1844
1384
0
0
0
0
1499
0
0
1293
1337
0
0
0
0
1452
0
0
1505
1286
0
0
0
0
755
0
0
0
1241
0
0
0
0
498
0
0
0
1183
0
0
3006009001200150018002100

Figure 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections.

Java platform3006009001200150018002100Limit
IBM JDK 1.1.7 OS/2
21921907----a----------------~ 600
Tower TowerJ 2.2.7 Linux
----b------------------------247
IBM JDK 1.1.7 Windows
25352181175314991452755c498c~ 3500
Microsoft SDK 3.1 Windows
236819431554----a------------962
Sun JDK 1.2 Windows
234018311482----a------------907
Novell JDK 1.1.5 NetWare
24872014184412931505----d----~ 1500
Sun JDK 1.2 Solaris
1528146113841337128612411183~ 4000
Blackdown JDK 1.1.7 Linux
----e------------------------247
JDK 1.1.7 FreeBSD
237135----f----------------~ 600

Table 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware. The Limit is the maximum number of simultaneous VolanoMark connections possible (shown as an estimate when preceded by ~).

Notes:

1 2 Page 1