"First bug bites Java 1.2 -- Mozilla stung" by Kieron Murphy
I am curious about what bug Felten and co. have found that needs to fixed in "all JVMs." I have looked in great depth at the JVM definition, and do not believe any class loading or type verification bug exists in the JVM.
"Early access" software clarification
I think "early access" to software is intended to flush out bugs. Early access software should not be used in a production environment; it should be used only by developers.
The sensational title of your article does not match its reality. How can you state in the title that the first JDK 1.2 security bug has been discovered when, at the end of the article, you state that it's not exploitable in the JDK?
Which is it? Your article states in the second-to-last paragraph: "I've just taken a look at Communicator 4.50 PR1, and the holes still exist there." However, the link you provide for Princeton University Secure Internet Programming (SIP) report of July 1998, clearly states:
This flaw is fixed in Navigator 4.5. We have verified that our demonstration applet does not work on Navigator 4.5.
Kieron Murphy responds
Readers, To begin, any defects in the composition of the article are mine alone. I stand by the article, but admit it could have been better. To clarify matters, I offer the following comments on some of the criticism the article has received. The main complaint seems to be that the conclusion does not satisfy the premise. One reader notes: "The sensational title of your article does not match its reality." I made several attempts to contact appropriate representatives at Netscape and Sun for comment on this point, but neither organization responded in any detail for the record. Informed sources told me, however, that the dynamic linking problem in JDK 1.x and all contemporary implementations of the Java VM is a serious flaw. A distinction must, of course, be made between the existence of a flaw and its actual exploitation. The importance of the Princeton hole, in my opinion, was due to its breaking the barrier of successful type confusion in current and planned designs -- especially, by mounting a three-phase attack. That type safety was only broken in Navigator 4.x should not be considered a disturbance; it should be an alarm. Princeton's SIP researchers have long been at work on a theory that the dynamic linking mechanism in Java may have unsound technical properties. The Hostile Applets home page's class loader bug, discovered earlier this year, and the security manager trick in Navigator simply provided the topography for an attack. In my view, they succeeded in accomplishing their mission. Java developers should be very defensive in creating code. Designing security policy for users is of paramount concern for the success of the language and platform. So, I do not think my premise and conclusion disagree. In a related vein, another reader writes that a discrepancy exists between the quote of Dr. Mark LaDue -- "I've just taken a look at Communicator 4.50 PR1 and the holes still exist there" -- and a statement found on the SIP site -- "This flaw is fixed in Navigator 4.5. We have verified that our demonstration applet does not work on Navigator 4.5." This person has me by the short hairs. Briefly, two answers. First, in editing the final copy, I probably quoted LaDue out of context. Though his hostile applets are essential to the overall Princeton attack, in going through my notes, I realize his remark applied only to his own applets at the time I wrote the article. So I am guilty of rushing. In this regard, I believe I owe JavaWorld readers an apology. Second, the SIP site currently states: "Last modified: Wed Jul 15 16:22:00 EDT 1998." In my notes, I have a printout of the SIP site taken on July 16. It does not include the statement in question. I do not know when this addition was made, but I did not use it in my article, because I was unaware of it. I did know that Netscape had been apprised of the hole about a week earlier. Another reader writes to remind me that "early access ... is intended to flush out bugs." I wish I could kiss this guy. Sometimes commentators of all stripes forget the big picture. Software development is just that: code in progress. News readers want to hear the latest dispatches from the frontlines, and news writers try to give them that, as rapidly and well as we can. We need to remember that we are all communicating with one another because we are trying to help one another. We are not on different sides. Kieron Murphy
"Which Java VM scales best?" by John Neffenger
For future consideration: IBM OS/400
I'm interested in seeing how the IBM OS/400 Version 4 Release 2 JVM compares to the other environments mentioned in your article -- especially against native-compiled Java. I bet I'm not alone in my curiosity. Maybe a future article?
IBM AS/400 VM -- "most scalable"?
I'm surprised you didn't include the IBM AS/400 Java VM in your benchmarks. IBM is advertising the AS/400 VM as the "most scalable" and only VM that addresses the threading and cleanup issues that have plagued other platforms. Also, as far as I know the AS/400 VM is the only pure 64-bit Java VM commercially available.
With the introduction of the 0K 170 AS/400 models earlier this year, the AS/400 platform should receive consideration when evaluating a scalable Java platform.
Jack and John, I would like to see how all of the other IBM systems perform with respect to Java: IBM AIX, OS/390, and OS/400. It's just a little harder (and it would be costly) to set those systems up myself in our lab. I would have included AIX but I was told IBM no longer supports an Intel version of that OS. I restricted my tests to operating systems that are available on the Intel platform, since it's impossible to compare Java virtual machine implementations once you start varying the hardware underneath. I made an exception with the Mac OS, since we have quite a few customers running Internet servers and our VolanoChat product on the Mac. I included the scalability tests on Solaris SPARC just to show that other hardware architectures can affect processor scalability, but only after I made the fair comparison to Windows NT with Solaris Intel Edition. Hewlett-Packard HP-UX, OSF, SCO OpenServer and UnixWare, and Silicon Graphics IRIX were also left out this time around. With well over 100 Java licensees, it's getting impossible to test all of the latest Java virtual machines on all of the available hardware platforms and still publish timely results. Once SPEC publishes its server-side Java benchmark suite, we'll be able to get these results directly from the Java vendors themselves. John Neffenger
What about Symantec?
Symantec claims its VM is faster than Microsoft's VM, but it isn't included in your comparison. Do you have any unpublished numbers for this VM?
Christof, I did include Symantec with the following Java virtual machines: JavaSoft JDK 1.1.6 JIT Update for JDK 1.1.6 Early Access 2 (Symantec JIT compiler Version x3.00.050) Novell JDK 1.1.5 Symantec Java! JIT compiler Version 3.00.040(x) for JDK 1.1.x JavaSoft JDK 1.2 Beta 3 Symantec Java! JIT compiler Version 3.00.023(x) for JDK 1.2 See the small print under each Java VM system configuration. John Neffenger
What do you mean by "restart"?
Great article. When I see the word scalability I always groan because of the inevitable misuse, but you were on the mark.
It would be interesting to know the CPU utilization of the VMs in the test, i.e., are the CPUs at 100 percent?
In our stress testing, we've noticed some VMs (almost all the recent Sun Solaris VMs) repeatedly core dumping under heavy load -- that's right, a traditional Unix-style core dump, time after time. We do net management, and this is with a message-processing throughput test that is very CPU bound. In fact it's sort of different than your tests in that the message is being parsed from ASCII and translated into CORBA. It isn't a connection-heavy test.
I'm also under the impression that, if the app threads are busy enough, some VMs never get around to running garbage collection. Have you heard this?
When you talk about having to restart a VM during a test to get up to a certain number of connections it's not clear what restart means -- is this the same as rerunning the test?
I noticed that some of the tests ran with the
-ss flag. Based on what may be voodoo, we've started running our apps with
-oss too, because it seems that there may be some garbage-collection/thread-depth bugs under Solaris.
Also, does your server examine or otherwise modify messages as they come in, or does it just basically pass messages out to all other sockets and chat users?
Dave, I'm glad you liked the article. CPU usage was 100 percent in the local loopback tests, but often fell below that when driving the fastest Java virtual machines over the network. With the fastest Java VMs, the test driver or the network itself eventually becomes the bottleneck. You should report those core dumps to Sun. Ever since we moved to the JDK 1.1.5 Solaris Production Release, we've seen no such errors, but we saw plenty of them prior to that release. I've never seen the garbage collector simply fail to run due to heavy CPU usage (this would result in a
java.lang.OutOfMemoryError). I tried to run the network connection scalability tests without restarting the server, meaning that I left the server running as I started up the client, first with 2 connections, then with 100, then with 200, and on up to 900 connections. In each of these tests, I started up the client with a new connection count. When I say I had to restart the server, it means that, for example, the test at 300 connections completed successfully, but then when I tried to start up the client with 400 concurrent connections, it failed. I then restarted the server, starting the client with 400 connections, and it worked. I haven't yet messed around with the
-oss flag. Many of the Java vendors suggested I reduce the native stack size (
-ss), but none of them mentioned the Java thread stack size. Finally, The VolanoMark server tries to do as little as possible with the message, but it does have to read-in the message (using a
DataInputStream), examine some of the strings, and write-out the message (using a
DataOutputStream). It uses a dynamic protocol based on object serialization, so it also uses the
Class.newInstance methods to deserialize the objects. John Neffenger
JDK 1.2 Beta 4 results
Just wondering if you got your VolanoMark numbers from JDK 1.2 Beta 4. Beta 4 is noticeably faster than Beta 3.
Also, have you found anyone to try HotSpot yet? Not that you could publish the results, but that would be an interesting test.
Stephen, I just tried JDK 1.2 Beta 4 and got the following result for the local loopback test:
Java virtual machine: JavaSoft JDK 1.2 Beta 4 Scores: 1275, 1284, 1288 Average (best 2 of 3): 1286
This result shows a 2-percent performance improvement over the JDK 1.2 Beta 3 release when running VolanoMark 2.0.0 Build 137. Other areas of the JDK not tested by VolanoMark may, however, have experienced much larger performance improvements. I haven't seen HotSpot yet, so I can't comment on it. John Neffenger
Client vs. server scalability
In an otherwise excellent article I have two negative comments:
You really ought to be testing with Warp Server SMP. OS/2 Warp 4 (client) is not an appropriate choice for heavy-duty server tasks requiring hundreds of connections, precisely what your benchmark is exploring. (Please let me know if you need a copy of Warp Server SMP.)
- Windows NT Workstation isn't appropriate for these tests either. If I understand the Microsoft licensing correctly, Workstation is not licensed for the number of connections you're throwing at it. You probably ought to be using Windows NT Server.
Again, thanks for exploring this issue.
Timothy, Actually, I used the client versions of all the operating systems that have both server and client packaging: Solaris 2.6 Desktop Intel Platform Edition, Windows NT Workstation 4.0, and OS/2 Warp Client 4.0. NetWare is only a server, and Linux and FreeBSD don't make the distinction. (For the processor scalability, I used Windows NT Server and Solaris Server.) As far as I can tell from my own tests, there is no difference in performance, stability, or scalability in VolanoMark between Solaris Desktop and Solaris Server, nor between Windows NT Workstation and Windows NT Server. IBM tells me there is a difference in those characteristics between OS/2 Warp Client and OS/2 Warp Server, so OS/2 may be the only system I tested that has a less scalable client version. Although Windows NT Workstation 4.0 is not licensed for connections from more than 10 computers, there's no problem in running a program like VolanoMark from a single computer that simulates hundreds or even thousands of client connections. To quote Microsoft, "The issue isn't about 'connections,' it is about computers accessing local data and operating system resources and services." For more information, see http://www.microsoft.com/ntworkstation/info/ntlicensing.htm We recommend Windows NT Server 4.0 when running our VolanoChat product on the Internet. John Neffenger