What is it about benchmarks that makes them both universally reviled and, at the same time, impossible to ignore? Whatever their attraction, benchmarks got a lot of J2EE (Java 2 Platform, Enterprise Edition) developer attention this past year, with a number of controversial and high-profile benchmarks that left some developers scratching their heads and asking themselves, what does it all mean?
For J2EE and .Net, the benchmarking battles kicked off in June 2001, when Oracle published benchmark numbers showing Oracle 8i's performance with an unnamed J2EE server on a modified version of Sun Microsystems' Pet Store Demo training application. Microsoft published test results that, it claimed, showed a dramatic performance improvement over the Oracle numbers. Soon BEA and Macromedia had released Pet Store benchmarks of their own repudiating Microsoft's findings, and Pet Store, despite that it was created as a teaching tool and not a performance benchmark suite, was fast becoming ground zero for the J2EE/.Net benchmarking wars.
Two months ago, J2EE training company the Middleware Company stepped in and attempted its own benchmark of the Pet Store Demo, inviting Microsoft to tune the .Net implementation and use its own engineers to tune the J2EE version.
That's when all hell broke loose.
The Middleware Company's numbers initially seemed like very bad news for the Java community. J2EE was consistently outperformed on two-, four-, and eight-way systems; it required seven times the lines of code of .Net, and the unnamed app server the Middleware Company used for the test was twice as expensive as the Microsoft solution.
The Java community immediately shredded the results and said that the J2EE code was not properly optimized; the benchmark should have used container-managed persistence (CMP) Enterprise JavaBeans (EJB) or perhaps no EJB at all instead of bean-managed persistence (BMP) entity beans; and the Middleware Company was wrong to allow only one vendor—Microsoft—to participate in the benchmark.
The Middleware Company now admits it made mistakes in its initial tests and is talking to some J2EE vendors about conducting a second benchmark. According to a company spokesperson, the Middleware Company now believes that "until alternative architectures such as CMP and no EJB at all can be tested, definitive conclusions about the performance of J2EE cannot be drawn."
But Gartner Group Research Director Mark Driver says there is at least one message in the benchmark numbers: the idea of .Net not being scalable is a myth. "The ironic thing was that it was Java bigots [who did the benchmarks]...Their purpose was to show that Java was faster, and they did everything that they could to level the playing field and get Java to perform."
Hotly debated benchmark numbers aside, .Net has made some significant strides this past year. In February 2002, Microsoft released its Visual Studio .Net development tool, along with the Common Language Runtime (CLR) and .Net class libraries that make up the .Net Framework. Driver says this software is getting decent reviews from the approximately 10 percent of Microsoft developers who have tried it.
In fact, there are compelling practical reasons for Windows users to adopt .Net—something that Gartner Group expects 90 percent of Windows developers to do by 2006. They are the same reasons that moved DOS users to Windows, and Windows 3.x users to Windows 95. If developers want to take advantage of the Windows platform's latest performance improvements and technical features, they will need to get on the .Net bus.
But has .Net really proved itself? IDC Research Director Rikki Kirzner says it makes sense for Microsoft-only shops to evaluate the platform right now, because it represents Microsoft's future, but she cautions that Java-.Net technical comparisons are premature. "The .Net tools are too early," she says. "You can't compare a language that's as old as Java to a set of technologies that are just appearing." (See sidebar, "J2EE and .Net: A Quick Comparison.")
Taking the .Net plunge
One developer seriously looking at .Net is Kevin Jarnot, CTO of Boston-based DebtX, a company that brokers the sale of secondary loans over the Web. Jarnot, a long-time Java user, experienced a serious case of culture shock when he began working for the Microsoft-only DebtX in May 2002.
For Jarnot, one of the main problems with the Microsoft environment was the choice of development languages. "In Microsoft, you had VB Script, or you could do things in C++, which is the world's worst nightmare," he says. But after a few months of experimenting with C#, Jarnot says the Microsoft way has not been as rough as he expected, primarily because C# is "a really well thought-out language that is incredibly Java-esque," but also because he is very happy with the performance of Microsoft's XSLT (Extensible Stylesheet Language Transformation) engine, which he considers "better than anything that's out there for Java right now."
DebtX has spent the last few months prototyping its loan engine on the .Net platform and expects to have a production-ready system by the summer of 2003, when Microsoft is expected to ship its Windows .Net Server.
Jarnot's experience points to one of the most widely touted benefits of .Net: its language independence. While the vast majority of J2EE programming is done in Java, Microsoft claims that .Net will take a different approach. The idea is that language developers will be able to create compilers that compile code from any language—Perl or Cobol, for example—to the Microsoft Intermediate Language (MSIL) that runs in .Net's CLR.
Microsoft has already funded a .Net compiler for the Python language, according to Python creator Guido van Rossum. "The prototype worked, but was unacceptably slow and revealed that the .Net Framework—at the time—didn't have enough support for dynamic languages to make this practical yet." Van Rossum says that Python has a large library of extension models written in C—all of which would have to be rewritten for the MSIL to take full advantage of Python.
Gartner's Driver says he expects Microsoft's three currently supported languages will make up about 90 percent of .Net development over the next five years, but that "pressure from .Net" will push the Java community into adopting a more language-independent approach, something he expects to be realized around the Java 3 timeframe.
The question of openness
Another difference between Java and .Net is that Microsoft has submitted parts of its platform, specifically the C# language and interfaces to its CLR, to the European Computer Manufacturers Association (ECMA). These technologies were ratified by ECMA at the end of 2001, which particularly upset Sun. Sun had tried and failed to standardize the Java 2 Platform, Standard Edition (J2SE) through ECMA two years earlier.
"This was a very cynical move on their part," says Sun Chief Technology Evangelist Simon Phipps. "They've taken an unproven and unused technology, and introduced an early version of it into ECMA." Phipps says that when Sun was trying to standardize J2SE through ECMA, it attempted to standardize the libraries as well as the Java language, and that, by not standardizing important .Net libraries like Windows Forms or ASP.Net, Microsoft retains control of .Net's most important components.
While John Montgomery, director for Microsoft's Developer Platform and Evangelism Division, admits the .Net component being standardized via ECMA is quite small relative to the whole .Net Framework, he says that it remains a "dramatic subset" of .Net.
Still, for the vast majority of the .Net platform, Microsoft is in complete control; and, while Sun still holds a veto vote over its Java Community Process (JCP), the JCP has proved itself open enough to involve a wide variety of Java developers, including the open source Apache Software Foundation.
Microsoft's ironclad control of the .Net platform is a major issue for its customers, says Gartner's Driver. The issue will become more apparent as Microsoft delivers on the long task of porting its entire product line to .Net and begins to leverage its control of the platform into other product areas like Office and SQL Server. "[Customers] are very concerned that the degree to which they will fully be able to exploit the benefits of .Net will increasingly be tied to their willingness to lock themselves into Microsoft products," he says.
The more open nature of the JCP translates into a very different type of developer community, says J2EE advocate Rickard Öberg, a senior architect with Senselogic. While Microsoft may have a popular developer community with the Microsoft Developer Network, "it's mostly a one-way communication," he says, "whereas most people associate the Java community with a two-way communication, as in mailing lists, active open source development, and so on. There's a huge difference in mindset here."
Öberg, one of the harshest critics of the Middleware Company's benchmark numbers (he is a former Middleware Company employee), says that while the runtime attributes and "syntactical sugar" offered in the .Net platform are interesting, he expects these kind of features will soon be part of the Java platform, as standards like Java Specification Request (JSR) 201 and JSR 175 bring features like enumerated type support, automatic primitive type-reference type wrapper conversion, and a metadata facility to Java.
As far as Öberg is concerned, .Net is simply a Java clone with "some neat new features" that does not have Java's cross-platform support or strong developer community. "I don't see why anyone would chose .Net, that is, from a technical point of view. From a political point of view, all bets are off."
Learn more about this topic
- The Middleware Company's J2EE/.Net benchmark
- Rickard Öberg's review of the Middleware Company's benchmark
- "The Pet Store...Again," Ted Neward's article claiming that the Middleware Company's numbers are an indictment of EJB, not Java (O'Reilly, November 4, 2002)
- Two Java-friendly takes on benchmarking .Net and Java:
- Sun's J2EE homepage
- Microsoft's .Net homepage
- JSR 201
- JSR 175
- "Microsoft .Net vs. J2EEHow Do They Stack Up?" Jim Farley (O'Reilly, August 2000)
- "Migrating Java Applications to .Net," Nitin Nanda and Sunil Kumar (JavaWorld, January 2003)
- "Rumble in the JungleJ2EE Versus .Net," Humphrey Sheil and Michael Monteiro (JavaWorld)
- "C#A Language Alternative or Just J--?" Mark Johnson (JavaWorld)
- Browse the Java 2 Platform, Enterprise Edition (J2EE) section of JavaWorld's Topical Index
- Visit the JavaWorld Forum's Enterprise Java dicussion
- Sign up for JavaWorld's free weekly Enterprise Java email newsletter
- You'll find a wealth of IT-related articles from our sister publications at IDG.net