Choosing a Java scripting language: Round two

If you are considering hooking a scripting interpreter into your Java application, the hardest part is choosing which one to use

By David Kearns, JavaWorld.com, 03/14/05

Print
Email
Feedback
Resources
Discuss

Digg
Reddit
SlashDot
Stumble
del.icio.us
Technorati
dzone

Page 2 of 5

Another way to protect yourself is to thoroughly test any scripting interpreter you plan to use. The distributions for some interpreters include a set of unit tests. When you test the scripting interpreter integration with your application, those unit tests can serve as part of the larger test suite you'll want to put together. When you test the integration between the interpreter and the application, you have your work cut out for you, because scripting interpreters are so flexible and expose so much functionality to the developer. You're making an investment by spending time on quality assurance early on, instead of when the application is in production or when customers need a critical bug fixed.

The new list of contenders

If you're looking for a scripting interpreter, there are plenty to choose from. Some interpreters have been written to support existing languages like Ruby, Python, JavaScript, Java, and Tcl. Other interpreters, like JudoScript, Groovy, and Pnuts, have chosen their own language syntax that is similar to Java but, in some ways, different. One of the biggest choices you'll have to make when you compare different interpreters is deciding what scripting language syntax is a good fit for you. Technology choices like this one, where developers' personal preferences come into play, can spark heated discussions among software development teams. Perhaps the information in this article can help settle some arguments. I collected the most recent releases of eight different scripting interpreters for comparison. The interpreters and their version numbers are listed below. If you are not familiar with these interpreters, I've also included a thumbnail sketch of the functionality and development activity on each one below.

Scripting language	Version number	Description
Jacl	1.3.1	The Java implementation of the Tcl interpreter. If you want to use Tk constructs in your scripts to create user interface objects, take a look at the Swank project for a set of Java classes that wrap Java's Swing widgets into Tk. Jacl has been around for quite a while and is still being actively worked on.
Jython	2.1	The Java implementation of the Python interpreter. One concern I have about this interpreter is that a new release has not been issued in quite a while. However, there are signs on the Jython Website indicating plans to reverse this trend, apparently with funding to back up the work.
Rhino	1.6.1	The Java implementation of the JavaScript interpreter. It also supports compiling the scripts into classfiles. The most recent Rhino release came out just a few months ago and added support for XML.
JRuby	0.8	The Java implementation of the Ruby interpreter. JRuby is under active development. I tested version 0.8, which worked well.
BeanShell	2.0 beta 2	BeanShell is a Java source interpreter. BeanShell continues to evolve and add new features. Version 2.0 provides full support for interpreting ordinary Java source code.
Groovy	1.0 beta 9	There actually are some groovy things about this new language that add some of the features of Python and Ruby to a Java-like syntax. You can compile the scripts directly into Java classfiles. Groovy plug-ins are available for numerous IDEs, and a JSR (Java Specification Request) committee is working on the language specification for Groovy.
JudoScript	0.9	JudoScript has a JavaScript-like programming syntax that is easy to learn and use. One of the stated goals mentioned in the JudoScript FAQ is to "support object-level, OS-level, and application-level scripting." I tested version 0.9, which worked well.
Pnuts	1.1 beta 2	Pnuts has a Java-like programming syntax and seems to be actively developed and released. You can compile the scripts directly into Java classfiles.

The first benchmark: Performance

For the first benchmark, I wrote equivalent scripts for each of the interpreters to do a set of simple tasks and then timed how long it took each interpreter to run the scripts. My benchmark scripts stick to basic operations like looping, comparing integers against lots of other integers, and allocating and initializing large one- and two-dimensional arrays. The benchmarking scripts for each of the languages and the Java programs to run them can be downloaded from Resources.

The most useful information that comes out of the benchmarking tests is an apples-to-apples comparison of how quickly the interpreters complete some extremely simple tasks. If throughput is a high priority for you, then the benchmarking numbers are useful.

I tried to code each test as similarly as possible for each of the scripting languages. The tests were run using Java 1.4.2 on a Toshiba Tecra 8100 laptop with a 700 MHz Pentium III processor and 256 MB of RAM. I used the default heap size when invoking the Java Virtual Machine.

In the interest of giving you some perspective for how fast or slow these numbers are, I also coded the test cases in Java and ran them using Java 1.4.2.

Here are the four performance tests:

Count from 1 to 1 million
Compare 1 million integers for equality
Allocate and initialize a 100,000 element array
Allocate and initialize a 500-by-500 element array

Have things improved since 2002?

Before I show you which of the interpreters is the fastest, I want you to look at the bar chart in Figure 1 that lists results for the most time-consuming task: comparing 1 million integers for equality. For the four scripting interpreters covered in the 2002 article, I show the times with the old release of the interpreter running on the 1.3.1 JVM and the interpreter's most current release running on the 1.4.2 JVM. Interestingly, when I tested Jython, I used the same version of the scripting interpreter I tested in my previous article, but the results show about a 25 percent improvement in speed.

Figure 1. Click on thumbnail to view full-sized image.

Since I ran the tests on the exact same hardware I used for my previous tests, I'd say that the 1.4.2 JVM substantially cut the time needed to run these benchmarks. Now look at what happened with Rhino, BeanShell, and Jacl. The newest release of Rhino running on 1.4.2 was 86 percent faster than the old release running on 1.3.1. For BeanShell, the improvement was about 70 percent, and for Jacl, about 76 percent. That's quite an improvement.

Total time for the four tasks

Since several of the interpreters closely resembled each other in terms of speed (at least for my benchmarks), I summed up the times for the interpreters on the four benchmark tasks and show the cumulative results in Figure 2.

Figure 2

The checkered flag

For these simple test cases, Rhino, Pnuts, and Jython were consistently the fastest, followed closely by Groovy, then JudoScript, and the others.

Whether these performance numbers matter to you depends on the kind of things you want to do with your scripting language. If you have many hundreds of thousands of iterations to perform in a scripting function and users of your application will be waiting on the result, then you might want to either focus your attention on the interpreters at the fast end of the spectrum, or plan on implementing your most demanding algorithms in Java code instead of in scripting code. If your scripts have few repetitive functions to run, then the relative differences in speeds between these interpreters is a lot less important. A faster computer can also make a big difference in these numbers.