A do-it-yourself framework for grid computing

Learn to build your own grid computing application using open source tools

By Anthony Karre, JavaWorld.com, 04/25/03

Internet computing is a term commonly used to describe the use of multiple networked computers to run dedicated computing applications. The most famous example is The Search for Extraterrestrial Intelligence (SETI) project, in which half a million home and corporate PC users voluntarily use their spare processing power to analyze radio telescope data for SETI.

The SETI project is typical of the Internet computing domain in that its application is both dedicated and vertically integrated. In other words, no generic use of the computing resources is or can be made, and all dataflow between the computing resource and the data server uses proprietary APIs. When an application upgrade is required, manual intervention is needed to download new executables, and typically no management layer exists for dynamically pushing new computing tasks or otherwise manipulating the available computing resources.

Transformation: Internet to grid computing

Driven by the success of the SETI project and others like it, researchers have been working to exploit the vast pool of computing resources connected to the Internet, but in a way that is secure, manageable, and extendable. In the last few years, various frameworks have been developed, with much emphasis on the management of shared resources. These frameworks typically provide the following features:

Machine independence through the use of Java and Web services
Security through Web service implementations like SOAP (Simple Object Access Protocol), with its built-in accommodation of HTTPS and user authentication
Task abstraction, that is, the ability for a management system to submit essentially arbitrary computing jobs to grid-based resources
Management of non-CPU resources, like file storage
Dynamic management of all of the above given a real-time assessment of the grid's status

Some frameworks have successfully evolved to the point where their architects have made them available in toolkit form. One example of such a toolkit is the Globus Toolkit 3.0, available through the Globus Project. This toolkit has been steadily evolving and has heavily influenced the development of the Open Grid Services Architecture (OGSA). Other published examples include a compute framework developed by Sun Microsystems researchers based on the Jxta peer-to-peer communication API.

Though powerful, these frameworks require a high degree of knowledge capital to successfully deploy and exploit. This can represent a barrier to entry of sorts for individuals or groups who wish to experiment with grid computing. For new programmers, this is like trying to use Struts for your first Web application, rather than writing your own servlets from scratch.

A minimal grid computing framework

In this article, I show you a small do-it-yourself framework that makes booting up a grid computing environment easy. This framework can be quickly developed using the common open source applications Tomcat and Axis, and, while simple, it still meets the following minimal grid needs:

Machine independence, achieved through the use of Java, Tomcat, and Axis.
Secure, scalable infrastructure, achieved through the use of SOAP-based Web services for client-server communication.
Abstracted task execution, achieved through the use of a client-side custom ClassLoader. With this ClassLoader, the client will be capable of identifying and executing arbitrary server-delivered Java thread classes.

Figure 1 illustrates a simplified view of the framework's architecture and execution sequence.

Figure 1. Simplified grid execution sequence. Click on thumbnail to view full-size image.

Build an example application

Throughout this article, I focus on building and testing this framework, and provide examples and illustrations in the following order:

Build a server-side SOAP service using Tomcat and Axis
Create connection stubs to support client-side use of the SOAP service
Build a custom client-side ClassLoader
Build the main client application
Build a trivial compute task designed to exercise the client ClassLoader

Build the SOAP service

The SOAP service I build in this article is the closest thing to a management layer that this framework will have. The SOAP service provides a way for our grid computing application to pull the classes it needs from the SOAP server. While my example service simply delivers a single specific jar file, this service's actual production version would likely have access to multiple jar files (each containing a different computing task), and it would contain additional logic to control which JAR was delivered to whom.

The first step in providing the SOAP service is to set up the SOAP infrastructure. I chose Tomcat as the servlet container/HTTP server because it is an open source project and proves to be extremely reliable and easy to use. I chose Axis as the SOAP services provider because it too is open source, supports an easy-to-use drag-and-drop service installer, and comes with a tool that creates SOAP client-side stubs from WSDL (Web Services Description Language) files (a feature I exploit later).

After downloading and installing Tomcat 4.0.6 and Axis 1.0, I wrote the SOAP service class GridConnection. This service fetches a known jar file, loads the file into a byte array, and returns the byte array to the caller. The following code is the entire file GridConnection.java:

////  GridConnection.java
//
import java.util.*;
import java.io.* ;
public class GridConnection {
  public byte[] getJarBytes () {
    byte[] jarBytes = null ;
          
    try {
      FileInputStream fi = new FileInputStream("/Users/tkarre/MySquare/build/MySquare.jar");
      jarBytes = new byte[fi.available()];
      fi.read(jarBytes);
      fi.close() ;
    }
    catch(Exception e) {}
    
    return jarBytes ;
  }
    
}

Deploy the SOAP service

The Axis Java Web service (JWS) deployment scheme eases the deployment of our new service. With Axis, you can write a simple service (like our own GridConnection class), change the .java file suffix to .jws, then drag and drop it into Tomcat's Axis webapp directory. Axis discovers the new .jws file, compiles it, and makes it available as a SOAP endpoint. Figure 2 shows where I put this file in my test environment and where the jws class files are created. Look for the GridConnection.jws file in the output of the Unix ls command.

1 | 2 | 3 | Next >