Remotely monitor Tomcat clusters using MC4J

Add the power of JMX to Tomcat

J2SE 5.0's monitoring and management support is comprehensive, with remote monitoring capabilities for the Java platform and for applications that run on it. Included in J2SE 5.0 is Java Management Extensions (JMX) remote monitoring, a new addition to the list of monitoring capabilities on a JVM. Java Specification Request 160 is the specification for JMX remote monitoring. It focuses on the mechanism for remote access by building a remote client API for a JMX-based agent.

Tomcat 5.5 is designed to leverage J2SE 5.0's built-in JMX capabilities. Version 5.5 (which was branched based on Tomcat 5.0.27) implements the latest Servlet (2.4) and JavaServer Pages (2.0) specifications and is the result of an extensive redesign and refactoring of the Tomcat server architecture. It proves more stable and improves upon Tomcat 4.x with enhanced performance, scalability and reliability, JMX monitoring, integrated session clustering, and application deployment.

With the latest version of Tomcat and J2SE 5.0's JMX capabilities, we can make the servlet container's attributes and methods available via JMX while simultaneously reducing the complexity of JMX-related code. In this article, I discuss how to launch the Tomcat servlet container with remote JMX monitoring enabled. We also look at a sample Web application that runs in a Tomcat cluster with session replication turned on. And, finally, we use a JMX client to view the details of cluster elements and HTTP sessions. But before we delve into the discussion of remote JMX monitoring, let's look at the components that make up a Tomcat cluster.

Cluster elements

A Tomcat cluster consists of six main components: cluster, membership, sender, receiver, replication valve, and deployer.

Table 1 shows how each component makes the clustering and session replication work in Tomcat.

Table 1. Tomcat cluster elements

Cluster elementClass nameDescription
Clusterorg.apache.catalina.cluster.tcp.SimpleTcpClusterMain element in the cluster configuration. Cluster creates the ClusterManager for all the Web contexts that have distributable tags in the web.xml.
Membershiporg.apache.catalina.cluster.mcast.McastServiceCluster membership is established by the Tomcat instances sending IP multicast pings. A multicast broadcast message contains the IP address and the server's TCP listen port. If an instance has not received the message within a given time frame, the member is considered unavailable. The attributes starting with mcastXXX are for the membership multicast ping.
Senderorg.apache.catalina.cluster.tcp.ReplicationTransmitterWhen session replication is requested, a Tomcat instance will use the host and port information and establish a TCP socket. Using this socket, the instance sends over the session information as serialized data. There are three modes of session replication in Tomcat 5.5: asynchronous, synchronous, and pooled.
Receiverorg.apache.catalina.cluster.tcp.ReplicationListenerThis element is responsible for the actual session replication. It listens on a specified IP address and port number for TCP cluster requests.
Replication valveorg.apache.catalina.cluster.tcp.ReplicationValveThe replication valve parses the HTTP requests and filters the files that do not need to be replicated. We don't want the session replication to trigger for static files such as HTML, JavaScript, Cascading Stylesheets, PDF, and images. We can filter these file types by specifying the list of file extensions that we don't want to replicate across the cluster.
Deployerorg.apache.catalina.cluster.deploy.FarmWarDeployerThis is a new element added to Tomcat cluster in the later versions of 5.0.x. It can be used for cluster-wide deployment of Web applications.

Fore more details on the clustering elements, refer to Tomcat 5.5's clustering documentation.

Tomcat 5.5 provides several improvements over its predecessors (Tomcat versions 4.1 and 5.0) in terms of clustering, session replication, and server monitoring and manageability. In Tomcat 4, several server components (such as host, engine, and service) could be monitored using MBeans (managed beans). But in Tomcat 5.5, Yoav Shapira, Filip Hanik, and the other Tomcat developers wrote JMX implementations for the cluster elements.

Now that we know the function of each element in a Tomcat cluster, let's look at various JMX client tools used for connecting to a Tomcat server cluster and monitoring the cluster details.

JMX clients

A JMX client is a graphical user interface (client/server or thin client) used to connect to a JMX agent (running on a local or remote machine). An ideal JMX client should have the following features to monitor an application server effectively without incurring any additional overhead:

  • Low overhead on system and network resources
  • The ability to maintain system stability and performance
  • Minimum or no special configuration needed (if some settings must be configured, they should be configured declaratively rather than programmatically)
  • Good reporting features

J2SE 5.0 comes with a JMX client tool called JConsole that can be used to look into the runtime JVM details. Tomcat installation includes a JMX servlet called JMXProxyServlet that can view and update Tomcat MBean attributes. It is a lightweight proxy for viewing and manipulating the MBeans running in a Tomcat container and proves helpful in integrating command line scripts for monitoring and changing Tomcat internals. JMX Query and Set commands can be used to query the MBeans and modify their attributes and operations respectively.

Other than these two tools, several third-party open source JMX client applications are available (links to which are available in Resources):

  • XMOJO
  • jManage
  • MX4J
  • Spring JMX
  • JMX-HTML adaptor
  • MC4J JMX Console

In this article, I discuss how to install and configure MC4J to remotely connect to a Tomcat servlet container and monitor all the MBean components in a server cluster.

JMX remote monitoring and management using MC4J

The MC4J console provides the following capabilities:

  • Tree view of MBeans, their attributes, operations, and notifications
  • Ability to set values on MBean attributes
  • Ability to execute MBeans operations
  • Connections to multiple J2EE application servers
  • Visual graphs of MBean attributes

I used MC4J 1.2 Beta 9 for this article's sample application. To install MC4J, download the executable (MC4J-12b9-Windows.exe) from SourceForge. Double-click on the exe file and select JDK Home and the MC4J installation directory (c:\dev\tools in the sample setup) when prompted during the installation process.

Set system properties for JMX

You must specify some system properties to see the remote JMX monitoring in action. These properties are specified in a properties file called management.properties located in the JAVA_HOME/lib/management directory. Table 2 shows the properties that must be set for JMX monitoring.

Table 2. System properties to enable JMX monitoring

Property nameValue
com.sun.management.jmxremote.port8999
com.sun.management.jmxremote.password.fileDefault location (JAVA_HOME/lib/management/jmxremote.password)
com.sun.management.jmxremote.access.fileDefault location (JAVA_HOME/lib/management/jmxremote.access)
com.sun.management.jmxremote.sslfalse

Setting the password file is very important especially in a multi-user environment. I set up the password file in the JAVA_HOME/lib/management directory as follows:

  • Copy the password template file (called jmxremote.password.template) into a file called jmxremote.password
  • Set file permissions so that only you can read and write the password file
  • Add passwords for the roles, such as monitorRole and controlRole
  • Set this system property when you start the JVM as shown in Table 2

Tomcat cluster configuration

I installed Tomcat 5.5.9 to set up the Tomcat cluster. The server cluster used in this article includes two Tomcat instances sharing the session state and a load balancer to distribute the requests between the server nodes. I used SimpleTcpCluster and DeltaManager (which are default options) in the cluster configuration. I explain the cluster setup in more detail in my series "Clustering and Load Balancing in Tomcat" (ONJava.com, 2004).

To enable remote JMX monitoring when starting Tomcat server, modify the Tomcat startup script (catalina.bat or catalina.sh) located in the CATALINA_HOME/bin directory as shown below:

 set JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.management.jmxremote.port=8999

Note: You must use a different port for the jmxremote port option on the second Tomcat instance (9999).

Figure 1 shows the cluster topology details.

Figure 1. Tomcat cluster architecture diagram. Click on thumbnail to view full-sized image.

Table 3 lists various configuration parameters used in setting up the Tomcat cluster.

Table 3. Cluster configuration details

ParameterCluster Node 1Cluster Node 2
Directoryc:\dev\tomcat-5.5.9-instance1c:\dev\tomcat-5.5.9-instance2
Server port80059005
Connector port80809080
Coyote/JK2 AJP connector port80099009
JMX port89999999
Multicast IP address228.0.0.4228.0.0.4
Multicast port4556445564
TCP listener IP address127.0.0.1127.0.0.1
TCP listener port40004001
JDK directoryC:\dev\java\jdk1.5.0C:\dev\java\jdk1.5.0

Connect to the JMX agent programmatically

Before we dive into the details of Tomcat monitoring using MC4J, let's briefly discuss a sample Java application that connects to a remote JMX agent (in this case, a Tomcat servlet container) using the remote JMX API. Begin by starting Tomcat with JMX monitoring enabled.

The sample JMX remote client is called RemoteJMXClient, which is basically a standalone Java application that acts as a JMX connector. This Java class is located in the sample Web application's src\com\remotejmx\client directory. Add the jmx-remote.jar and jmxri.jar files in the classpath when running the Java application. The following steps explain how to remotely connect to the JMX server:

  1. Start by creating JMXServiceURL with the protocol, host name, remote JMX port number, and credentials HashMap (with username and password):

     

    JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:8999/jmxrmi");

    Map map = new HashMap(); String[] credentials = new String[] { "monitorRole" , "QED" }; map.put("jmx.remote.credentials", credentials);

  2. Create a JMXConnector using the URL and credentials HashMap defined in Step 1. Once you obtain a reference to the JMX connector, call the getConnectionId() method to ensure you have a valid connection ID:

     

    JMXConnector conn = JMXConnectorFactory.connect(url, map); System.out.println("JMXConnector="+conn.toString());

    String id = conn.getConnectionId(); System.out.println("Connection Id=" + id);

  3. Next, get an MBeanServerConnection from the JMXConnector object:

      mbsc = conn.getMBeanServerConnection();
       String domains[] = mbsc.getDomains();
       System.out.println("# of domains="+domains.length);
       for (int i = 0; i < domains.length; i++) { 
          System.out.println("Domain[" + i + "] = " + domains[i]); 
       }
    
    
  4. Once you have the MBeanServerConnection object, call the MBeans-related methods just as you would call on an MBeanServer if connected to the JMX server locally (within the same JVM). You can check the number of domains and their types in the JMX server. You can also get the number of MBeans registered in the server and their attributes and operations. The code snippet below shows these steps:

     

    mbsc = conn.getMBeanServerConnection(); String domains[] = mbsc.getDomains(); System.out.println("# of domains="+domains.length); for (int i = 0; i < domains.length; i++) { System.out.println("Domain[" + i + "] = " + domains[i]); }

    // Get MBean count Integer mbeanCount = mbsc.getMBeanCount(); System.out.println("mbeanCount : " + mbeanCount.intValue());

  5. Next, query the MBeans in the server and display their attributes and operations. Also retrieve the MBeans details for Cluster's object type, as shown in the code below. We look at the same cluster details using MC4J in the following section.

     

    Set mBeanSet = mbsc.queryMBeans(null, null); System.out.println("mBeanSet.size() : " + mBeanSet.size());

    Iterator mBeanSetIterator = mBeanSet.iterator(); while (mBeanSetIterator.hasNext()) { ObjectInstance objectInstance = (ObjectInstance)mBeanSetIterator.next(); ObjectName objectName = objectInstance.getObjectName(); String canonicalName = objectName.getCanonicalName(); System.out.println("canonicalName : " + canonicalName);

    if (canonicalName.equals("Catalina:host=localhost,type=Cluster")) { // Get details of cluster MBean System.out.println("Cluster MBean Details:"); System.out.println("========================================="); getMBeanDetails(canonicalName); }

    String canonicalKeyPropList = objectName.getCanonicalKeyPropertyListString(); }

  6. Finally, close the JMX MBean connection to clean up the resources:

      conn.close();
    
    

Sample Web application setup

This section explains the sample Web application used to test the failover and session replication in the Tomcat cluster. I deployed the Web application on both cluster nodes and wrote a test client program to run a load test by creating and modifying HTTP sessions in the servlet container.

The following steps get the server cluster and load balancer up and running:

  1. Start the two server instances with the remote JMX monitoring feature enabled.
  2. Start the load balancer. I use a load balancer tool called Pen, a simple load balancer for TCP-based protocols. It distributes the load based on algorithms such as round robin and automatically detects servers that are down and distributes clients across the available servers in the cluster. For more details on how to install and configure Pen, refer to the Pen Webpage.

    I used the following command to start the load balancer using the round-robin load balancing option:

     pen -r -a -f -d localhost:8080 192.168.0.10:9080 192.168.0.20:10080
    

    where

    • r: Uses round robin for load balancing
    • a: Prints the data sent back and forth in ASCII format
    • f: Stays in foreground
    • d: Enables debug mode
  3. Launch MC4J by double-clicking on the executable file (C:\dev\tools\mc4j\MC4J Console 1.2b9.exe). (Note: In the sample application, I ran the JMX client and Tomcat cluster on the same machine, but in real-world scenarios, the JMX client would run in a remote machine, not the machine where the application server runs). Once launched, the MC4J application console looks like the screenshot shown in Figure 2.

    Figure 2 Screenshot of MC4J console window. Click on thumbnail to view full-sized image.

After starting the console, create a new connection to bind to the JMX server. Create a new connection called Tomcat-5.5.9-instance1 using the Create Server Connection option from the Management menu. Use the settings shown in Table 4 for Tomcat connection.

Table 4. MC4J Tomcat connection settings

Property nameValue
Connection Nametomcat-5.5.9-instance1
CredentialsQED
Initial Context Factorycom.sun.jndi.rmi.registry.RegistryContextFactory
PrincipalmonitorRole
Server URLservice:jmx:rmi:///jndi/rmi://localhost:8999/jmxrmi
Auto Connectfalse

Note: You need the catalina.jar, catalina-cluster.jar, and catalina-optional.jar files (located in %CATALINA_HOME%\server\lib\) specified in the classPathEntries property.

Once we have the Tomcat servers running with remote JMX enabled and have configured MC4J to connect to the servers, we need to run the test Java client and log the session details using Log4J. In the following section, we look at the instrumentation layer details and test the client runtime parameters.

Instrumentation layer

I ran the replication client in a multithread mode for a specified number of iterations. Session replication details (such as how long it took to propagate session changes across the cluster and time taken to process the request) were logged for every 100 requests. And JMX then monitored the console session details.

To run the test client and monitor the server statistics, follow these steps:

  1. Run replication client: I ran the SessionReplicationClient, to simulate the load test on Tomcat servers, using the following settings:
    • Number of threads: 2
    • Number of iterations: 1,000
    • Delay between requests: 1,000 milliseconds
    • Number of test samples: 1,000
  2. Monitor the cluster elements in the MC4J console: As shown in Figure 3, you can monitor the cluster details from the Cluster and ClusterSender components. The details such as replication mode (replicationMode), number of requests (nrOfRequests), and total data transferred (totalBytes) are displayed in the Properties window.

    Figure 3. Screenshot of Tomcat cluster MBeans. Click on thumbnail to view full-sized image.
  3. Check the session details using the session manager MBean: Figure 4 shows the details of HTTP sessions including session count, number of active sessions, and expired sessions. You can receive information on how long session replication takes from the log file.

    Figure 4. Screenshot of session details. Click on thumbnail to view full-sized image.

    You can see how remote JMX provides the API for viewing runtime details of a Tomcat server cluster and session replication, and how a JMX client lets us view these runtime details in a graphical user interface without writing any JMX code.

    Conclusion

    In this article, we looked at how to remotely monitor a Tomcat server (especially clustering and session replication modules) using JMX MBeans technology with the help of a JMX console (MC4J). You can see how easy and powerful it is to monitor a J2EE application server (or a servlet container) using the JMX capabilities provided by J2SE 5.0.

    Developers, network administrators, and operations groups will realize several benefits from using JMX technology for server monitoring and management needs. With remote JMX, developers can monitor their applications running on the server and find areas in the J2EE apps that need tuning and optimization to improve application performance and scalability. Network administrators can look at the server statistics, such as CPU usage, number of threads, and memory usage, to assess the current and future capacity planning needs. And the operations group can use remote JMX monitoring to detect any server-related problems in advance by checking server status and raising alerts when it reaches threshold levels.

    Security is another important factor to consider when using JMX to monitor application servers, especially in production environments. For example, access to the JMX console must be controlled by secured access (using username/password), so only authorized users can access, view, and modify the MBean attributes and operations. System administrators should control this access at the MBean attribute or operations level to achieve fine-grained access to the server monitoring consoles. Also, all JMX-based user activities should be tracked in a log file for future reporting and auditing purposes.

    A JMX client provides a centralized monitoring console for remotely binding to various application servers and for monitoring the server status and all applications running on these servers. This makes remote JMX monitoring a great solution for all enterprise server and application lifecycle management needs in a corporation.

Srini Penchikala presently works as an information systems subject matter expert at Flagstar Bank. His IT career spans more than 10 years with systems architecture, design, and development experience in client/server and Internet applications. He has been involved in designing and developing J2EE applications using Java and XML technologies since 1998. Penchikala holds a master's degree (Southern Illinois University, Edwardsville) and a bachelor's degree (Sri Venkateswara University, India) in engineering. His main interests include researching new J2EE technologies and frameworks related to Web portals. He has also contributed to ONJava.com, DevX.com, and TheServerSide.com. He enjoys spending time with his wife, Kavitha, and 6-month-old daughter, Srihasa, and watching Detroit sports teams.

Learn more about this topic

Join the discussion
Be the first to comment on this article. Our Commenting Policies
See more