Functional requirements for mission critical applications often include software service level agreements (SLAs), which specify the maximum allowable latency for specific operations. Such requirements necessitate the monitoring of application performance, either at system testing stage or post deployment. This article will demonstrate how real-time performance tracing/monitoring can be achieved for spring applications using the functionality provided in Spring AOP and the Apache Log4j toolkit.

Introduction

Assume that as a solution developer/designer, you are given a list of semi-feasible functional requirements which a proposed application has to meet. Whilst reading through the list you parrot the seemingly non-controversial contents back to yourself. That is of course until you reach a line that reads something along the lines of "requests of type ‘y’ have to complete in ‘x’ seconds at the most".

Given that a typical application will have several points of failure which are not within your control as a solution designer, you will be forgiven for going back to your user base and explaining that such a requirement is quite simply not feasible. If you are really unlucky, you may have to go home and start working on yet another revision of your curriculum vitae. But since most users are actually reasonable people, you will probably achieve a compromise along the lines of "if requests of type ‘y’ are taking longer than ‘x’, then perform this action". Suffice it to say that the action to be performed falls under the general heading "details"; but you would have at least reduced your requirement to something a bit more tractable.

This type of functional requirement is no longer uncommon and is often referred to, interchangeably either as a Latency Requirement or Software Service Level Agreement (SLA for short). SLAs generally take the form "operations of type 'y' must complete within 'x' units of time". As an example, the backend for an online stock trading platform might include a restriction of the form "all trades must be submitted to the exchange within 5 seconds of being placed".

The challenges that performance SLAs present to software developers are two-fold. At the design and implementation phase, the emphasis is on ensuring that the application is built in such a way as to encourage fast enough throughput in order to bring the average-case latency in line with the SLA. The second challenge concerns the monitoring of the application's performance in real-time, either at system testing stage or post deployment, so as to detect/predict possible breaches of its SLA.

In this tutorial, we will focus on how to leverage the functionality of the Spring Framework to achieve real-time performance measurement and monitoring. This is a hands-on tutorial, so please download the companion source archive and load the files into your preferred IDE. The source files which comprise the archive are shown in Figure 1.

Note that, in addition to the Spring binaries, the lib folder of the companion archive also contains the binaries for the following open-source dependencies:

• Apache Log4J
• JAMon

Although not strictly necessary, it is advisable to download Apache’s Chainsaw application, which can be used for viewing the logs of remote applications.

For the benefit of the uninitiated, we will begin with a very simple primer on Aspect Oriented Programming (AOP); this will cover the basics needed to follow the examples in this article. For more detailed information, please see the links provided in the resources section. If you are already familiar with AOP, it may well be better to skip the next subsection altogether.

A Primer on AOP

Object Oriented (OO) programming provides for application entities and behaviour to be encapsulated in a set of classes. Although this is quite effective, and would suffice for most application, the truth is that there is certain behaviour in applications that appear everywhere and yet do not really belong anywhere. Examples abound when you look at non-AOP code in detail: the logging of entry and exit trace messages, operation-level authentication and authorization, and transaction control to name but a few.

AOP, or Aspect Oriented Programming to give it its full name, is a means of modularizing and externalizing functionality that spans multiple classes. In AOP terminology such functionality are referred to as cross cutting concerns. If we go back to the example on logging, a logging/error-reporting strategy actually spans multiple classes, and may not really be relevant to the functionality encapsulated by the logged classes. With AOP the log strategy can be externalized as a concern to which classes and methods can be added as and when necessary.

The main concepts from AOP which we are interested in for the purposes of this tutorial are: cross cutting concerns; advice; pointcut; and aspect. The first of these has already been described in the preceding paragraphs; the others are described next.

• Advice: Encapsulates externalized behaviour which can be applied to class instances at configured points during the execution of an application; for example, the logging of an entry or exit trace message can be thought of as advice types. Advice can also be thought of as the physical code which models/implements the logic of a cross cutting concern.
• Pointcut: A point in the execution of a program at which cross cutting concerns should be applied. This is the point at which the code encapsulated by an advice is executed.
• Aspect: An aspect is the combination of an advice and one or more pointcut definitions.

For more information on AOP, please see the resources section of this article. In the next section we will look at a basic service implementation, whose performance we will monitor via Spring AOP.

The Service to Monitor

The service to monitor is defined by the contract interface com.myorg.service.MyService, where the operation being monitored are implementations of the epsilon() method.

The objective of this tutorial is to show how the start and end time of invocations of this method can be intercepted, so as to provide the duration of such invocations.

We will focus on two means of monitoring with Spring AOP, with Spring's own PerformanceMonitorInterceptor and the JamonPerformanceMonitorInterceptor. The latter, although part of the spring distribution, relies on the JAMon application monitoring library. It is worth stating that the full features of JAMon (specifically the monitoring web-based UI) is only really available to web applications, and not standalone Java apps.

Although separate implementations of the service are provided for the Spring and JAMon interceptor examples, this is not a necessity for real world implementations. As you will see, the reason for doing so is merely to ensure clarity of object hierarchies in the tutorial's Spring configuration file.

The first example, covered in the following section, looks at Spring's PerformanceMonitorInterceptor.

Monitoring with Spring's PerformanceMonitorInterceptor

Using Spring's performance monitor is quite simple and can be done without embedding profiling code directly into application classes. To examine how this is done, we first of all need an implementation of the service interface com.myorg.service.MyService; this is provided by the class com.myorg.service.springmon.MyServiceSpringImpl which is shown in Listing 1.

Listing 1: MyServiceSpringImpl

The implementation of the contract method epsilon() simply causes the thread to sleep for a preset interval. Consequently, we should expect the performance monitor to register a value reasonably close to the sleep interval.

To make use of this service, we have to declare it in the application's Spring configuration file. Not only that, but we also have to indicate that the epsilon() method is a pointcut at which we want monitoring advice executed in any thread of execution cutting across it. The snippet of the configuration to which this relates is shown in Listing 2, which is an excerpt of the application configuration file beans-context.xml.

Listing 2: beans-context.xml

The configuration file opens with a definition of the service bean, with id springMonitoredService, and then a declaration of the performance monitor springMonitoringAspectInterceptor. The latter of these serves as the advice which we want applied at the invocation of the service's epsilon() method.

Having defined the service, and a representation of the advice, we specify the pointcut at which we want the advice applied. This is done within the <aop:config> element of the bean definition file. This contains a pointcut with id springMonitoringPointcut, which specifies a regular expression that matches all invocations of the epsilon() method in objects of type com.myorg.service..MyServiceSpringImpl. Observe the use of the double period (..) notation in the specification of the class name. This is akin to the pattern com.myorg.service.*.MyServiceSpringImpl. Observe also that the pointcut is actually a bean in itself.

Finally, all we need to complete this example is a definition of the aspect; which, if you recall, is a combination of the pointcut and the advice. This is contained in the element <aop:advisor>, which references both the pointcut and the advice to form an aspect.

Before we move onto the client application, please note that a log4j properties file is included in the config folder of the companion archive. This specifies appenders and layouts that will process the log statements generated by the sample application. At this point you may be wondering where such log messages will come from since we have not actually performed any logging thus far. The answer lies in the logic of the Spring performance monitor, which actually prints task durations via log4j using a TRACE level message.

Observe that the log4j file provided declares a SocketHubAppender, which will allow viewing of log messages with Apache's chainsaw. The use of chainsaw is not strictly necessary for this example, but is provided simply as a means of demonstrating how a complete solution can be setup in the real world. This is more relevant for the JAMon example as we do not have recourse to the JAMon GUI--it is only really applicable to web applications.

All that is left to do is to provide a client for invoking the service. A single client application is used for this tutorial; however, a different method is provided for invoking each example. As such, some light and judicious use of Java's commenting construct will be required to exclude the examples that you do not want to run. Listing 3 shows the main contents of the example client com.myorg.MyClientApp.

Listing 3: MyClientApp

The main method simply loads the application context, and calls the three methods which respectively invoke the service implementations corresponding to the examples provided in this tutorial. In this case, we are only concerned with the invocation
executeSpringMonitoringExample(context);, and as such we can comment out the other invocations at this stage.

Executing the example client results in output not too dissimilar to the following.

When viewed via chainsaw (i.e. by setting up a SocketHubReceiver on the port specified in log4j.properties), the output is of the form shown in Figure 2.

If you examine the output closely, you will note references to StopWatch; this, in fact, is the name of the Spring class that performs the actual task of timekeeping. To see how the class works, we will try to replicate the functionality in the last example without the benefit of AOP. In other words, we will actually embed the profiling logic directly in our service implementation. Apart from providing a more in depth view of how the monitoring code works underneath, the extra coding required should hopefully dissuade most people from doing this. And, as such, drive home the logic behind using AOP in place of anything else!

The class com.myorg.service.springmon.MyServiceWithEmbeddedSpringProfilling, which is shown in listing 4, is a different implementation of the service that invokes the performance monitoring functionality explicitly within the epsilon() method. It is easy to observe from inspection of the source code that performance monitoring is accomplished by simply instantiating a new instance of the class org.springframework.util.StopWatch.

Listing 4: MyServiceWithEmbeddedSpringProfilling

Like its physical namesake, the stopwatch simply records the time between the invocation of its start(...) and stop() methods. Information about the tasks performed between starting and stopping the monitor can be obtained by calling the getTaskInfo() method.

To execute this example, simply comment out all other invocations in the main method of the test client MyClientApp with the exception of

You will note that the above output is quite similar to the output from the first example. In essence, we have achieved not very much extra (in terms of functionality) by embedding the monitoring code directly in our application; other than of course cluttering the solution needlessly.

In the next section we will look at monitoring with the JAMon library.

Monitoring with Spring's JamonPerformanceMonitorInterceptor

JAMon consists of a library for performing latency monitoring as well as a web GUI which can be used in web applications to view the output from the monitoring library. Given that a number of applications do not run in a web container, the GUI is not a feasible option for many solutions. However, its base package offers functionality quite similar to that which is native in spring. Also, the spring distribution includes a performance monitor interceptor (JamonPerformanceMonitorInterceptor) which serves as a bridge to the JAMon toolkit. This can be combined with Apache's Chainsaw to replicate functionality that is not too dissimilar (for practical purposes) to that of the JAMon GUI.

Although not necessary in any other context, we will use a separate implementation of the service to illustrate monitoring with the JAMon interceptor. This implementation is shown in listing 5, and is only provided for the clarity of the Spring AOP configuration. As such, you will note that it is identical to the implementation of the service shown in the first example.

Listing 5: MyServiceJamonImpl

The definition of the service and the spring aspect are shown in the following excerpt from the spring configuration file beans-context.xml. Note that the only material difference between this and the first example is that the advice bean is an instance of the class org.springframework.aop.interceptor.JamonPerformanceMonitorInterceptor.

Listing 6: beans-context.xml

To execute this example in isolation of the other two, comment out all invocations in the main method of the test client MyClientApp.java with the exception of the line executeJamonMonitoringExample(context);. The output generated by doing so should be similar to the following listing.

Figure 3 illustrates the output of this example as seen through the Chainsaw GUI. Note that this example actually utilises the JAMon binaries which are included in the lib folder of the companion archive. An interesting exercise for the reader to perform is to export the service implementation as a web service, package and deploy the application to a web container taking care to also configure the JAMon GUI in the web application descriptor.

In the next section, we will examine some alternative techniques for performance monitoring, which do not fall within the remit of this tutorial.

Other Solutions for Performance Monitoring

To be clear, the technique described in this tutorial is reliant on AOP and may be difficult to apply to legacy applications­­especially if the applications are not built with the spring framework. That said performance analysis of legacy/existing applications can also be accomplished with a number of commercial tools (e.g. JProfiler) which allow developers to identify potential bottlenecks in existing code.

There are also more intrusive commercial third-party infrastructure components, such as JVMs, which offer facilities for real-time performance monitoring. Amongst the products in this space is RTPM from Azul systems. Choosing one of these tools obviously involves making implementation/deployment choices which may take solution developers down a route which is less travelled by those in their respective organizations.

Having said that, there are some real drawbacks to the Spring AOP approach. The most important of which is that there is no facility for asynchronous notification. If monitoring is enabled in the deployment environment, then it would be nice to have some means of issuing notifications when the application performance has degraded beyond a pre-set watermark.

If your monitoring requirements are complex in nature, than you may well be better off investigating a dedicated and commercial solution to the problem.

Summary

The emphasis of this tutorial was on how to augment spring applications, in a non intrusive manner, so as to obtain real-time performance metrics which can either be used to provide feedback to an iterative design process, or to monitor the behaviour/performance of applications in their deployment environment/home habitat.

The tutorial has provided a number of examples which demonstrate how to use classes native to the Spring application framework to achieve real-time performance reporting and monitoring. It has provided an example to show how Spring AOP can be combined with the JAMon application monitoring tool.

In the first installment of this three part series, I described the problem of deadlock; focusing on the logical conditions that are necessary for a deadlock to occur. I also introduced the deadlock-prone No Arbitration anti-pattern which is evident in applications where interdependent processes are allowed to contend for resources without effective arbitration to determine priority of access at critical junctures. An example of this was provided in the form of a train simulator.

This installment will examine yet another deadlock-prone anti-pattern, which we shall refer to by the moniker ‘Worker Aggregation’. Note that this installment also involves working through code samples, so please feel free to download this series' companion source code archive and load the files into your preferred IDE.

Worker Aggregation

An interesting experiment to try is as follows:

1. divert all voice calls from phone A to B
2. simultaneously divert all calls from B to A
3. attempt a call to phone A and observe the result

Have no fear, we are not about to write a telecoms switch simulator! The above example only serves to exemplify the conditions one must take into account when building systems that incorporate autonomous processes capable of self aggregation. Examples of self aggregating software include, but are not limited to, systems where processes can: cooperate with each other to break-down and perform composite tasks; divert portions of their tasks to other processes; subjugate pooled processes to perform sub-units of work. Process aggregation inevitably leads to chains of workers performing tasks in concert. In such architectures, a problem which becomes clear sooner or later is how to enforce acyclicity of process chains. Where cycles do occur, there is every possibility that the chains in question will never terminate, and in some cases may even deadlock.

Consider processes P₁ ,P₂, and P_n, such that they form a cyclic task chain of the form { P₁,P₂,...,P_n, P₁}, where P_n holds a resource which is required by P₁, and P_n cannot release the resource until P₁ has terminated. In such a scenario neither process will be able to proceed further; the chain will simply deadlock.

Let us simplify the above example by examining the same concept expressed in code. To do so we will use the compilation units contained in the package example.workeraggregation, which illustrates what happens when worker thread dependencies become cyclic. The package consists of the following source files:

The Worker class, shown in listing 6, models a worker process. Worker recruitment is encapsulated in the field coWorker. The worker also maintains an internal lock which is held whilst it is in service. Thus when a co-worker is involved in performing a task, its lock is held by the recruiting thread. This can be seen more clearly by examining the run() method.

Listing 6: example.workeraggregation.Worker

Note that a staggered sleep is used to simulate work before the co-worker is engaged. Now consider what would happen if we created two worker threads worker1 and worker2 such that the dependencies between them is as follows: worker1 -> worker2 -> worker1? This question is answered by executing the class WorkerRunner which is detailed in listing 7.

Listing 7: example.workeraggregation.WorkerRunner

Examining, or stepping through the code, you will notice that worker2 tries to acquire a lock already held by worker1, which in turn cannot terminate and release the lock until worker2 has completed. Voila, deadlock!

There are of course many ways to go about solving this particular conundrum. A brute force approach would simply be to adopt the same strategy applied by some telecoms engineers: prevent the dependency chain from exceeding a pre-determined size. A more complex solution is to build and enforce an acyclic worker-dependency graph in real-time as the application progresses.

To simplify the illustration, we will use the former approach to show how this situation can be prevented from occurring. Listing 8 details an extension Worker2 of the class Worker which includes a chainSize attribute that is used to track the size of the dependency chain. This attribute is also used to ensure that all second-hop dependencies are ignored.

Listing 8: example.workeraggregation.Worker2

WorkerRunner is also replaced with WorkerRunner2 (shown in listing 9) which is responsible for specifying the dependency chain size. Executing this class, produces the following output:

Listing 9: example.workeraggregation.WorkerRunner2

Summary

In this installment, we have looked at the problems involved in worker aggregation; specifically focusing on how cyclic worker chains can lead to application deadlock. Code examples were provided to demonstrate this, as well as to show how this anti-pattern can be modified so as to reduce the risk of deadlock.

The next, and final, installment of this series will examine the incremental locking anti-pattern.

Starvation occurs when one or more threads of execution are prevented from proceeding beyond a given point due to a predicate that will never be satisfied. Deadlock is a special form of starvation where threads of execution are prevented from making progress due to predicate conditions that are directly dependent on them. To illustrate, consider threads A and B, with shared resources R1 and R2. If thread A holds resource R1, and thread B holds resource R2; a deadlock will occur if thread A needs resource R2 to proceed and thread B is waiting on thread A to release resource R1. Since neither thread releases the resource it is holding, and both need each other’s resource to continue, they both become engaged in a deadly embrace and cannot proceed any further.

Considering that parallel threads of execution quite often have to rely on shared/mutual state, and that locking is used as the predominant means of preventing destructive updates, it is not surprising that deadlocks are an issue that multi-threaded application developers have to contend with at some point or another.

The adage “prevention is better than cure” is rather prescient when considering the problem of deadlocks. Actually, anyone who has had the displeasure of diagnosing deadlock will probably agree that the British and Australian version “an ounce of prevention is more valuable than a pound of cure” probably does a better job of emphasizing the contrast between the effort involved in preventing and fixing deadlocks respectively.

Thankfully (or sadly, depending on which way you look at it), deadlocks are mostly “man made” i.e. down to programmer error. Thus, preventing them can simply be a case of avoiding concurrency anti-patterns that have been shown to trigger them. We know from E.G. Coffman’s 1971 article that there must be four conditions for a deadlock to take place. These are:

1. Mutual exclusion: The concurrent threads of execution must share state which cannot be accessed by more than one thread (or class of threads) at any given point in time.
2. Resource hold and wait: Processes are not prevented from requesting new resources regardless of whether or not they are already holding other resources.
3. No Preemption: A processes may not be forcibly deprived of the resources which it currently holds.
4. A circular wait condition: This is the final and most identifiable condition required for a deadlock. It occurs when one or more resources form an interdependent and circular chain, such that each process T_i in the chain waits for a resource that is already held by the next process T_x in the chain, and T_x equally requires (directly or indirectly) the resource held by T_i.

Thus, preventing deadlock can be as simple as ensuring that all of the above four conditions never hold at the same time. Unfortunately life is not that simple. In reality, the first three conditions are actually quite useful for a variety of reasons which most concurrent application developers are fully aware off. For this reason, deadlock prevention often centres on preventing the final condition taking root.

In the following series of posts, I will explore three concurrency anti-patterns that can lead to circular wait conditions. In addition to describing these anti-patterns, this series will also show code samples to illustrate each one, and discuss solutions and workarounds to counteract them. Please note that the aim of this series is not to be a comprehensive collection of anti-patterns, but rather to encourage discussion around the topic. As such, readers are invited to contribute additional anti-patterns that they have come across—preferably with code samples.

This series is hands-on, so you are advised to download the companion source code archive, located at http://www.javaworld.com/javaworld/jw-03-2009/hpj032309-src.zip, and load the source files into your favourite IDE.

For this installment, let us begin with a relatively simple anti pattern, which I refer to as the ‘No Arbitration’ pattern.

No Arbitration

For a rather amusing take on deadlock-making functional requirements, examine this piece of legislation passed in the Midwestern US State of Kansas: "When two trains approach each other at a crossing, both shall come to a full stop and neither shall start up again until the other has gone".

One can only speculate as to the sequence of events that occurred the first time that two trains did actually happen upon each other at a crossing or indeed how long this statute lasted on the books.

The pertinent question however is: what would happen if we modelled this statute in software? To answer that question, we will use the example.noarbitration package in the companion source archive. The package consists of the following compilation units:

We will first of all explore the Train class which extends java.lang.Thread and which, as its name implies, models the behaviour of a train. This class is outlined in listing 1.

Listing 1: example.noarbitration.Train

Observe that the other attribute holds a reference to the other train at the crossing. As dictated by the statute, the run method models what takes place the moment the train arrives at the crossing. It waits for the other train to move before it itself starts to move.

We use the CrossingSimulator class, shown in listing 2, to encapsulate the behaviour of both trains as it relates to the crossing. As can be seen from the listing, the code creates and starts two train simulators (one southbound and the other eastbound).

Listing 2: example.noarbitration.CrossingSimulator

If we run the code in the previous listing, we should get the following output:

Not surprisingly, our trains never go anywhere once they reach the crossing; both threads quite simply deadlock, each waiting for the other to move. A rather perfect example of how not to run trains!

The question then is: what exactly is wrong with the model of concurrency on which this example is built? The answer in this particular case is quite simple; there is no arbitration between the trains to decide who gets to go first when conflict occurs. A situation that can quite easily occur when building systems based on independent and co-operative processes of equal priority and which rely on each other's internal state for deciding direction.

More formally, when there is contention between different threads of execution, a means must be present to determine the order in which the threads are granted access to the area or state in contention.

A simple solution is to introduce an arbitrator, which, when contention of this nature is detected, decides, based on some well defined scheme, which of the competing processes gains the upper hand. To apply this to the example model, some modifications are required to the Train and CrossingSimulator class logic, as well as some dedicated logic to provide arbitration.

The Train class is extended by Train2, as shown in listing 3, in order to provide three methods which play a key part in arbitration.

Listing 3: example.noarbitration.Train2

The isStopped() method indicates that the train is stationary; and forceTrainToMove() is invoked by the arbitrator to force the train to move. Finally, getArrivalTime() provides the time at which the train arrived at the crossing.

The arbitrator implementation is encapsulated in class TrainArbitrator, and is shown in listing 4. As is evident from the run() method, the arbitrator simply forces the train which has been at the crossing the longest to start moving. This will, in turn, enable the other train to move also.

Listing 4: example.noarbitration.TrainArbitrator

The executor class CrossingSimulator is also replaced by CrossingSimulator2, which is detailed in listing 5.

Listing 5: example.noarbitration.CrossingSimulator2

If we execute CrossingSimulator2, we should get the following output:

The output shows that when the arbitrator thread starts, it detects that both trains are standstill, and forces the eastbound train to move on, thus opening up the crossing for the second train.

Summary

Please note that the term arbitrator is used here in an abstract context. There are many ways to implement/achieve the same behaviour, and in fact, you may be able to achieve the same effect using concurrency constructs available in standard edition Java—depending of course on the structure of your problem and solution.

In the next installment, we will examine the Worker Aggregation anti-pattern.