Featured White Papers

• Measuring Risk to Improve Java Software Quality
• Do You Really Get Classloaders?
• Pragmatic Continuous Delivery with Jenkins, Nexus, and LiveRebel
• Your Next Java Web App: Less XML, No Long Restarts, Fewer Hassles
• Coding with JRebel, Java Forever Changed
• 2012 Developer Productivity Report: Java Tools, Tech, Devs, and Data

Sponsored Links

• Graph databases are 1000x faster than relational.

  Learn More!

• Career opportunities at Harman- Careers That'll Rock Your Socks Off.

  Learn More!

• Developer-friendly Java PDF libraries by Qoppa Software

  Live Demos

• Stop blaming your app! Diagnose Java performance issues.

  Free tool >

• UrbanCode: Go From Continuous Integration to
  Continuous Delivery

  Learn More!

Sponsored Links

Optimize with a SATA RAID Storage Solution
Range of capacities as low as $1250 per TB. Ideal if you currently rely on servers/disks/JBODs

Open source Java projects: Storm

Parallel realtime computation for unbounded data streams

By Steven Haines, JavaWorld.com, 12/05/12

Page 4 of 7

Storm topologies are deployed to the Nimbus and then the Nimbus deploys spouts and bolts to Supervisors. When it comes time to execute spouts and bolts, the Nimbus communicates with the Supervisors by passing messages to Zookeepers. Zookeepers maintain all state for the topology, which allows the Nimbus and Supervisors to be fail-fast and stateless: if the Nimbus or Supervisor processes go down then the state of processing is not lost; work is reassigned to another Supervisor and processing continues. Then, when the Nimbus or a Supervisor is restarted, they simply rejoin the cluster and their capacity is added to the cluster.

The relationship between the Nimbus, Zookeepers, and Supervisors is shown in Figure 2.

Figure 2. Nimbus communicates with Supervisors through Zookeepers (click to enlarge)

Setting up a Storm development environment

Storm can run in one of two modes: local mode or distributed mode. In local mode, Storm executes topologies completely in-process by simulating worker nodes using threads. In distributed mode, it runs across a cluster of machines. We'll set up a local development environment as follows:

1. Download the latest release of Storm and decompress it to your local hard drive.
2. Add the Storm bin directory to your PATH environment variable.

The bin directory includes the storm command, which is the command-line interface for interacting with a storm cluster. You can learn more about each command-line option from the Storm documentation. At a high-level, the storm command allows you to deploy your topologies to a storm cluster, as well as activate, deactivate, and kill topologies.

Creating a storm application in local mode is a great way to test out a Storm application before deploying it to a cluster, where the processing capacity will increase by orders of magnitude.

Storm demo: Crunching prime numbers

While you might not feel inclined to compute prime numbers for the rest of eternity, doing it once is a great demonstration of Storm's potential that is also easy to implement.

For this demo we'll create a NumberSpout that emits numbers, starting with 1 and incrementing until we kill the topology. To this spout we'll connect a PrimeNumberBolt that evaluates the number, which is passed in a Tuple, and prints the number out to the screen if it is prime. The NumberSpout and the PrimeNumberBolt are packaged together inside a PrimeNumberTopology class.

Note that you can download the demo application source code anytime.

NumberSpout

Listing 1 shows the NumberSpout, which is responsible for emitting numbers, starting with 1 and incrementing until the spout is killed.

Listing 1. NumberSpout.java

package com.geekcap.storm;

import backtype.storm.spout.SpoutOutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseRichSpout;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Values; 
import java.util.Map;

public class NumberSpout extends BaseRichSpout 
{
    private SpoutOutputCollector collector;
    
    private static int currentNumber = 1;
        
    @Override
    public void open( Map conf, TopologyContext context, SpoutOutputCollector collector ) 
    {
        this.collector = collector;
    }
    
    @Override
    public void nextTuple() 
    {
        // Emit the next number
        collector.emit( new Values( new Integer( currentNumber++ ) ) );
    }

    @Override
    public void ack(Object id) 
    {
    }

    @Override
    public void fail(Object id) 
    {
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) 
    {
        declarer.declare( new Fields( "number" ) );
    }
}

In Listing 1, the open() method saves the SpoutOutputCollector: Storm invokes this method when the spout starts up. The collector is the mechanism through which the NumberSpout emits numbers. Storm invokes the nextTuple() method to retrieve the next tuple to pass to subscribed bolts -- it builds a new Value object that contains the next number in the sequence and then increments that number so that the next invocation will send a new number.