Newsletter sign-up
View all newsletters

Enterprise Java Newsletter
Stay up to date on the latest tutorials and Java community news posted on JavaWorld

Sponsored Links

Optimize with a SATA RAID Storage Solution
Range of capacities as low as $1250 per TB. Ideal if you currently rely on servers/disks/JBODs

JavaWorld.com > Java Development Tools >

Open source Java projects: Storm

Parallel realtime computation for unbounded data streams

By Steven Haines, JavaWorld.com, 12/05/12

  • Print
  • Feedback
.

Page 7 of 7

log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
1
3
2
7
5
11
...
3471031
3471283
3471317
3471407

The process runs for 10 seconds and in that time we are able to observe 3.4 million numbers.

Deploying a topology to a Storm cluster

Deploying a topology to a Storm cluster is an involved process, but once you go through it once, it will not be hard to go through it again. The Storm homepage hosts a good, in-depth Storm clustering tutorial but I'll summarize the process here.

  1. Define a class with a main method, just like we did for the PrimeNumberTopology class.
  2. Use the Storm TopologyBuilder to build a topology.
  3. Use the StormSubmitter class's submitTopology() method to name, configure, and submit your topology to the storm cluster.
  4. Package your topology, including spouts and bolts and helper classes, in a .jar file with the mainClass defined to point to your topology class, like we did in the pom.xml file in Listing 4.
  5. Execute the storm command (in your storm installation's bin folder) with the jar command, passing it a reference to your .jar file and the main class name (which uses the StormSubmitter.submit() method to submit the topology to the cluster).

At this point you can execute storm activate to activate your topology in the cluster, storm deactivate to deactivate your topology in the cluster, and storm kill to deactivate and kill (stop) a topology from running.

If you wanted to deploy the PrimeNumberTopology to a Storm cluster, your code would look something like this: 

Config conf = new Config();
conf.setNumWorkers(20);
conf.setMaxSpoutPending(5000);
StormSubmitter.submitTopology( "primenumbertopology", conf, builder.createTopology() );

In conclusion

Storm provides a framework for executing a continuous stream of data to produce business value. Storm applications are comprised of topologies that contain spouts that emit data and bolts that consume data or emit additional data for other bolts to consume. Storm differs from Hadoop in that Hadoop jobs have a target completion, whereas Storm topologies are meant to run indefinitely. In simple terms this means that Storm and Hadoop solve similar but different problems.

We've reviewed the use cases for Storm applications, the structure of a Storm application, and a Storm deployment environment and built a simple PrimeNumberTopology that computes prime numbers across the range of integer numbers. This example demonstrates how Storm tuples pass from a spout to a bolt to compute business value, and also should leave with you an idea of Storm's processing power, even running on a local instance.

See the Resources section to learn more about Storm.

About the author

Steven Haines is a technical architect at Kit Digital, currently working onsite at Disney in Orlando. He is the founder of www.geekcap.com, an online education website, and has written hundreds of Java-related articles as well as three books: Java 2 From Scratch, Java 2 Primer Plus, and Pro Java EE Performance Management and Optimization. He lives with his wife and two children in Apopka, Florida.

Read more about Tools & Methods in JavaWorld's Tools & Methods section.

  • Print
  • Feedback

Resources
  • Download the source code for this article's demonstration app.

Recent articles in the Open source Java projects series:

More about Storm: