Nelson Minar likes distributed processing. So far in his career, first at the Santa Fe Institute and then at the MIT Media Lab, Minar has done basic research in loosely-coupled distributed processing, trying to get a handle on the nature of efficient distributed computing. At MIT's Media Lab, Nelson studied with Pattie Maes in the Software Agents group, where his work culminated in the release of Hive, a software platform for creating distributed applications.
Now, he's created Popular Power, a Silicon Valley company dedicated to linking cycle-wasting idle computers together to form virtual supercomputers. Popular Power will provide a source of income to the computer owners, and high-test computing power to Popular Power customers.
JavaWorld columnist Mark Johnson recently had the opportunity to speak with Minar about his company's business model, the technological challenges facing Popular Power, and Java's role in answering those challenges. You'll find numerous informative URLs, including a link to Popular Power, in the Resources section.
JavaWorld: What's the mission of Popular Power?
Nelson Minar: Our goal is to take all the computers out there on the Internet, make use of their idle resources, and resell to those who have large computation or network jobs.
JavaWorld: Tell me more about your first application, the influenza virus study.
Nelson Minar: Part of our business plan is to do a mix of nonprofit and for-pay applications. We're starting out with a couple of nonprofit applications, the first of which is this influenza research. The goal is to develop computer models and to do computer simulation of effective vaccines for the flu on the human immune system to help people develop more effective flu vaccines in the future.
JavaWorld: And so there's a human immune system model and a flu vaccine model, and the two models crunch against one another in this big distributed supercomputer?
Nelson Minar: Exactly.
JavaWorld: It seems that most distributed supercomputing projects like this focus on one of three kind of main areas: either mathematical problems like crypto cracking, prime number searches, and obscure number-theoretic problems; distributed rendering; or signal processing, like SETI@Home. Why are there so many of these projects in just these small areas and what new categories of projects do you envision for the future?
Nelson Minar: One of the early challenges of building this type of system was to find large problems that were easy to break up into lots of little bits of work. And in particular, with the earliest projects, like the key cracking, they were also trying to keep the code size and data size down, to make the networking problem simpler.
These problems we mentioned, like key cracking, were great examples of that. SETI@Home changed the model a little bit in that they started shipping around a fair amount of data. The average data set size in SETI@Home is about 300K. There we started to see more complex data processing and distributed computing. Popular Power brings a new capability to this idea. We're using mobile code to change the kinds of jobs that your computer runs.
Instead of having to get one carefully-tuned client that works exactly right and ship that out to everyone, we ship a metaclient, something that's able to do any kind of work. When we tell a client to do more work for us, we don't just send them some data, we also send them the program to run. And that allows us to have a very flexible distributed system, where we can dispatch different jobs all the time.
JavaWorld: So this is actually a coarse-grained, loosely coupled distributed computing infrastructure, not just an app.
Nelson Minar: Exactly. That's the key factor distinguishing our technology from what's been done in the past. We have a general purpose infrastructure, rather than individual applications.
JavaWorld: Obviously, problems that require low latency or involve data that are not cleanly partitionable, as you said, aren't appropriate for this kind of approach. Or are they? Are there any other nonrealtime, computationally intensive problems for which you think this kind of computing is better suited, or worse?
Nelson Minar: You mentioned a couple of problems that are hard for this kind of architecture. One of them: if you need incredibly fast response, like under a second. Obviously, that's not going to work very well with our sort of distributed system. The other one is when the problem does not break up into individual pieces. That's not very effective on our system.
However, we've found that many times if you look at these problems in a different way, it turns out we could be helpful anyway.
JavaWorld: I read recently where someone said a supercomputer is a device that turns a computing problem into an I/O problem.
Nelson Minar: [laughter]
JavaWorld: That's interesting because you're creating a supercomputer out of nonsupercomputers. Can you describe Popular Power in light of that comment?
Nelson Minar: We're taking thousands and thousands (or up to millions, at some point), of unreliable, small computers out there on the Internet, and we're ringing them all together with our dispatcher architecture to build one large, reliable supercomputer-type system. And, so, very simply, we get lots and lots of little jobs, and we dispatch them out to the clients. And most of our server-side technology is about how to do that dispatch effectively.
You mentioned I/O-type problems. All computing problems involve some sort of I/O. The Internet distribution actually puts higher requirements on that, right? Internet links are much slower than local area networks, and so part of the requirement in building a job into Popular Power is making sure that you manage that I/O. And, there are some problems for which that's not possible. If you need to work on a terabyte database, we're not going to be able to distribute that very effectively today.
But again, often, you could break up the data into smaller chunks and ship each chunk to a different node and that works fine.
JavaWorld: That's interesting about the I/O. You know, it's one thing to get organizations to donate processor time they're not using anyway. But it's another to ask them to put up with additional network traffic. With that in mind, how do you answer this potential objection in an organizational setting?
Nelson Minar: On an organizational side, we're providing tools so that an IT manager can control what the Popular Power clients on their network do. So, for instance, if they want to enforce hours of operation, the clients [could be] only allowed to run in the middle of the night. We offer that opportunity for an IT manager as a way to control the nodes on their network.
JavaWorld: If I'm an IT manager, it's easy for me to say, "Don't run those stupid things, because I don't want to be messing with this stuff." What's your story with them?
Nelson Minar: Popular Power actually provides an IT department the chance to make some money with their computers. Because they're able to use the spare resources of their own computers and get paid for that time. So there's a financial incentive for an IT department to use this on the public Internet.
We also have a local intranet version of this system where a big company that does drug design, for example, might have tens of thousands of computers on their intranet, internal network, and they would run Popular Power solely within their own network -- and do their own work inside of it. We're seeing a lot of demand for that type of capability.
JavaWorld: I see. So you can go to someone like, say, Pfizer, or Burroughs-Wellcome, and say, "How would you like to turn all these computers that are sitting on overnight, heating up your offices, and turn them into a virtual supercomputer?"
Nelson Minar: For your own application.
JavaWorld: With all the various distributed supercomputing projects appearing, (following the success of SETI@Home, again), what do you think are going to be the deciding factors in attracting participants to a company like yours?
Nelson Minar: There are several key points, I think, that are going to encourage people to run our system.
On one hand, you want to have the system not impact your own life on your own computer. If somebody's running Popular Power at home they probably don't want to be bothered by it when they're using their computer on their own. So, for instance, Popular Power, especially in Windows, runs as a screen saver. It only activates when you're literally not using your computer.
We've also put a fair amount of work into making sure that as soon as you move the mouse or return to the computer, it goes out of memory immediately. It's the "you don't want Popular Power to annoy you" incentive.
Another part is, "How does it improve your life? How does it help you?" There are a couple of different incentives here, the simplest one is that we're going to pay people for their time. So, if you run Popular Power, you can make somewhere between and 5 a month. That's 50 percent off on your Net bill, if you're on a dial-up.
Another incentive is the ability to contribute to these nonprofit projects -- that's actually been the primary incentive in the past. People just volunteer to do this stuff because it was fun, or they wanted to help out. It's amazing how many people really are motivated by that.
JavaWorld: So, more incentives?
Nelson Minar: Actually ... the first of these distributed computing efforts I ran was called DESCHALL. It was back in 1997, cracking a DES key. And there was this wonderful war that was going on between MIT, where I was a student, and Carnegie-Mellon, where we were running neck-and-neck. And, every week, they'd publish new stats, and one of us would pull ahead and start teasing the other people, and say, "Yeah, your computers are too slow. There are not enough of you." And they would work really hard to sign up a bunch of people and beat us the next week.
And, that kind of competition was great for the project, right? Suddenly thousands of more computers are signing up.
JavaWorld: So, you're harnessing not only excess compute time, but excess testosterone?
Nelson Minar: [laughter] There is a certain geek pride in that kind of thing. Again, it's the feeling that you're part of something bigger. In this case you're not just part of all of Popular Power, but you're part of the neighborhood, which is your local friends. And you can see how the neighborhood's doing as well.
JavaWorld: Let's go technical now. What projects have you worked on in the past that have influenced the design of the system?
Nelson Minar: The big project there is the Hive project, which was at the MIT media lab, which I headed up. Hive is an open source toolkit. You can actually get it off the Net [see Resources]. And what I was doing with Hive was trying to build peer-to-peer, decentralized systems with mobile agents.
It's a lot of the same rough technologies. You have lots of nodes out there that are all contributing to a larger network of computation. You have mobile code moving around the system. With Hive, I was trying very aggressively to build a really decentralized system. What it came out looking a lot like was Jini, but with a different design philosophy underneath it. I was working on device integration networks, so the applications were similar to some of the early Jini work.
JavaWorld: What sort of hardware are you using for your servers and where did it come from?
Nelson Minar:The servers are actually also built on Java. We're using WebLogic Server as our server platform. And the hardware they run on is currently VA Linux rack mounts. That will probably change some time in the future, but that's what we started out with.
JavaWorld: So you don't anticipate moving from one hardware server platform to another to be a concern?
Nelson Minar: No, not at all. The server is designed to be easily installable.
JavaWorld: Because of the portability of Java?
Nelson Minar: Correct. The big issue for us is that we need a server-side Java VM that is reliable and scalable. And the Linux Java is a little farther behind than I would like it to be. So, you know, at some point, we might switch over to Sun, just to get a more reliable Java environment.
I'm a big Linux fan, just as an aside, here. And I really much prefer to use Linux hardware and Linux OS, but as I said, the VMs are a little further behind in Linux right now.
JavaWorld: Does your system allow a node to work on several different projects, depending on whatever needs doing, or what your priorities are? Or does a node have to be signed up to do a particular project?
Nelson Minar: Oh, no. The nodes are able to switch projects at will.
JavaWorld: And they can switch projects? This week I'm doing influenza, and key cracking next week, and it just takes turns? Or do I need to switch projects, manually?
Nelson Minar: On the client side, you don't need to switch anything. There are a couple of ways that you interact with the jobs in our system. First of all, there's a screen saver that could change by different jobs. If it's running influenza, we have a flu screen saver. If you're running -- let me pick a simple example -- some sort of search-engine application, you might see some animation of the Internet being searched. So you would see different jobs on your screen.