Sun's main Java site has an intriguing interview(http://java.sun.com/developer/technicalArticles/Interviews/community/pepperdine_qa.html) with Java performance expert Kirk Pepperdine that covers a lot about bottlenecks, pain points, tuning, memory, databases, etc. But what's most interesting are his methods of calming down a team of stressed-out, frustrated developers when he's been hired by companies to step in and solve their performance problems.
Among other things, he's "rolled VMs through a cluster; neutered the HTTPSession object; used GC to slow down certain parts of the application to improve throughput; and tuned memory to some very insane configuration so that the application would run for a working day."
Here are the basics from the interview:
"What I'd discovered very early on is that the customers were often frustrated. They had tight deadlines, aggressive plans, and intense pressure to deliver -- and no matter what they did, things just didn't seem to be working. To make things worse, they 'knew' the problem wasn't their fault. I let them rant and rave and in the process vent their frustration. What happened is they ended up explaining exactly what was wrong with the system. So it both allowed them to vent and I could parrot back what they told me in the form of a diagnosis. They got to release all of their stress, and I ended up looking brilliant…
Stress prevents us from learning. The first thing I look for in an SOS engagement is a pressure-relief valve, some hack or trick to reduce the level of stress in the room. In one case, I put in a cron job that ran every 15 minutes, looking for any database transaction that had run for more than some threshold period of time. If it found one, it would kill that session. This is an ugly hack. The user whose transaction was killed certainly wasn't happy. But the hack stabilized the system enough so that most of the users who had customers in their faces got work done. It also took pressure off the developers.
Every time the system went south, which was quite often, the phones would start -- and just think of the rat in the cage being buzzed at random times. You could imagine what a relief it was to get the phones to stop ringing. You could see the stress drain out of the room and the brains turn back on. It set up an environment were we could have a meaningful discussion about a permanent fix.
I've used lots of release valves to calm stressed-out developers: I've rolled VMs through clusters, neutered the HTTPSession object, used GC to slow down certain parts of the application to improve overall throughput, tuned memory to some very insane configuration so that the application would run for a working day, and on and on.
This is triage, and my only goal is to keep the patient alive to give developers time to start fixing what is broken."
I'd be curious to know about the experiences of others. What do you do when code goes haywire and it's driving you and everyone around you nuts in high-pressure situations? What hacks and tricks have you tried to create minimal system stability or functionality as a kind of holding action that reduces stress so you can move on to solve the problem?
|
Previous
Index
Next
Threaded
Edit
Reply
Quote
