The Audible Gasp

While it may be absolutely great to be surprised by an audience reaction while performing a theatrical production, being surprised by something happening in production is not great. How can we avoid the “audible gasp” moment in our code?

When I was younger I used to be very active in amateur dramatics. I got a real charge out of performing in front of an audience. Some of my friends would tell you that I’m still a ham but maybe not as willing to show it off to the world as I once was.

One of the first plays I acted in was a play called “Angel Street.” I suspect more people might know this play by it’s original (I think) title “Gaslight.” In case anyone is unfamiliar with the plot of the play (and later movie) it involves a man who is trying to drive his wife insane by making her constantly question her perceptions of reality. The title comes from the fact that at several points during the play the gas lighting in the house (the play is set in Victorian England) will dim and the wife will point it out and the husband will say “What dimming? I don’t see any dimming of the lights?” If you’ve ever heard the term “gaslighting”—this is where it comes from.

So we rehearsed this play and we finally got to perform it in front of a live audience. I was backstage awaiting my cue to enter toward the end of our first performance when I heard an audible gasp from the entire audience. I mean it was an audible “OMG” type of moment. I confess I hadn’t paid close attention to the staging of that portion of the play so I had to ask other folks in the cast what had happened. The scoundrel (played very well by a gent named Dave Eichler) was tied up on-stage while the police were on their way to arrest him. He asked his wife (also played quite well by a young lady named Terry Korn) to fetch his straight razor to cut the bonds that were holding him. Remember he’s been trying to drive his wife mad for the entire play. Terry got the razor from one side of the stage and then held it to his throat for a moment while she delivered some lines. It was when she held the razor to his throat that the audience gasped. They really thought for a moment that this wife who had been so badly abused by her husband was going to cut his throat.

All of that is a neat memory for me but why do I mention this? Because in all of our rehearsing we’d done this same thing several times. No one in the cast thought anything of it—because a.) we knew the script so we knew the evil guy was safe and b.) we’d seen it so often while rehearsing. It wasn’t until a live audience saw this moment that we realized exactly how suspenseful and actually kind of frightening that moment was for the audience.

Avoiding The “Audible Gasp”

When you’re building a production system the last thing you want is that “audible gasp” moment. And it’s tougher for developers to avoid surprises because, like the cast of that play, you get accustomed to certain things and hence they no longer register as possible problems to you.

You don’t want to be surprised by anything that happens in production. Even pleasant surprises (for example, “oh this runs faster than we thought it would”) are really not welcome because that means you’ve done something you didn’t expect to do. Something running faster than you expect may mean that important steps are being missed for some reason.

So how do we as software engineers prevent the audible gasp moment? I’d suggest there are a few ways we can do this:

1. Run our tests in a system that mirrors production as closely as possible. If you’re going to be running your production system against a sharded database then run your tests against a system that uses a sharded database. It may be tougher to set up but it’s worth the extra effort to avoid surprises when you get to production.

2. Run tests that really reflect expected use. If a user is expected to go through several screens in order to enter data it makes no sense for you to directly insert data in order to speed up your testing. In fact I’d go so far as to say that it makes some sense to test alpha and beta versions of your software with users who have no idea of what they should do to help flush out errors caused by usability.

3. The data in test is the same as the data in production! I realize there are good reasons to avoid using production data in test systems—confidentiality, difficulties involved in syncing the environments, etc. etc. But, to the greatest extent possible, you need to have data that reflects actual production data in your test system because some errors will only show up in the presence of certain data.

4. Test with users that have the same privileges as your end-users will have. One source of surprises I’ve seen over the years is when an error occurs because a user doesn’t have sufficient permissions to do something. The developer, who was effectively the root user of the system, never spotted any permissions issue because he or she never tested with a more limited set of permissions.

5. Try to test with actual expected load. This is, admittedly, a tough one to do but it’s worth doing your best to do it. Some issues only show up when several people are doing operations concurrently. Someone forgot to lock a resource? You won’t find it till you’ve got 5 people trying to work with the resource. Someone forgot to unlock a resource when they were done using it? Again, this sort of issue won’t show up until the system is under load. It’s also very much worth trying to do this since these sorts of “only happens under load” errors can be extremely difficult to reproduce and to solve. All of us would much rather find problems ourselves than have users find them for us.

6. When you discover a surprise in production, add tests to prevent it from recurring. Rich Hickey pointed out one time that every production bug is code that’s gotten past all of your prior testing. So rather than just patching things and assuming that the error will never happen again, build tests to assure yourself. Code in production is never completely static and later changes sometimes revert our fixes.

Finally I will say that in most projects I’ve worked on over the years, two things are usually skipped when deadline pressure looms—documentation and testing. Suddenly the very thorough test plan is being compressed. “Well actually that one test will cover three cases. We don’t need to worry about the other two tests.” Try your hardest to stick to thorough testing and, more importantly, to fixing any major issues that come out of that testing.

Also be extremely wary of tests that don’t turn up issues. If you’re not spotting problems with your testing it’s not that your code is of such sterling quality; it’s that the tests aren’t testing what they should. There is no computer software in the history of mankind without bugs; your team isn’t going to be the first to write bug free software. Testing is intended to find the bugs before our customers do so we can mitigate the worst ones before they cause the customers big problems.