Friday, March 19, 2010

Lessons Learned from a "Testapalooza"

I know some of you are scratching your heads at the title, so let me explain. My last post dealt with the issues and realities of being what I call a “Lone Gunfighter” in a company, i.e. the single dedicated tester. Of course, when there is only one person in a company who is dedicated to testing, that leaves a whole lot of potential gap space. I have long ago come to the conclusion that it is impossible to test anything 100%, and that to put the product in the hands of people who are not testers per se, but who have experience with a product and can work with it, they are likely to find things that you have missed. What’s more, they are going to find things that become blindingly obvious once they have told you about them.

Thus, in my company, about twice a year, we hold what we have dubbed “Testapalooza” events. It’s basically a set amount of time (usually two hours) where we set up ten machines, get ten people from different groups in the company, and give them an opportunity to go wild on the application being tested. Our best participants have been our sales team and our customer support group members. Developers can participate, too, but only on projects they are not directly performing development work. I, however, am not allowed to test. I act as Proctor for these test parties, and I answer questions, offer clarification when necessary, but more times than not I utter something similar to “Oh, would you look at that? Yeah, that’s an issue, go ahead and write that up”, while under my breath saying “You have got to be kidding me… I missed that?!”

One could get the sense that Testapalooza’s could be seen as a form of punishment for the tester, an indictment on their skill (or lack thereof) in finding problems. I will admit that the first time we performed this exercise that was exactly how I felt. I saw the bug list created, and it wasn’t trivial. There were a lot of issues that I felt I had tested around and passed that, upon further review and inspection, showed areas I had missed, and often, areas I had totally not considered or even looked at. I thought for sure I was going to be shown the door after this, and I think I stayed up half the night afterwards wondering if I was going to get canned. The good news is that, no, I didn’t get canned, but I learned some valuable lessons from this exercise, and now I’m a fan of it. From fear and dread to acceptance and embrace? How do we get from there to here?

1. Every tester brings a different perspective: I realize this may be trite to say, but this cannot be over emphasized. No two people have the same view of the world, and no two people will approach tasks the same way. What is acceptable to one person will be downright annoying to someone else. One person’s common practices will not be someone else’s. Some people like to enter text a particular way, using the keyboard and every shortcut in the book, while others like to use the mouse and bounce around. Perspective and habits are important, and ones habits can have huge impact on results, even little things that might never rise to the conscious level (ah, but shouldn’t us testers be aware of those mannerisms and habits? Yes, we should, and as a result of these Testapalooza’s I am now more aware of them than I used to be).

2. Inattentional Blindness is real: I owe Cem Kaner for this phrase entering my lexicon. Cem showed a testing video a few years back highlighting this phenomenon. The most famous video of this can be seen that showed people playing a game with a basketball and the people passing the ball around. The video then asked if anyone saw anything out of the ordinary take place. About 50% of the respondents said that they did not see anything out of the ordinary, even when shown on playback that a man with an umbrella had walked into the picture or that a man in a gorilla suit had likewise walked into the picture. I’ve seen these videos, and I’ll confess, the first time I saw them, I missed the abnormal things, too, because I was so focused on counting how many times the ball was being passed. In testing, we have the potential to be thrown off by Inattentional Blindness, because we see things all the time and focus on key areas, but fail to see others, even when they are right in front of us. It’s only after they are pointed out that they are obvious.

3. There’s always a different order to doing a task: if you ask ten different people to do a job, but don’t tell them exactly what to do, it’s a good bet that no two people will do exactly the same things. The differences may be very subtle, but timing, order, duration, choice of terms, etc. will invariably happen. This is why, if you get a chance to get many people looking at a goal or a task, you will be much more likely to uncover issues because of the different methods people use. As a tester, you may look at something and say to yourself “now come on, I did that!” but on closer examination, your original method didn’t uncover the problem, but just a slight variation showed you a different way to do it. Of course, after seeing the slight variation, you then train yourself to check it in that manner, which when it is fixed, you keep it in mind to look at it with that calibration.

I’m happy to report that I have since come to appreciate and enjoy these opportunities, because I decided that it was way better to learn from them and improve our testing practices than it was to get frustrated and defensive about not being "the one" to find those issues. Yeah, I admit it, I can get defensive and frustrated when things that now appear obvious are missed. What’s important, though, is to note that these same issues didn’t just get past me, they also got past our developers and they also got past a number of other people. It took the combined forces of a number of people to confirm that, yes, indeed, there are more issues to be found, and more testing would of course be beneficial. At that point economics of scale come into play. Yes, these Testapalooza events are a great way to kick some adrenaline into testing efforts, and have a great chance of uncovering many additional issues, but having these individuals work on testing all the time like this would be incredibly expensive, since these people would not be doing their primary jobs (which likewise add benefit and revenue to the company). So I do my best to learn from them what I can, review my test plans, test cases, and methods of testing, look at what I can modify to incorporate the new information, and see if the next time, we have done better. It also tell everyone that quality is not just up to the test team, but it’s everyone’s job to make sure that we put the best product out that we can, and its way better for us to find these issues than having our customers find them.

So to all of my testing friends, if you don’t do Testapalooza’s, consider them. You just might be amazed at what your co-workers can help you find :).

No comments: