Consider the Source: A Tester Invades Philmont

One of the more entertaining aspects of our trek would have to be the final two days, in that we were joined by an expedition mate that, let's just say, is a little different than the average hiker:

Testhead readers... meet "T.K."

Actually, his real name is "30", which is the number he wears around his neck for identification purposes,  but part of the "Burro Expedition" experience is that each crew gets a burro to join them for two days, and the crew gets to name the burro for the duration of the trek. Needless to say, it would be impossible to keep track of the names changing all the time, so the burros are officially known by their numbers, but are also conditioned to respond to any name they get called. 

During our trek, the boys talked a lot about the Anime series "Angel Beats" (which also happens to be one of my favorites), and we joked a lot about who our favorite character in the series was. That character was "T.K.", a long haired, always moving dancer who would speak in nonsensical English phrases (even in the original Japanese recording). When we met burro number 30, and he came up to us and looked at us with a circular head bob, we all laughed at the similarity to the movement T.K. did in the anime, and we all said "Hey, let's call him T.K.!" With that, T.K. became an official member of the crew, and would help us on the trek for the last twenty miles or so by carrying some of our gear for us.

As we were making our way to the Baldy Skyline camp area, we had a number of choices we could take to get there. As we scanned the area, we saw that there was a road that looked to be a straight shot up to the area, and pretty short by comparison. We started walking towards the camp and for the first bit of the stretch, T.K was walking alongside us and made no fuss. When we saw a fork in the road, and the trail that continued up to the camp area came into view, TK started walking that way, but we guided him back to the main road. Over the course of the next few minutes, TK walked slower and slower, and finally stopped. We tried to coax him forward, and we'd get a few steps in, and then he'd stop again. We did this about half a dozen times, and each time, it was getting to be progressively harder to move TK further up the road.  

Additionally, as we looked at T.K., we had the impression he was looking at us as if to say "hey, you guys are all idiots! Did you see that trail that forked off the main road a while back? That's where we're supposed to go. Trust me, I've done this trek more than a few times, I know how to get there!" Still, we felt confident that our short cut would work and save us time, so we persevered. I've had experience leading horses and burros in the past, so I took it upon myself to hold his bridle and lead rope and coax him up the road, and ultimately, he let me lead him up the road.

The road we had chosen changed. It became steep. It was straight, sure, but it was also a laborious climb to get up to the ridge line where the camp was. It was tiring, it was in direct sunlight, and by the time we reached the top of the road and the ridge line, we realized that, while we were at the top of the hill, the actual camp area we needed to be at was still about two kilometers east. As we trudged to the camp area, we saw the trail converse with the upper road, and we also saw that the trail was coming out of a wooded area, with plenty of shade, the trail meandering through a range of switchbacks that, while longer in total distance, would have been much kinder to our knees and backs. As I looked at T.K., it felt as though he was saying "See, I told you you were an idiot! I could have led you here and saved you a lot of hassle, but no, you had to go your own way. How did that work out for you?"

Does this sound familiar? I know it does to me. I've had this experience with my teams. We are so sure that what we are going to do will save us time, it will be different than all the previous times we've tried to meet a goal, but someone, usually one with a lot of experience, puts up resistance and tries to point out what we should be doing. Often, that advice goes unheeded, only to come back and bite us later when we realize that the insight was valid, accurate, and based in experience, but our hopes and optimism for a faster way (or a looming deadline) blinds us to the hard earned experience. 

I chuckled as I thought about this the following day, when we left the Baldy Skyline camp to head down to the next area and followed the main trail. T.K. just started going, and he barely stopped. He knew where he was, and he knew where we needed to go. Sure, we consulted our maps to be sure, but otherwise, the trail was the most direct route, and he just plowed ahead until we chose to call a halt for a rest break. During the remainder of the trek, as we were debating ways we should go, I'd quietly nudge T.K. to move forward, and I'd see where he wanted to go. Typically, I'd encourage following his lead, and each time, it would prove to be the most direct and easy to navigate route. Sure, it would have been more adventurous to bushwhack, but I'm sure T.K. appreciated not having to climb over or around felled trees or other obstacles.

There are times when new vistas are open to you, and there is no trail to follow, and one way is as good as any other. At times, though, there are those who have gone before us, and instinctively know when we are going the wrong way. It's wise to step back and listen to those who express reservations. Who know how much sweat and toil they might save us if we listen to them ;).

Come See TESTHEAD Speak at STP-CON 2016 in Millbrae, California

It's just a few weeks away. Software Test Professionals is bringing its conference to my home town. Well, it's bringing it to the town just south of my home town, but that's close enough. If you will be attending STP-CON 2016 at the Westin in Millbrae, CAlifornia, and you are looking for a couple of sessions to attend, there's lots of choices, pretty much all of them great. However, if you are still undecided, I'd like to make a couple of recommendations... specifically, come to my sessions ;).

First of all, I will be giving a half day workshop/tutorial based around "Teaching New Software Testers". I'm sure some of you looked at that title and said "ah well, I'm not a new software tester, so that's not relevant to me." However, this workshop is not geared towards new testers, it's actually geared towards those who will mentor new software testers. My guess is that's a significantly larger audience. The workshop is built around the materials that were prepared with several contributors, along with myself, for the SummerQAmp program.

We put together six modules (What is the Scientific Method?, What is Testing?, What is Context?, What is Bias? What is the SDLC/STLC?, What is Bug Reporting? How Does the Web Work?). These modules have been field tested, we've received a good deal of feedback, and some cool experiments have grown out of the work with these modules and test mentors who have used them to train new testers, some with zero experience in testing whatsoever. This session will be light on talk, and will focus much more on exercises and action. In short, each participant will get a chance to mentor and be mentored. Each participant will get the stack of materials that made up the SummerQAmp materials (developed with the help of the Association For Software Testing and creative commons licensed). In short, if you want to possibly pick up some new approaches to talking about testing with new testers, this may be up your alley. Come join us April 4, 2016, 9:00a.m. - 12:00p.m. in the Poplar Room.

My second talk (this is just a talk, so less of a time commitment) covers The Intersection of Accessibility and Inclusive Design. Wait, what?!

Accessibility focuses on designing products that allow those who otherwise could not use them to gain access to information and services. It makes this possible through assistive technology (screen readers, close captioning, voice dictation, joystick controllers, etc.). This is important work, but I want to also emphasize there's a lot we can do to make software applications even more usable by as many people as possible. This is the core idea behind Inclusive Design. We will give examples of both accessibility, and show apps that have taken advantage of, whether by design or by happy accident, principles to make software that is usable by the most people possible. Come join us April 6, 2016, from 2:30p.m. - 3:30p.m. in the Hickory/Hawthorn room.

Just a couple weeks to go. Can't wait to see you all in my neck of the woods!

Spreading Some E-Book Holiday Cheer: #Packt5Dollar Deal

For the life of this blog, I have deliberately avoided any direct advertisement, or made any attempt to "monetize" it. I plan to keep it that way, since not accepting advertising allows me to say exactly what I want to say the way I want to say it.

Having said that, there's no question that several book publishers have been more than kind, giving me several dozen titles for free over the years as review copies. More than half my current technical library consists of these books, and many have been great helps over the years. If I can return the favor, even just a little bit, I am happy to do so.

Packt Publishing, located in the United Kingdom, is one of those publishers, and they are currently holding a special $5 E-book sale. Any and all e-book titles from Packt are available for $5 for a limited time (until January 6, 2015).

Again, Packt has been very generous to me over the years, and I appreciate the fact that they are offering this opportunity. I've already take advantage of it... and have added even more books to my Tsundoku list (LOL!). For those curious, I decided to purchase the following:

1. Backbone.js Cookbook
2. Java Script Security
3. JMeter Cookbook
4. Kali Linux Network Scanning Cookbook
5. Learning JavaScript Data Structures and Algorithms
6. Learning Python Testing
7. Responsive Web Design By Example
8. Selenium Design Patterns and Best Practices
9. Selenium webDriver Practical Guide
10. Web Development with Django Cookbook
11. Wireshark Essentials

Happy reading :)!!!

Teaching by Example: Slowing Down to Get Ahead

It's been an interesting month with my daughter, Amber. We have been engaged in an interesting little "tug of war" over the past several weeks. I am realizing that I have to be very careful here, because enthusiasm on the part of the "teacher" can either inspire a student, or completely demoralize them and turn them off.

We've agreed to a simple goal, and for now, that goal is the Codecademy streak. I've told her I don't care if she only does one lesson on a given day, as long as she at least does that one lesson. Of course, I'd be really happy if she did two, or three, or ten, or heck, even do fifty like I did about seven months ago. Yes, I spent a free Saturday and completely pounded through the entire jQuery course. No, I'm not convinced that was the best use of my time, considering how much I still have to look up to write anything today.

What I realized from that particular experience was that my own enthusiasm to "push through" caused me to take shortcuts. I violated what I jokingly refer to as "The First Zed Commandment" which is "thou shalt not copy and paste". Those who have read any of Zed Shaw's books on "Learn [Language] The Hard Way" know that he espouses this pretty heavily, and I do too... most of the time. However, my own impatience often gets the better of me, and yes, I find myself cheating and copying and pasting. If I were looking to get in physical shape for a run, would I get the same benefits if I agreed to run 5K each day, but when I got impatient, I hopped on my bike and rode the rest of the way? Would I get the same training benefit? Would I get the same physical conditioning? The answer is, of course, no. Writing code is the same way.

At this stage, I may wish my daughter were moving faster, but if the net result is that she scores a lot of points, gets a lot of badges, but doesn't remember fundamental syntax or hasn't put the time in to recognize where she has made a mistake, what am I really teaching her? This has prompted me to, instead, ask her what time she wants to sit down with me, her with her computer, me with mine, and we work either face to face or side by side. This way, she sees what I am doing, and how I'm doing it. It may encourage her to do likewise, or she may say "hey, that looks odd, what are you doing?" Either way, it will start a conversation, and then we can discuss the fundamental details. I have to realize that it's better for her to pull from me rather than me push to her.

So fundamentally simple, so easy to say, yet so hard to do when "Dad Brain" wants to carry her along as fast as he can. On the positive side, this is also allowing me to step back and make sure I really understand what I think I understand, with the often neat realization that, hey, I really didn't know that as well as I thought I did. Consider this my solid recommendation to anyone wanting to learn how to code... teach your kid how to code. If you don't have one, see if you can volunteer at a school's computer club and offer your time. What I'm realizing anew is that the example of working through problems for them is a pretty amazing teacher in its own right.

Guest Post: Inside an "Hour of Code" with Amber Larsen

This blog entry comes courtesy of my daughter. On Saturday, December 13, 2014 in San Bruno, she and a number of kids from her intermediate and the local elementary schools participated in "Hour of Code". Amber signed up for the "Intro to Java" class. The materials being used for this class can be seen at the Khan Academy, and is an "Intro to Drawing" using Java syntax.

I'll let Amber take it from here:

I would say that the web site we used was very "child friendly". It helped to make it possible for children and teenagers to work on code. It had videos, so instead of reading it, you could see it happen in front of you. We also worked on projects where we were able to make shapes and add colors and work with a palette. It was a good introduction and it was easily doable in an hour.

The course was listed as an "Intro to Java", and while I learned how to make shapes and fill in colors, and yes, I know that I was using Java to make those changes, it felt like I was working with a very small area of the code. I don't feel like I learned how to make websites or an actual program, but then I don't think that was the point. They had a full programming tutorial for Java that we can continue with at the end of the hour.

I think some of the explanations needed to be listened to a couple of times. Some of the other kids I was working with got stuck, but we were able to talk together and straighten it out. It reminded me of the HTML and CSS modules I have been working with in Codecademy.

Speaking of Codecademy, I think my having spent the last month working through the projects there helped me a lot, maybe too much. I finished the set of videos and projects 25 minutes before everyone else.

One of the nice things about what the Intro to Drawing course had was that the videos could run and we could change the code as it was put on the screen. Also, there wasn't a big setup time. When I downloaded Python I had to install it, set up my IDE to work with it, make sure that the compiler worked, that my PATH variable knew about Python, and that the IDE could use the compiler. If we all had to do that, we would have easily spent an hour just getting all that set up.

If I had to say there was anything I didn't like, it's that right away it told me if I made a simple mistake (well, sure, but I'm not finished yet, hang on!). Maybe it's because I'm used to the Codecademy approach, where you fill in what you want to write, and then submit the whole thing, and if there's an error, the screen shows it and it makes a suggestion, and you have to figure out what you did wrong. In a way, that felt more like "testing". With this, it came right out and told you what you were doing wrong. I think I might have learned more without the frequent reminders, but it was an intro, so I understand.

I think that instead of calling it an Intro to Java, it should be called an Intro to Drawing (using Java) because we focused more on the drawing (making lines, making rectangles and circles, filling them in) than we did on the Java. Having said all that, I think it makes sense to do what they did, because they want to make it interesting for kids to want to learn more, and with that, I think they did a pretty good job.

Teaching My Daughter to Code and Test: Beginnings

Earlier this month, I brought my 13 year old daughter Amber home from a presentation at Google/YouTube called "Made With Code". Amber came out of the presentation energized, excited, and saying how cool it was what they showed her.


This is the first time that Amber has been "excited" about how computers work outside of a "users" perspective and thinking about computers and computing from a "makers" perspective. As we talked, I was thinking about some freelance coding that I do, and how it might be fun to have her learn a bit about web development, do some stuff and "push it to production", and pay her a bit for her efforts.


It also seems that there might be some interesting "discoveries" and reactions from two perspectives, mine and hers, that might make for a cool blog series. With that, we both decided to work towards building some practice times into our days, and see if the concepts that I have been learning over the years are easily teachable, or if I might learn more from her interactions than she does from me.


Additionally, I figured it would be more interesting to see the experiments and the realizations we come to in somewhat real time rather than wait several months to do a more formal synopsis, so that's also what we will be doing over the coming weeks and months (for those who wondered if my Tsundoku post owed in part to this initiative, the answer is "definitely yes" :) ).


So you will start to see some "shared posts" in this space. If it's my perspective, it will be in a standard layout and font color. If you see highlighted green text, that's Amber, speaking in her own words. Over time, she may choose to make full blog posts here, and those will reflect that in the title, but for now, just know when you see green highlighted text, that is her.


One of the things we decided to start with was to get her focused on something simple, where it would be easy to see and make changes. To that end, we thought it would make sense to have her practice with Codecademy to learn basic details of web formatting and style. Early on, we both decided that a little each day would be a better approach than trying to do a whole bunch at one time.

For the past three weeks I have been working on learning the basics of HTML and CSS in Codecademy. I have found that it is easier to do a little of it every day and keep the streak up than do it all at once in one big shot. I have learned the very, very basics.


One of the things that has been fun, if not a little annoying, has been to have my Dad sitting next to me and helping me with some of the assignments and examples. I say fun because I like the fact that he can help me understand what is happening. I say annoying because he's my Dad. What I mean by that is that sometimes he's a little to quick to tell me when I am doing something wrong without me learning myself. We finally made an agreement that I would work on my own computer and that I would call him over only when I felt stuck or confused. While I appreciate his input, I told him I was not going to learn anything with him always hovering over me and telling me what to do.


Other than that, I am happy to say that, since joining Codecademy on November 10th, I have done a little bit each day, and I have a 19 day streak as of now. My Dad checked up on me every day that he was away in Ireland to make sure I kept my streak alive, and I was happy to say I did.

It has been interesting to see the ways that Amber interacts with me as we work together on the Codeademy projects. Speaking of which, we have a session scheduled for a little later today (and that will extend her streak to twenty days ;) ). I am really curious to see where this journey will lead us both, and what we both learn from the experience.

Silence Can Be Powerful

This past Tuesday evening, the Boy Scout troop that I am Scoutmaster for (Troop 250 in San Bruno), held its big annual Homecoming Court of Honor. This is typically a big affair, in that it encompasses our Scout Camp week and all the awards earned while there (which is to say, a lot of them).

One of the things I have been trying to do with the boys in my troop is encourage them to take on challenges and take chances. Ultimately, a well run Troop is run by the boys, not by the adult leaders. Still, it's very common for them to ask me a lot of questions about what they should do or shouldn't do, and normally, I'm ready and willing to provide them answers.

This time, though, I decided to do something unprecedented, at least as far as a Court of Honor was concerned. The scouts are familiar with what we call a "silent" campout. At a silent campout, the adult leaders camp in a site adjacent to the one the scouts are camping in, and we follow all rules of safety and emergency preparedness, but other than that, we stay in our camp site, and they stay in theirs. They set up, cook, clean, make and break down fires, and anything else they need, all without any input from the adult leaders. The goal is to have them learn from their own mistakes, and to have them work with each other to solve their problems rather than have the adult leaders do it for them.

I decided to take this one step further and declared that our Court of Honor would be a "silent" Court of Honor. In other words, the scouts would run it, they would do all of the specific details (give out awards, recognize rank advancements, etc.) and I and the other adult leaders would sit back and watch. We would not speak, we would not direct, we would not answer questions.

So how did it work out? Splendidly!

Granted, there were several times where I had to sit on my hands and keep my mouth shut when I so wanted to say "no, not that way, do it like this!" that, however, was not the point. It wasn't my Court of Honor, it was their Court of Honor. If they neglected to bring something out, put something on display, or do something that they had seen me do dozens of times, that didn't matter. What I wanted to see from them was what they felt was important. I wanted to se what aspects of a Court of Honor they wanted to do. They jumped at the chance to do a skit. they liked doing silly one liners. They enjoyed the awards part, and they left me a little time at the end to speak my peace. Which I did, but only at the end.

Many times, I think we do a disservice to those who are learning and trying to figure out what's important and what isn't. Coyote Teaching focuses on leveraging the environment and addressing real needs, as well as focusing on the art of questioning, or asking more questions rather than giving direct answers. I'd like to add to that the very real teaching tool of "be quiet". Sometimes it's best to not answer, or to remove ourselves entirely. Sure, there may be stumbling, there may be things said that are not perfect, or there may be some key stuff that gets forgotten, but that's OK.

What's important is to give the people you are working with a chance to discover what is important to them, and let them reach that conclusion themselves. It would have been very efficient to correct them and tell them what to do, but it would be far less effective than giving them the chance to run with the program all on their own. They've participated in several Courts of Honor over the years that I have run, and regardless of how flawlessly I may have done them in the past, none of them will be as memorable, or mean as much, as this one will. The reason? BY giving them silence, they got to experience and do for themselves what they wanted to do, and honor the troop in the way that actually mattered to them.

I'm super proud of all of them... and yes, I took notes ;).

Going "Coyote": Overcoming Fear and Uncertainty with "The Craddick Effect"

For those who have been following my comments about mine and Harrison Lovell’s CAST 2014 talk ("Coyote Teaching: A new take on the art of mentorship") this fits very nicely into the ideas we will be discussing. We’ve been looking back at interactions that we have had over the years where mentorship played an important role in skill development. During one of our late night Skype calls, we were talking about skateboard and snowboard skills, and how we were able to get from one skill level to another.

One aspect that we both agreed was a common challenge was “the fear factor” that we all face. In a broader sense, we both appreciate that snowboarding and skateboarding are inherently dangerous. Push the envelope on either and the risk of injury, and even death, is definitely possible. Human beings tend to work very hard at at unconscious level to keep ourselves alive. The amygdala is the most ancient part of our brain development. It deals with emotions, and it also deals with fear and aggression. It’s our “fight or flight” instinct. One one level, it’s perfectly rational to listen to it in many circumstances, but if we want to develop a technical skill like jumping or riding at speed, we have to overcome it.

About fifteen years ago, I first met Sean Craddick, a fellow snowboarder who was my age and was, to put it simply, amazingly talented. I used to joke whenever I saw Sean at a competition that I would say “oh well, there goes my shot at a Gold Medal!” He humored me the first couple of times, but the third time I said it, he surprised me. He answered “Dude, don’t say that. Don’t ever say that! I could try to throw some trick and land it badly, and scrub my entire run. I could miss my groove entirely, or miss a gate on a turn, or I could catch an edge and bomb the whole thing. Every event is up in the air, and every event has the potential of having an outcome we’d least expect. Don’t say you don’t have a chance, you always have a chance, but you’ll never get the chance if you don’t believe you have it.”

Because we were both the same age and had fairly similar life experiences, I’d hang out with Sean at many of these events, and sometimes run into him on off days when I was just up at the mountain practicing. One time, he noticed that I kept going by a tall rail and at the last moment, I’d veer off or turn and ride past it. After a few times of seeing this, when he saw me about to veer off again he yelled “Hey Michael! The next time you veer of, stop dead in your tracks, unbuckle your board and walk back up here. I want to talk to you about something.” Sure enough, I went down, veered off course, and slammed to a stop. I took off my board, and then I walked up the hill. Sean looked at me and said:

“Take it straight on, and line your nose with the lip and where the beginning of the rail is.”

I nodded, buckled in, and then went down to the rail transition. I veered off. I stopped. I walked back up the hill.

“Lean  back on your rear heel just a bit. It will give you a more comfortable balance when you first get on the rail.”

I nodded, bucked back in, went down again, and again I veered off. I stopped, unbuckled, and walked back to the top of the hill again. By this point I was winded, my calves were aching, my heart was pounding, and I was getting rather frustrated.

“One final thing. Do an ollie at the end of the rail.”

What? I hadn’t even gotten on the rail, why is he telling me what to do when I get off of it? I shrugged, buckled in, went for the hit, and this time, I went straight, I lined the nose up, I set my weight back just a little bit, I slid down the rail, and I did a passably adequate ollie off the end of the rail, and landed the trick. When I did,  Sean whooped and hollered, then came down after me and hit the same rail.

“Awesome, lets go hi the chairlift!”

As we did, Sean looked at me and said:

 “You can understand all the mechanics in the world, but if your brain tells you 'you can’t do it, it’s too dangerous, it’s too risky', you need to get your body to shut your brain up! That’s what I had you do. I knew why you were sketching the last few feet. You were afraid. It felt beyond you. You might crash. It might hurt real bad if you do. The brain understands all that. It wants to keep you safe. Safe, however, doesn’t help you get better. Whenever I find myself giving in to the fear, I stop what I’m doing, right there, and I walk up the hill, and I try it again. If I pull back again, I walk the hill again, and again, and again. What happens is the body gets so fatigued that every fiber of your being starts screaming to your brain ‘just shut up already and let me do this!' Exertion and exhaustion can often help you overcome any fear, and then you can put your mechanics to good use.”

Yeah, I paraphrased a lot of that, but that’s the gist of what Sean was trying to get across to me. Our biggest enemy is not that we can’t do something, but that we are afraid that we can’t do something. That fear is powerful, it’s ancient, and it can be paralyzing. That ultra primitive brain can’t be reasoned with very well, unless we give it another pain to focus on. At some point the physical pain of exertion and exhaustion will out shout the feelings of fear, and then we can do what we need to do.

In a nutshell, that’s “The Craddick Effect”. There may be a much fancier name for it, but that’s how I’ve always approached mentorship where I have to overcome fear and doubt in a person. When some one is afraid, it’s easy to retreat. As a mentor, we have to recognize when that fear is present, and somehow work with it.

You may not do something as extreme as what Sean did with me, but you may well find other, more subtle ways to accomplish the same thing. Imagine having to take on a new testing tool where there’s a lot that needs to be learned up front. We could just let them go on their own and let them poke around. We can take their word that they are getting and understanding what they need to, or we can prod and test them to see what’s really happening. If we see that  they don’t understand enough, or maybe even very little, don’t assume lack of aptitude or drive, look for fear. If you can spot fear, try to coax them in a way that they can put their energy somewhere else for a time so that they can get to a point to shout down the fear. It may be having them do a variety of simpler tasks, still fruitful, but somewhat repetitive and tedious. After awhile, they will get a bit irritated, and then give them a slight push to move farther forward. Repeat as necessary. Over time, you may well see that they have slid past the pain and frustration point, and they just “get” what they are working with. It just clicks.

As a mentor, look to help foster that interaction. As a person receiving mentoring, know that this may very well be exactly what your mentor is trying to do. Allow yourself to go with it. In the end, both of you may learn a lot more about yourselves and your potential than you thought possible. It’s pretty cool when that happens ;).

Try Always to See What Happens When You Violate the Rules: 99 Ways Workshop: Epilogue

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific. My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #100: Try always what happens when you violate the rules. So here's the second sentence also. - Teemu Vesala

The original eBook ended with a number 100. Why? Just because. It was meant to be a little poke in the ribs to say "ha, see, we put in 100, just because we wanted to". In other words, expect the unexpected.

Epilogue: Where I've been, and what I've learned

This has been a wild ride, to tell the truth. It was started a bit anxiously, then felt like I got into a bit of a groove, then I decided to go for it and put out two a day where I could. For some of the entries, it was easy to come up with examples. For some others, it felt like pulling teeth to make examples that were coherent and understandable. Still, I found when I was pushing out two posts a day, I felt confident I could push out two posts a day. Then CAST (the Conference for the Association for Software Testing) came, and I had to travel, go and participate in the conference, and I decided to put down this project for a week to focus on other things. I figured it wouldn't be too hard to pick it back up again. Wow, was I wrong.

As I've said in other items I've written, the habit becomes the driver. When we create a habit and put it into practice, it become easier to meet the habit. When we put down the process, even if it's just for a week, it's so much harder to pick it back up again.

September and October also coincided with a increase of writing and focusing on initiatives preparing for talks I will be delivering at Oredev in November. I found myself fighting with, and trying to make time, to finish this project. One stretch of working through the Excel example, and condensing it down so that I could put it into a blog post in a way that was coherent, took almost two weeks for me to get together.

One of the funny things I noticed as I was writing these examples: I would find myself talking to people in other contexts, and as I was talking with them, I had to stop and think "wait, is that something I wrote about?" If the answer was yes, I would go back and see if I agreed with what I wrote originally, or if I would want to modify what I had said. If I was discussing something that wasn't in the examples thus far, I would make notes and say "hey, that conversation would be great as a topic for number 78. Don't forget it!" 

Mostly, it feels really good to know that I made it from start to finish. Seeing re-tweets and favorites on Twitter, plus likes on Facebook and the comments to the blog posts themselves, shows me that a lot of you enjoyed following along with this. More to the point, this project has doubled my daily blog traffic. Now, of course, I feel a little concerned… will all those readers drop away now that this project is finished? Will they stick around to see what I have planned next? What do I have planned next?

I do have a new project, but it's going to take me longer to do it. Noah Sussman has posted on his blog what he feels would make a great "Table of Contents" for a potential book. The working title is "How to Become a More Technical Tester". I became intrigued, and said I would be happy to be a "case study" for that Table of Contents, and would he be OK with me working through the examples and reporting on them? He said that would be awesome, and thus, I feel it necessary to dive in and do that next.

So, did I just throw out a "bold boast" immediately after completing another one?! Didn't I learn anything from this experience? The answer is yes. What I learned the most is that creativity strikes and skill grows when we actively work them. Without that up-front work it stagnates, and becomes harder to draw out. Therefore, I would rather "keep busy" and make more "bold boasts" so that I can keep that energy flowing. This will, however, be a more involved process. I am not going to make any promises as to how much I can update or how frequently. This may take a few months, it may take a year. It's hard to say on the surface. I do know that I want to give appropriate attention to it and do the content justice. Who knows, maybe Noah will be willing to consider me a collaborator for this book… but I'm getting ahead of myself again ;).

My overall goal for this project was to do more than I figured I ever would if I just read the list and said "hey, those are cool". Here's hoping that my example will encourage you to try to likewise reach in side and fins something you can use. Your outcome may be remarkably similar to mine, or entirely different. If you do decide to take any of them on, please blog about them (and please, put a comment in my blog so I can see what you have written).

Now, however, it's time to close this project, at least for the time being. Time will tell if we've seen the last of me on this (hint: probably not ;) ).

Question the Veracity of 1-98, and Their Validity in Your Context: 99 Ways Workshop #99

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #99: Question the veracity of 1-98, and their validity in your context - Kinofrost

Heh, I think it's somewhat appropriate that we close out this project (and land on #99) with a need to talk about context directly. Yes, I admit it, I consider myself an adherent and a practitioner who believes in and tries to follow the context driven principles (below in the workshop, btw ;) ). Too often we talk about context driven testing as though "it depends" solves all the problems. I'm going to do my best to not do that here. Instead, I want to give you some reasons why being aware of the context can better inform your testing than not being aware or following a map to the letter.

Workshop #99: Take some time to apply the values and principles of Context-Driven testing, and call on them when it comes to determining of anything from these past 98 suggestions actually make sense to be used on what you are working on right now.

First, let's start with what are considered to be the guiding principles of context-driven testing (these are from context-driven-testing.com).

- The value of any practice depends on its context.

- There are good practices in context, but there are no best practices.

- People, working together, are the most important part of any project’s context.

- Projects unfold over time in ways that are often not predictable.

- The product is a solution. If the problem isn’t solved, the product doesn’t work.

- Good software testing is a challenging intellectual process.

- Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

The most likely comparison you will hear is something on the polar oposite ends of the spectrum, i.e. the differences between testing a medical device like a pacemaker or the control software for the space shuttle vs. a video game app for an iPhone. On the surface, this should feel obvious. The scope of a project like a medical device and the repercussions of failure are huge. The outcome is literally life or death for some people. When it comes to a video app or an inexpensive game, it hardly warrants comparison. 

With that in mind, let's try something a little more direct in comparison. What level of testing should go into the actual control software for a pacemaker vs. a monitoring application that resides on a computer for a pacemaker? The pat answer doesn't work as well any longer, but there is still a question here that's not trivial. Are there differences in testing? The answer is yes. The pacemaker controller itself would still be going through much greater levels of scrutiny that the monitoring software would. In the event of system failure, the monitoring system can be rebooted or the program turned on or off, with no effect at all on the pacemaker itself. If the monitoring software did cause the pacemaker to malfunction, that would not only be seen as catastrophic, it would also be seen as intrusive (and inappropriate). 

This opens up different vistas, and again, begs the question "how do we test each of the systems?". The first aspect is that the pacemaker is a very limited device. It has a very specific and essential functions. There's less to potentially go wrong, but its core responsibility has to be tested. In this case, the product absolutely has to work, or has to work at a astonishingly high level of comfort for those who will be using them. For them, this is not a math theorem, this is the power of life or death to them. The monitoring software is just that. It monitors the actions of the pacemaker, and while that's still important, it's of a far secondary level of importance compared to the actual device. 

  

This brings us back to our past 99 examples. advice I've given may work fine for your project(s), but in some cases, it may not be wise to use the approaches I gave to you. That is to be expected. I can't pretend I will know every variable you will need to deal with, and for you to say "well, that may be fine for your project, but my manager expects us to do…" and yes, there you go, that's exactly why we tend to not spell things out in black and white when it comes to context driven testing commentary. We need to look at our project first, our stakeholders next, and the needs of the project after that. IF we are planning our testing strategy without first taking those three things into account, we are missing the whole point of context-driven testing. 

Bottom Line:

In this last statement here, I' going to be borrowing from my own website's "What it's all about" section. In it, I share a quote from Seth Godin that says "Please stop waiting for a map. We reward those who draw maps, not those who follow them." In this final post, I want to make sure that that is the takeaway that this whole project gives. It would be so easy to just look at the 88 Things, assign the ideas to our work, and be done with it. I've strived to put my own world view, my own context, and write my own map in these posts. I may have succeeded, I may not have, but if there's any one thing I want to ask of anyone who has followed this is to not follow any of these ideas too closely. 

If these workshop ideas feel uncomfortable for what you are doing, don't get frustrated. Instead, focus on why they feel uncomfortable. What is different in your case? What could be modified? What approaches should be dropped altogether? It's entirely possible that there are better ways to do any and all of these suggestions than what I have spelled out here. I encourage you to find out for yourselves. I've drawn a map, but it may not be the best map for you. If it's not, please, sit down and draw your own map. the testing world is waiting to see where you will take it.

Test What Matters: 99 Ways Workshop #98

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #98: Test what matters - Rosie Sherry

Ahhh, it all comes down to that, doesn't it? It's simple, elegant, really easy to understand, and yet, try as we might, it's so very hard to actually do (no, seriously, it is). See, what matters is tremendously subjective. Who are we talking about? Are we talking about our end users? Our management team? Our co-workers? Our shareholders? 

We'd love to believe that each and every one of those groups are aligned in purpose and intention, that they would all want the same things, and that what maters to one matters to all. Sadly, that's not true. Thus, to make sense of what matters, we first have to make a solid determination as to "who" matters.

Workshop #98: Get a feel for the five biggest customers that your organization wants to keep happy. Hint: they may not be end users of your product. Once you find them, get to the heart of the matter and discover what really matters to them (ask informally if you can't get direct answers from the one's who call the shots). Then structure your testing regimen to focus on what matters to those people (hint: they will not always be aligned).

A famous phrase that many of us have heard over the years is "Quality is value to someone who matters" (thank you Jerry Weinberg). Therefore, what matters is what we can identify that is important to the person or people that matter. Those people can shift, and they can have significantly different goals. Therefore, what should we do? Do we skip around from person to person and find out what matters most to them, and make sure that we deliver it to them? We could, but I would also hazard that it would make us look schizophrenic, and quite possibly untrustworthy. Quality is, indeed, "value to someone who matters" and the "who matters" part can be hard to pick out at times.

Therefore, rather than a fragmented and manic rush to figure out what is most valuable to any one person at any given time, I'd much prefer to go at it from another route, which is to provide information that will help those people that matter make the best decisions they can. I'm not there to "check off the list that makes my manager happy" or "work on the story that makes the director of development look good" or " deliver under cost or ahead of schedule so that we can maximize sales ahead of the upcoming holiday season". Those things are all valuable, and they are all, in their sphere, important. If I choose to chase any one of those, I will be doing a disservice to everyone else who matters. So what should we do?

It comes down to what I feel is the fundamental thing that testers do, and it's not find bugs, or prove that software is "fit for use". Instead, it's to provide information about the state of the product in ways that are meaningful, and to let those in other positions in the organization make the best decisions that they can based on what we have aggregated, analyzed and synthesized. In the end, the development team really doesn't care how many test cases I ran if I didn't find the issue that is most embarrassing to them. The CEO doesn't care that I was meticulous and covered multiple testing scenarios if, when they stand up and give the demo to customers, the program crashes. The customer doesn't care how many features were delivered if the one that they actually care about still doesn't work. 

We need to be more focused than that, and we need to contribute to more than just working in our predefined box and testing what we are told to test. If we are information providers, than we need to be bold and brave enough to provide information. Even when it isn't convenient. Even when it may embarrass some people. Even if it may mean we have to announce a delay. If we try to please all entities, we will end up pleasing none of them. If we are honest, and show integrity, we may still not make a whole lot of people happy, but we will do one thing for certain… we will be providing the key information for all parties to make the best decision possible. If we truly believe that is what matters, then that is what we need to deliver. That information, the kind that helps make an informed decision.

  

Bottom Line:

So much of what we do is laced with politics, cronyism, and what I often refer to as a "perverse reward system" that tends to honor the short term benefits over long term health. If we focus too much on the short term goals, we can win many battles, but ultimately lose the war. We can paint ourselves into a corner, and have no way to get out without causing a mess. Pick whatever metaphor you want to, but realize that what will please one person may royally irritate someone else. Quality works the same way, and playing sides will ultimately win you few friends. Instead, pledge to make the story, the whole story, the most important thing that you can deliver. By doing so, you can make sure that you are delivering something of real value, and value that will last. Ultimately, that is what really matters, so go forth and do likewise :).

You Won’t Catch All the Bugs, and Not all the Bugs You Raise Will Get Fixed: 99 Ways Workshop #96 & 97

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #96: Be prepared, you won’t catch all the bugs, but keep trying - Mauri Edo

Suggestion #97: Be prepared, all the bugs you raise won’t get fixed - Rosie Sherry

This is really two sides of the same coin, so it pays to focus on them together. In the world of testing, nothing rolls downhill faster than "blame". If everything goes great, the programmers are brilliant. If there are problems in the field, then the testers are marched in and demands made to know why we didn't find "that bug". Sound familiar? I'm willing to bet it does, and if it doesn't sound familiar, then count yourself very lucky. 

This comes down to the fact that, often, an unrealistic expectation is made of software testers. We are seen as the superhuman element that will magically make everything better because we will stop all problems from getting out. Show of hands, has that ever happened for anyone reading this? (crickets, crickets) … yeah, that's what I thought.

No, this isn't going to be about being a better shield, or making a better strategy. Yes, this is going to be about advocacy, but maybe not in the way that we've discussed previously. In short, it's time for a different discussion with your entire organization around what "quality" actually is and who is responsible for it.

Workshop #96 & #97: Focus on ways to get the organization to discuss where quality happens and where it doesn't. Try to encourage an escape from the last minute tester heroics, and instead focus on a culture where quality is an attribute endorsed and focused on from day one. Get used to the idea that the bug that makes its way out to the public is equally the fault of the programmer(s) who put it there, as it is the testers(s) who didn't find it. Focus on maximizing the focus of quality around the areas that matter the most to the organization and the customers. Lobby to be the voice of that customer if its not coming through loud and clear already.

Put simply, even if we were to be able to catch every single bug that could be found, there would not be enough time in the days we had to fix every single one of them (and I promise, the number of possible bugs is way higher than even a rough estimate could give). The fact of the matter is, bugs are subjective. Critical crash bugs are easy to hit home. Hopefully, those are very few and far between if the programmers are using appropriate steps to write tests for their code, use build servers that take advantage of Continuous Integration, and practice proper versioning. 

There are a lot of ways that a team can take steps to make for better and more sable code very early in the development process. Contrary to popular belief, this will not negate the need for testers, but it will help to make sure that testers focus on issues that are more interesting than install errors or items that should be caught in classic smoke tests.

Automation helps a lot with repetitive tasks, or with areas that require configuration and setup, but remember, automated tests are mostly checks to make sure a state has been achieved, they are less likely to help determine if something in the system is "right" or "wrong". Automated tasks don't make judgment calls. They look at quantifiable aspects and based on values, determine if a should happen, or if something else should. That's it. Real human beings have to make decisions based on the outcomes, so don't think that a lot of automated "testing will make you less necessary. They will just take care of the areas that a machine can sort through. Things that require greater cognitive ability will not be handled by computers. That's a blessing, and a curse.

Many issues are going to be state specific; running automated tests may or may not trigger errors to surface, or at least, they may not do so in a way that will make sense. Randomizing tests and keeping them atomic can help with the ability to run tests in a random order, but that doesn't mean that the state that will be met when the 7,543rd configuration of that value on a system is met, or when the 324th concurrent connection is made, or when the access logs record over 1 million unique hits in a 12 hour period. The point here is, you will not find everything, and you will not think up every potential scenario. You just won't! To believe you can is foolish, and to believe anyone else can is wishful thinking on steroids. 

Instead, let's have a different discussion.

- What are ways that we can identify testing processes that can be done early as possible? 

- Can we test the requirements?

- Can we create tests for the initial code that is developed (yes, I am a fan of TDD, ATDD and BDD processes)?

- Can we determine quickly if we have introduced an instability (CI servers like Jenkins do a pretty good job of this, I must say)?

- Can we create environments that will help us parallelize our tests so we know more quickly of we have created an instability (oh, cloud virtualization, you really can be amazing at times)?

- Can we create a battery of repetitive and data driven checks that will help us see if we have an end to end problem (answer is yes, but likely not on the cheap. It will take real effort, time and coding chops to pull it off, and it will need to be maintained)?

- Can we follow along and put our eyes into areas we might not think to go on our own in interesting states (yes, we create scripts that allow us to do exactly this, they are referred to as "taxis" or "slideshows", but again, they take time and effort to produce)?

- Can we set up sessions where we can create specific charters for exploration (answer is yes, absolutely we can)?

- Are there numerous "ilities" we can look at (such as usability, accessibility, connect-ability, secure-ability)?

- Can we consider load, performance, security, negative, environmental, and other aspects that frequently get the short end of things?

Even with all of that, and even with the most dedicated, mindful, enthusiastic, exploratory minded testers that you can find, we still won't ferret out everything. Having said that, if we actually do focus on all these things early on, and we actually do involve the entire development team, then I think we will be amazed at what we do find and how we deal with them. It will take a team firing on all cylinders, and it will also take focus and determination, a willingness to work through what will likely be frustrating setbacks, lots of discoveries and a reality that, no matter how hard we try, we can't fix all issues and still remain viable in the market. 

We have to pick and choose, we have to be cautious in what we take on and what we promise, and we have to realize that time and money slide in opposite directions. We can save time by spending money, and we can save money by spending time. In both circumstances, opportunity will not sit still, and we have to do all we can to somehow hit a moving target. We can advocate for what we feel is important, sure, but we can't "have it all" No one can. No one ever has. We have to make tradeoffs all the time, and sometimes, we have to know which areas are "good enough" or which areas we can punt on and fight another day.

Bottom Line:

No matter how hard we try, no matter how much we put into it, we will not find everything that needs to be found, and we will never fix everything that needs to be fixed. We will have plenty to keep us busy and focused even with those realities, so the best suggestion I can make is "make the issues we find count, and maximize the odds that they will be seen as important". Use the methods I suggested many posts back as relates to RIMGEA, and try to see if many small issues might add up to one really big issue. Beyond that, like Mauri said at the beginning, just keep trying, and just keep getting better.

Try to Find Problems and Victims: 99 Ways Workshop #95

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #95: Do not try to find errors or bugs - try to find problems and victims. Testing is more than checking. - Thomas Lattner

Wow, that's a provocative statement, and it's one I agree with 100%. One of the things that happens a lot is that we end up couching "issues" and "glitches" into safe words like that. We don't talk about "problems" or "bugs" or "catastrophic failures" because they might have the potential to hurt someone's feelings. We are encouraged to report dispassionately, to stick with facts and to couch words in ways that are not inflammatory.


For the long term survival of our jobs and sanity, that may well be appropriate. However, when we test, drop the polite shtick. As Steve Martin quoted many years ago, 'Comedy is Not Pretty". Neither is testing, if it's done right. We don't have to speak to others like murderous pirates, but we sure had better do so to the code we hope to test, otherwise we will miss something important.

Workshop #95: Exercise the product you are testing with an air of "what's the worst thing I could do here?" Think of a person that could be harmed, and how. Do your absolutely diabolical worst to see if you could expose the most sensitive aspects of their data and exploit it. Then write it up (in whatever dispassionate verbiage you choose) and lay it on the table. See who stands up and reacts.

Yeah, these are getting a little more tricky to write here at the end, I will admit it. I already feel like I've said this several times in other workshops, but I'll say it again here. We do our best persuading when we can make the bugs we find personal, when the programmers and product team can most directly empathize with the pain. Therefore, set yourself up so you can really bring the pain (or give it your all trying).

- Create a persona.

- Make it as detailed and as data rich as you can.

- Give this person a back story, and as much "dirt" as you would want to keep hidden.

- Then do everything you can to expose that dirt (or have one of your teammates try to do it).

Some of you are saying I'm taking "bug hunting" and I'm creating an inversion. In a way, yes, that's exactly what I am doing. I'm approaching the application from the aspect that I want to learn all I can about that person, and I want to do so with whatever restraints I can configure. The more restraints, the more aggressive I want to try to overcome them. 

Can I determine a password? 

Can I sent HTTP requests that will send me back raw data in clear text? 

Can I get to their credit card information? 

Can I order things on their behalf? 

One of the oft heard phrases is "well, no user would do that". They are right; no normal user we have ever envisioned that would be friendly to our product would aim to do such things. We're not testing for nice people. We are testing to help us thwart truly rotten, obnoxious and dangerous people. If we could put a human face and emotion to the issues we find, we will get much more attention than if we have some abstract, corner case feeling bug. Think about it, what's going to draw your attention more, someone saying: 

"in this obscure case, where I entered in the same password 700 times, I was able to throw an exception and leave the machine in a bad state" 

or 

"hey, check this out! Running this looped bash script and cURL, I was able to clobber the machine, get to the database prompt, and I now have the credit card information of all our customers. Yee Haw, who wants a Harley?!!" 

Bottom Line:

Sometimes it's a little to easy to get into the mindset of focusing on the features we are testing, but neglect the risks that might reside around those features (and yes, that's a point I made in another post a few days ago). For many, though, as Roland Orzabel* so aptly put it in "Goodnight Song", "Nothing ever changes unless there's some pain." It's up to us to help our customers see the potential pain, and  prove that it is possible. Not an abstract pain, but one that has a person's name, face and anguish behind it. The more you can make the hair stand on end for those discussing your findings, the better :).

* Tears For Fears, Elemental, 1993

Understand Your Domain (and Your Competitors) Fully: 99 Ways Workshop #94

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #94: Understand your domain fully and as importantly your competitors - Stephen Blower

There's no question that having knowledge of your domain is critical if you want to be successful in whatever job you are doing. I used to believe that I was capable enough that I could test anything. Yes, I suffered from hubris when I was younger, but indeed, I believed that all i'd have to do was apply some testing maxims and I'd be able to take my testing knowledge anywhere. In superficial cases, that's true. Domain analysis and equivalence class partitioning are the same everywhere, but how you use them, when you use them and how they will actually inform you of something relevant depends on the domain and your understanding of it. Put more bluntly, I don't care how great a game tester you are, if you want to apply at Genentech and work in their biomedical engineering initiatives, you had better know something about the world of pharmaceuticals, chemistry and biomedical engineering.

That covers the domain, sort of, but what do we do about understanding our competitors? Ahhh, now we can talk about something fun and entertaining, at least, if you do it the right way. 

Workshop #94: Round up some of the primary applications in your area of work and expertise. Your own products and those of your competitors (demo or trial software is fine). Create a matrix of applications and then set up criteria you would like to compare. Walk through all of the applications with an eye towards dispassionately seeing how they all stack up against each other (i.e. an application "shoot out"). gather your findings and analyze them. Determine which applications come out on top in the various categories and write up executive summaries. Share these findings with your development/product team.

I'll be frank, Competitive Analysis is fun, if you approach it with the right mindset. 

First and foremost, you have to make one simple commitment… set your personal biases, passions, and loyalties at the door. If you want to do competitive analysis, you have to be prepared for the fact that your product may not perform well. 

Second, you need to construct workflows that are representative of the way that your company and your competitors products interact with their users. Get creative with this, and see how many of these workflows you can create. Work through ways that you can quantitatively and qualitatively represent the interactions. 

Some examples can use benchmarking tools, or can use your automation tool of choice to run the system through hundreds or thousands of looped tests. How did your app do? How did the competitors app do? Can you record the details in such a way that, when you declare the winners/losers that you can do it based just on the numbers, and not on your knowledge of what's been entered? Note: this works best when you have a team that you have given random numbers and those numbers correspond with a platform that you, as the evaluator, don't know about. 

Quantitative reviews are fairly simple; if the specs and numbers are faster for one app vs. the other, it'll be there in the numbers. Qualitative reviews are more tricky. How do you rate user experience? What feels good vs. feels not so good? In qualitative reviews, language needs to be precise, and it needs to be consistent. Since I am comparing many different examples, you want to make sure that the level of your review, and the language you are using, is applied fairly. Collect the data from these activities in a database or, if you want to be old school, use a spreadsheet. See my earlier post about dashboards; competitive analysis is a fun place to play around with dashboards, because I am looking at lots of interesting data points that can be fiddled with.

Doing competitive analysis is a great way to do a little bit of sleuthing and play detective with competitors products, but it has a marvelous knock-off effect. Looking at a whole range of products in a similar product space will quickly help you to become a domain expert, not just on the products, but on the business range as well. Back when I worked at Connectix, I did a lot of testing on all of the available virtualization options at the time, and by the time I got through with doing that, I learned a tremendous amount about how virtualization was being used and who was using it for what. The more examples and workflows you compare, and the more products you cover, the deeper the domain knowledge you get to develop. It's cool how that works :).

Bottom Line:

Learning your domain is important,. Learning how your competitors handle your domain is equally important. Going through the process of studying up on your competitors, how they do what they do, and with a critical eye for what and how you can improve testing of your own products/applications will give you an edge over testers who don't do this. Oh, and did I mention that it can be a lot of fun performing these product and workflow shoot-outs? Seriously, it is :).

Understand Your Business and Customer Needs: 99 Ways Workshop #93

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #93: Understand your business and customer needs, not just the requirements - Mike Hendry

I was so tempted to put this into a combined post with the last one, because a lot of the suggestions would be the same. However, a  funny thing happened on the way to throwing in the towel and saying "OK, this could use the same advice as the one that I just wrote". I participated in a conversation on Twitter with another tester that was frustrated with the fact that "meaningless metrics" were getting their team nowhere. While they were still being asked for measurements, there was a chance that management might be open to another conversation. In the process of that conversation, we came down to a piece of advice that I realized answered this suggestion pretty well. I love when things like that happen :).

Workshop #93: Sit down with your management/product team and have a frank discussion about risk. What are the real risks that your product faces? Get specific. Map out as many areas as you can think of. Identify key ares that would be embarrassing, debilitating, or even "life threatening" for an organization. Develop a testing strategy that focuses on one thing, mitigating those risks.

Most organizations couch testing in terms of coverage or bugs found. It's a metric, and it may even be a meaningful metric, but very often, it's not. "Coverage" is vague. What do we mean? Are we talking about statement coverage? Branch coverage? Usually, the term "test coverage" is used, and again, probe deeper and see if that word means what the speaker thinks it means. If someone asks if "all possible test cases have been identified", we've got problems. At this point, it would be helpful to instruct and show that complete, exhaustive testing of all scenarios is impossible (not to mention infinite).


In most of the places I have worked, "risk" has not been directly assessed, and there are valid reasons for this. True and honest risk assessments are hard to do. They are subjective. Risk to who? Risk in what way? Under what circumstances would something we consider to be low risk become high risk?

Risk is not a sure thing. It's the probability that something could go wrong. The more something is used, the higher the probability that something will go wrong. Not all risks are weighted the same. Some risks are trivial and easily shrugged off ( typos and cosmetic errors) because the "damage" is minor. Other risks are much more important, because the potential for damage is very high (an iFrame can open you up to cross server scripting attacks). Risks are identifiable. They may or may not happen, but you have a handle on how they could happen. 

Here's a process you can use. The next time that you sit down for a story workshop (or whatever you refer to an initial exploration of a new feature idea and implementation), take the time to ask the following questions:

- What would be the downside if we didn't deliver this feature?

- What would be potential problems that would prevent us from implementing this feature?

- What other areas of the code will this feature associate with?

- In what capacity could we run into big trouble if something isn't configured or coded correctly?

- What are the performance implications? Could this new feature cause a big spike in requests?

- Is there a way that this new feature could be exploited, and cause damage to our product or customers?

Yes, I see some of you yawning out there. This is a blinding flash of the obvious, right? We all do this already when we design tests. I used to think the same thing… until I realize how much we were missing at the end of a project and we opened it up to  much larger pool of participants. Then we saw issues that we hadn't really considered become big time bombs. We all started working on interrupts to fix and close the loops on areas that were now bigger issues. It's not that we hadn't tested (we had, and we did lots of testing), but we had placed too much of our focus on areas that were lower risk, and not enough focus on areas that were higher risk. Our test strategy had been feature based, instead of risk based.

A risk analysis should consider a variety of areas, such as:

- defects in the features themselves and customer reaction to those defects

- performance of the feature under high load and many concurrent users

- overall usability of the feature and the user experience

- how difficult will it be to make changes or adapt this feature based on feedback

- what could happen if this feature were to leak information that could allow a hacker to exploit it.

Each of these areas are risks, but they do not have the same weight. At any given time, these risks can change in weight based on the feature, platform and audience. Security may be a middle-weighted issue for an internal only app, but much more heavily weighted if it is a public facing app. Performance is always a risk, but have we considered certain times and volume (healthcare.gov being a perfect recent example)?

Additionally, identify features that are critical to the success of a project, their visibility to users, frequency of use, and whether or not there are multiple ways to accomplish a task or only one way. The more an audience depends on a feature, the greater the risk, and the more focused the testing needs to be. Likewise, if the item in question is infrequently used, is not visible to a large audience, and there are other avenues or workarounds, then the risk is lower, and the need for voluminous tests lessened. 

Ideally, we would spend most of our time in areas that are high risk, and very little time in areas with little to no risk. Making that small shift in thinking can radically alter the landscape of your test cases, test coverage and focus.


Also, we would like to prevent another insidious risk factor that can change this equation and balance, and that's time pressure. If we have to compress a schedule, we radically alter the risk profile. Issues that are relatively low risk given enough time to test become much higher risk when there is time pressure and a "death march" mentality to getting the product out the door.

Bottom Line:

Everyone's risks are going to be different. Every organization has different fears that keep them up at night. Making a check list to consider every potential risk would be pointless, but a framework that allows us to examine the risks that are most relevant to our product owners, company and customers will help us to set priorities that are relevant, and place our efforts where they will have the most potential impact. 

Again, risk isn't black and white. It might happen. We might do something that could cause us damage down the road. We might be performing lots of "cover our butt" tests that, really, have a very low likelihood of actually happening in the wild, while missing important areas that have a much higher chance of occurring. Shift the conversation. Move away from "how many tests have we performed" to "how many risks have we mitigated, and are we focused on mitigating the right risks?"

Hang Out With Developers, Designers, Managers: 99 Ways Workshop #92

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #92: Hang out with developers, designers, managers - Rosie Sherry

This is a great piece of advice in general. We learn a great deal about our role as testers when we have a better understanding of what other people do in their organizations and roles. What matters to them, what pressures they face, the issues that are unique to their spheres, all of these can give us a better view and frame of mind to help us approach how we test. 

Beyond that, though, is the potential for knowledge transfer and skill acquisition. Ever wonder why certain tags and attributes are used on the web site you test? You could look up those tags, or you could ask your front end developers why they choose to use what they do. Ever wonder why certain features seem to outweigh others, but you can't understand the rhyme or reason as to why? Talk with the project and product managers and see what they are dealing with. If you are interested in learning a new programming language, you could read up online about it, or you could go attend a Meet-Up related to that language, meet some people who attend, and ask them what makes it a good choice (or not, believe me, programmers are happy to share both the good and bad about what they deal with).

Workshop #92: Make a plan to do several "gemba walks". Each of them should be in a different place in your organization, or in a different aspect of what you want to learn more about. Attend team meetings, talk to individuals, join and attend meet-ups related to the skills and areas (UX, programming, management, lean coffee, etc.). Try to see how what you learn from these experiences can add to your ability to test more effectively.

As I mentioned in a previous entry, "gemba" is a Japanese term that means "where the action is". therefore, the best chance for you to learn what developers, designers and managers know, need, fear, and aspire to is to go where they actually do their work. Some basic ways to do this are as follows:

Pair with a programmer regularly if the opportunity arises, or make opportunities that are beneficial to both you and the programmer. One great way to do this is to see if you can focus on a story together in the early stages of it being programmed, or to collaborate on automated tests you have decided need to be created.

If you want to see what is important to project managers, ask to get on their calendars as they are looking to scope out new features that have been requested. Offer to sit with them and brainstorm approaches and ideas, letting them lead the way while you ask them questions. As they are scoping out ideas for features, you can also make suggestions that will help with testability, or give some suggestions that will help broaden (or narrow) the focus of a feature so that it can be more easily implemented, described, and yes, tested :).

Find out who the managers are in various capacities in your organization, and make the point that you would like to learn more from them as to what is important for organizational success. This doesn't have to be a formal meeting, it could be a lunch break, or a visit to a local pub after work, or in some other capacity. The point is, get to know what these managers are doing, and why they need what they do to be successful. Be open and honest about your intentions. Also, be persistent, and be the one to make the invitation, and do it regularly. You may notice at first that "oh thanks, but I'm too busy" is often the first response. That's OK, but follow up and ask again, and again. Usually, the response will be "Oh, OK, you're serious about this. Um, sure, how about Friday for lunch?" 

Barring any of these (it's possible that you work remotely or in a way that you can't directly access the people you'd like to talk to), go the Meet-Up route. I am blessed to live in an area that has an embarrassment of riches when it comes to structured meet-ups of every imaginable type. Sure, getting together with my team of developers is effective, but if none of them are working with the language I'm interested in learning more about, then going external is necessary. Same goes if I want to learn more about Lean Startups, mobile applications, User Experience, or software management. Another benefit to the Meet-Up approach is that you can see what other people value in their organizations, and see how other people tackle problems that may inform the ones that you have. 

Bottom Line:

We can learn a lot by reading, experimenting, and trying things out on our own, but ultimately, our success depends on the organization as a whole being successful. The better we understand the unique challenges other groups face, the better equipped we will be to target our testing to the areas that have the greatest risk (i.e. those ares that each groups feels are the greatest risks to them). By getting a feel for and an understanding of what each group does, actively tries to accomplish, and yes, even secretly or not so secretly fears, it is possible to focus our attention and energies on the areas that really matter to a broad and diverse set of stakeholders.