Learning, Upskilling, and Leading to Testing (Michael Larsen with Leandro Melendez at PNSQC)

You all may have noticed I have been quiet for a few hours. Part of it is that I was giving a talk on Accessibility (I will post a deeper dive into that later, but suffice it to say I shook things up a little, and I have a few fresh ideas to include in the future).

Also, I was busy chatting with our good friend Leandro Melendez (aka Señor Performo), and I figured it would be fun to share that here. I'm not 100% sure if this will appear for everyone or if you need a LinkedIn login. If you can't watch the video below, please let me know.

 

We had a wide-ranging conversation, much of it based on my recent experience being a testing trainer and how I got into that situation (the simple answer is a friend offered me an opportunity, and I jumped at it ;) ). That led to talking about ways we learn, how we interact with that learning, and where we use various analogs in our lives. This led us to talk about two learning dualities I picked up from Ronald Gross' "Peak Learning" book (Stringers vs. Groupers) and a little bit about how I got into testing in the first place.

It's a wide-ranging conversation, but it was fun participating, and I hope you will enjoy listening and watching it :).

Off the Cuff Accessibility and Inclusive Design: #PNSQC Live Blog

And here I was wondering what I was going to say to give a highlight and wrap-up for my own talk that I gave yesterday.

The PNSQC Crew beat me to it. They asked me to give a video summary of my talk and some of my impressions of Accessibility and Inclusive Design.

This video chat was unrehearsed, so if you are used to my nice and polished podcasts, where I seem to speak super smooth and without flubs, today you get to see the real me: no edits, no do overs and a much more realistic representation of how I talk most of the time, hand flails and all.

To quote Ms. Yang Xiao Long... "So... that was a thing!"

How Can We Interview Testers Better? - Live from #TestRetreat

One of the challenges that anyone who has been involved with hiring software testers can tell you is that it can be maddening to interview testers. We can find people, but getting the right people is often a struggle. We have all had the experience of reading a resume and seeing what looks to be very promising history and experience, only to have them in an interview and have to ferret out what they actually understand or do not understand. is there a way that we can do this better? Dwayne Green led a group of us to discuss how we approach these interviews and how we can improve the process.

Many of us have worked through resumes and had to make phone screens or initial interviews, and a common phrase that came up was to "audition" the candidate. There are several approaches to auditions that we can use. Some people will use a sample program and walk through how to test it. Some people like to use their own company's product as part of the audition, and to see how the candidate tests the product, or if they can find problems we already know about.

A discussion I recently had with our VP of Engineering at my company was the way that we expect people to work. Most of the programmers that write code have certain things open on their desk at all times: their IDE or programming environment, a browser with a google tab open and Stack Overflow or some other reference sites. the point being made is that, when we work, we tend to have these tools open to help us get to the things we need. When we interview, we deprive these candidates of that ability, and we expect them to "code" on a whiteboard. One of the ways that was suggested that we could change this would be to ask in advance what they like to use to work, and to feel free to bring that in with them. When we put the audition challenge in front of them, we also say "use the tools you would use". This has tended to give a more representative view as to how those programmers actually code. We discussed the idea that people should do the same thing when it comes to software testing. Let a candidate come in and use the tools they already understand.

One of the questions that we asked was "how could we make these interview approaches more real, but avoid a situation where we are making people "work for free" as part of the interview". There is always a danger that putting people into a simulation or scenario with the company's product runs the risk of replicating real work without compensation. One approach could be to use a virtual machine with a version of a release and a story that has already been worked, and have a list of issues that have already been discovered. Do we want to have them run an abbreviated test session and see what they discover from our list? Do we want to have them sit with our team members and pair for a particular period of time? In my view, I think we need to make clear that we are going over material that has already been covered, and that we are using it as an evaluation criteria and not as a way to get "free testing" out of someone.

Some people like to use games or use sample programs to do these experiments. The issue there is that some people are not good at games, or are not good at particular applications. Does that mean they are a bad tester? Probably not, but it does mean we need to take into consideration multiple avenues. They might not be good at one area, but be great in others. If we find that they don't do well in multiple areas, that might well give insights as to how well they will perform or not perform in our work environment.

We all discussed some of the worst experiences we had with interviews, and I shared that would often be asked some strange questions related to math problems or other specific questions that seemed to have no bearing on what we would be testing. Over time, I came to realize that these questions served one purpose: does this person think like I do? To be fair, it's been years since I've had to deal with those types of questions, but I know that some people still do, and if I can make one plea, it's to say "knock it off!" ;).

Tea Time With TESTHEAD?

For a good number of you, this is probably already old news. For some of you, this may be new. In either case, I would like to thank Lalit Bhamare and the staff of Tea Time With Testers for featuring me not once, but twice, in the July 2013 issue.

First, I wrote an editorial for the opening of the magazine titled "A Degree in Software Testing?". In it, I lay out the ideas that I think would make for a good degree program, and I state that, as far as I'm concerned, we should stop waiting for the institutions of higher learning to catch up to us, and instead, we should forge ahead and make our own path. In my mind, a software testing degree would look a lot like a classical Liberal Arts degree, with additional emphasis on computer science, philosophy, writing, mathematics and, yes, some core focus on actual nuts and bolts testing.

Second, Lalit and the staff at Tea Time With Testers came up with a list of questions to ask me about how I got into testing, what I'm doing now, my reasons for live blogging (and blogging in general), tester education, SummerQAmp, developing technical chops, Weekend Testing and a bunch of other areas. It was a lot of fun to do, and I appreciate the chance to share some of my ideas with the broader community.

Is this an elaborate pitch to have you go and get the July issue of Tea Time With Testers? As a matter of fact... yes, yes it is. So what are you waiting for :)?

The "Silent Partner" in the Interview

A friend of mine has been recently interviewing. In the process, they have had to face the sometimes enjoyable, sometimes frustrating, and occasionally anxiety ridden "silent partner" of the process... the whiteboard.

In the past few years, it seems that the whiteboard and its quiet involvement has become the primary means of interviewing. I think there are several reasons for this, the most basic being that you can't really script for it. We've all had the opportunity to hear the constant refrain of generic interview questions over the years, to the point where people are automatic in their responses. They are also uninspired; I can't say I blame them, I hate answering those canned questions, too, so I look forward to the whiteboard experience. It should also come as no surprise that I'm the "whiteboard interviewer" on my team today.

Now, before I get something thrown at me, let me explain why I think the whiteboard is so important. I'm a software tester, and one thing that we know well in our field is that there is no "one correct answer" (all right, sure, if you are asked what 2+2 is, I will expect you to answer 4, and if you answer something else, you'd better be prepared to really entertain me ; ) ). For most things, though, there isn't a pat answer. There's a lot of variety, and much of it can be seen by examining the history and experience of the tester. That's why generic questions are terrible, but creating a wire frame of an application and asking someone "how would you test this?" is a lot more interesting.

As I said, I'm not really interested in the "correct answer" as I am to see how someone thinks, how they process information as it comes in, how they pivot and adapt to that information, and what they actually believe. Yes, in software, I want to know what you believe in, and what you will fight for. We use a phrase in testing. We call it "Bug Advocacy". What it really means is "Can you sell me on why this issue is important?" One of the experiments I like to do (and I owe Cem Kaner for this idea) is to build a wire frame of an application, usually rudimentary and missing a bunch of stuff, and see how far the candidate will go. If they only say "well, I'd do this and this and this..." and jut talk about the features they see, that gives me an indication of their level of involvement and their focus. Note: I'm not being critical here, some testers really will focus on exactly the cases they can see, and they will perform them flawlessly. What gets me excited, though, is when I post the question "so, do you think we're ready to ship this?"... and they stop, at first apprehensively, and then you see it, that wicked little gleam in their eye, where they say "hey, I know that the requirements call for this, but what if we added this option? I think we are not being consistent if we don't have this control..." and they start looking at the requirements both explicit and implicit, and start proposing additional ideas, and how they would test them.

You might think that this would be a clincher if they do this, and often, it's a really good indication, but I do keep one more trick up my sleeve (and again, Cem Kaner gets the credit for this approach). After we discuss the areas that they would test, and how they would offer improvements, if they made the suggestions, I then go back and make a second revision of the wire frame, and I will deliberately leave out the item that they made the biggest deal about. This serves two purposes. The first is that I want to see if they will remember what they suggested, and if they do, will they make a point about bringing it up to remind me? If I offer resistance to putting it in, do they just shrug and go on, or do they try to sell me on why it should be there. When I see someone willing to make a case for a feature they suggested (not argumentatively, but making a clear and informed case as to why it deserves to be there), that's when I know I've got someone I want to talk to more in depth, or make a serious effort to get on board with us.

For those who are programmers or who produce art, this may not be your experience, but if you are hiring testers, I would absolutely suggest making the whiteboard a silent partner in the interview. The secrets the board will reveal may prove to be way more valuable than you might think.

The Best Podcast We Are Going To Put Out This Week (By Some Definition Of Best)

Today marks a milestone. This morning, Episode 131 of This Week in Software Testing has been posted. In it, we conclude the series of Matt Heusser's keynote adress at the Pacific Northwest Software Quality Conference (Part 1 and Part 2 can be obtained here).

This also represents 125 episodes that I have personally produced (I came onto this project back in July of 2010, and my first show, Episode 6, was posted on Aug. 6, 2010).

It's been a lot of fun, I've learned a lot along the way... and you can probably tell with this buildup that there's a change in the air.

There is.

Matt and I have decided that, for now, we are going to put the TWiST podcast on hiatus. No, there's no bad blood between Matt and me (as a musician, I am very familiar with the first question asked whenever a project goes on hiatus; it's assumed that there must have been a fight of some kind). We decided to take stock of what we have covered, who we have talked to and areas that may yet to be covered, and decided that now was a good time to take a breather after two and a half years of weekly production work.

We both love producing this podcast, so for those thinking it's going to go away forever, rest assured, that is not our plan. We both, however, want to take the time to focus on some other opportunities for a bit, and give those projects a chance to get some atmosphere to breathe (there are, after all, only so many hours in a given week that we can devote to those opportunities).

To those who have helped make this a fantastic journey, my thanks. To Matt, first and foremost, for being willing to be in the hot seat week after week, in one way or another, either in interviewing, gathering talent for discussions, or recording "on location" events along with me.

Thanks to our recurring "panel guests" over the past year plus, who helped us make for some great conversations, with special thanks to Benjamin Yaroch and Wade Wachs for their frequent contributions and going above and beyond the call so many times.

Thanks to everyone we have interviewed over the past two and a half years. It's your stories and your experiences that make the show what it is.

Thanks to the people behind the scenes these past two and a half years who have helped with show production, editing, and distribution (Rick Baucom, Rich Hand, Thomas Ponnet, Justin Rohrman, Janette Rovansek, Mark Tomlinson and Farid Vaswani).

Most of all though, thanks go out to YOU, our listeners. You are the reason that we were able to put out "the best software podcast we were going to put out this week (by some definition of best)", week after week. We hope that you have enjoyed these presentations, and will continue to enjoy them well into the future. We look forward to coming back and following the pulse of the software testing world in the not too distant future.

Until then, thanks for listening, and keep discovering :).

Regards,
Michael and Matt

I'll Be the Roadie...

As my son gets older and takes on more interesting challenges, it's been interesting learning a bit how his mind works and what interests him. Currently, he has an interest in film production.

This past weekend, he told me that he needed to do some work for his final in Film, and he decided he wanted to do something where he interviewed street performers in San Francisco, specifically the stretch between Fisherman's Wharf and Pier 39. Seeing as I really only have a handful of these opportunities left (he's a Junior in high school now; three more semesters and he'll be graduated and off to college), I wanted to help him meet his objectives.

At first, there was some resistance. "Dad, it's OK, I don't need help, I can do this on my own!" Well, of course you can, dude, but that's not why I'm asking to come along. I wanted to see if there were ways I could see how he works and understand what he does, and in the process, be of some assistance to him (and sure, keep an eye on him, just a little... I'm a Dad, cut me some slack here ;) ). I finally swayed him with some simple words. "You handle the vision and the talking. I'll be your roadie." Meaning, I'll carry all of the gear for you, and you focus on finding the people you want to film and talking to them. He liked the sound of that, and thus, we were off to the City.

Seeing as it was a cold, foggy Saturday (typical for San Francisco) and expected to rain (not as typical) we were concerned that there might not be enough people to interview in the time we had allotted, so I offered to do some advanced scouting with him. As we walked towards Fisherman's Wharf, he noticed a Rastafarian bongo player setting up with a coffee can for tips. My son walked up to him and asked him if he could film him and use him in his project. No answer. Stone silence but him playing his drums. I waited to see how my son would react, and what he would do. Would he ask again? Would he move on? Do something else? I realized this was a different level of communication for my son than he was used to, and decided that this was a time that a "Seasoned roadie" could offer some assistance. I reached in my pocket, took out a few dollar bills, placed them in the drummer's coffee can and asked him the same question. He lit up a bit and said "sure, what would you like to know?" With that, we set up the microphone and camera, and Nick did a quick and somewhat stilted interview with the drummer. Not real communicative, but that's OK, we have to start somewhere.

As we walked down Fisherman's wharf several times, we noticed there weren't any performers out. Was it the weather? Were we just early? What could we do as a contingency plan. It was here I suggested to my son that, maybe, talking to some people on the street about their experiences with street performers might prove to be interesting. It would offer a different perspective to his project. He thought about it for a minute and said, "sure, let's give that a try". We set up the gear again, and we waited. Most people just walked past us hurriedly, but a couple of people did stop to talk with us. After  about ten minutes, he decided we should take a look farther down the wharf and back to Pier 39.

When we made it back to Pier 39, there was a performer with a Chapman Stick. I lit up at this, since I was very familiar with the instrument, and maybe my fan boy-ness got the better of me, but I suggested that this would be a great person to talk to. Since he was also set up in a prime spot right outside of Pier 39, I figured he was likely a professional who rented the space (turns out I was correct with that assumption). I also figured, if I could strike up a conversation with him about his instrument and what I already knew about it, that might work in our favor, too. I mentioned how much it blew me away when I first heard Tony Levin playing the Chapman Stick in the early 80's with King Crimson, and of curse the performer was familiar and we talked about other musicians who likewise used the instrument. thus, when we asked if we could interview him for my son's project, he was very gracious and let us record several numbers and he provided a lot of great interview commentary.

As we walked a bit further back towards Fishermans' Wharf, I noticed a lady and a gentleman get out of their car on a side street in Dickens' era garb. Performers? Certainly. Street performers? Hmmm, maybe not, they were near the Cannery, and that has a pretty swank hotel in it. My guess is they were performers for the hotel by  the level of their outfits, but I figured, hey, they're here, let's see if they'll talk to us. My son agreed, and we went over to chat with them. Turns out they were waiting for another member of their group, and yes, they worked for the hotel, but they were happy to talk to us about what they did, and why they enjoyed performing here particularly, and when their third compatriot arrived, they sang a song for my son to film.

As we were feeling pretty flush with the excitement of capturing two good interview subjects, we walked back to the wharf and noticed an elderly man dressed like he was from the Roaring Twenties performing a magic act on a portable table. MY son looked at me, winked and said "let me handle this one!" He went up and talked to the man and asked if he'd be willing to be interviewed. the magician said sure, but not for free. I casually handed my son $10, and motioned to the box on the table. He put in the $10 and then the magician said he'd be happy to perform his entire act for him, provided he'd be willing to send him a copy of the footage (sure thing!).

After filming this, it was starting to rain, and we figured that this was going to be it for the day. As we were walking back to the car, my son saw, across the street, one of the guys he really wanted to interview... the elusive and legendary "Bush Man". This guy is famous (and infamous) for scaring and freaking out tourists, and he would have definitely made for a fun part of the project, but with the rain coming down and not wanting to risk damaging his equipment, he decided to let it go.

All was not lost, however, since just around the corner, under a protective awning, one of the "Gold Statue Dancers" was there. For those not familiar with the "Statue dancers", they typically stand perfectly still until someone steps up and drops some money into their cup, and then they start dancing in a robotic fashion. This one stood on top of a milk crate, and incorporated the crate into the performance, actually sliding and stepping with it (quite proficiently, I might add). At first I was leery, thinking this guy wouldn't want to be interviewed, but my son, having seen how the rest of the day had gone, decided to give it a try. He walked up, dropped a $5 into the dancers cup and asked if he could ask him some questions. Gold Statue Dancer Man turned out to be the most animated and entertaining of all the interviews, as well as the longest.

As we drove home, I told him a bit about how today was a good example of what I do as a tester every day. With him as my product owner (my customer) I acted as roadie and helped set up the gear, helped evaluate situations, offered suggestions of avenues to try, provided support and alternative approaches when things weren't panning out, and provided some domain knowledge to help him communicate with the performers where needed. Mostly, though, I provided him with information that may or may not have been effective in helping him make decisions about what to do. All in all it was a successful outing, and a god reminder to me that "playing the roadie" can be a good metaphor for good testing, both in software and in film production.

Ask the TESTHEAD

I have an interesting opportunity.


Software Test and Quality Assurance (ST/QA) Magazine runs a feature called "Ask the Tester" where they pick anywhere from 10 to 15 questions and present them to the person selected. They then answer those questions and they become an article in the magazine.

For the May issue, that tester will be me :).

A number of people have asked me various questions already, but I wanted to throw this open to anyone who would be interested in participating. If you would like to ask me a question that has something to do with software testing, please include it as a comment to this message I will then submit the questions to ST/QA and they will pick the ones that they would like to use. If you would like to leave a Name and a City where you are located, I can pass that along as well as part of the question. If you don't leave a name, we'll just include "Name withheld" with the question.

So here's your chance.If you've ever had a question you have ever wanted to ask me (again that's related to software testing ;) ), now's your chance!

A TESTHEAD ST&QA Double Play :)

     In the October 2011 (Vol. 8 No. 5) version of ST&QA Magazine, I get to have a double play :).

     First, this issue is dedicated to the release of our book "How to Reduce The Cost of Software Testing" (available from both the Publisher, Taylor & Francis and from Amazon) and there are a number of stories that are focused on the topic. Many of them are all new material that was written after the book had wrapped. The cover story, coolly enough, is an excerpt of my chapter (Trading Money for Time: When Saving Money Doesn't (and When It Does).


Second, and really humbling is a piece called "What is the Cost of Testing?" in which Rich Hand interviews, well... me! We had a great conversation a while  back, and only about 50% of it appears in the article. Rather than structuring it like an interview, it's a narrative with many of my thoughts interjected. Maybe this is a better approach; Rich was able to edit out areas where I probably rambled a bit (LOL!). Still, I think it's an accurate and representative piece, and I'm happy to have had a chance to be a part of it.

So for those interested in getting a copy ST&QA, go to the SoftwareTestPro.com site and download it, or you can read both the Cover Story or my Interview right there on the site. If you do, please leave a comment or two, I'd love to know what you think :).

What Motivates You?!

This morning, Episode #68 of This Week in Software Testing went online, and in many ways, this was one of my favorite episodes to produce. We still deal with the challenges of optimizing Skype for multiple conversations (it's getting better) and it was tough to decide what to keep and what to trim out, but I think the result is worth it.

This show also celebrates the first appearance of Jonathan Bach as a contributor. We've been trying to get Jon on the show for months, but various scheduling issues and other factors have prevented it from happening. We're glad he could join us for this, especially since we cover one of my favorite topics, Motivation.

It's a challenging thing to deal with what motivates people. We often get it wrong. What motivates me may or may not motivate you. Some people are motivated by money. Some are motivated by attention or fame. Some just want to see an idea of theirs take shape. Some want to grow internally and spiritually. Some just want to have fun.  Many want varying combinations of all of the the preceding.

I discovered that, in a way, my biggest motivation is this... I want to be part of a vanguard movement.

This really doesn't surprise me. It was my primary motivation as a musician, above and beyond just playing and dreaming of being a rock star. I really liked the idea of representing San Francisco and the San Francisco rock scene. More than just wanting to be a rock star, I really wanted to be a rock star "that came from San Francisco". Ultimately that dream fell short of my ultimate and intended goal, but I'll dare say I went a lot farther with it than many people I knew that had similar dreams did. To be fair, many more went much farther than I did, of course, but the motivation was still there, and that motivation defined my involvement.

In Scouting, I've held a lot of leadership positions, ranging everywhere from Tiger Cub Den Leader, Cub Scout Den Leader, Cubmaster, Scoutmaster, Venturing Crew Advisor, Explorer Post Advisor, Order of the Arrow Chapter Advisor, and Assistant District Commissioner (sort of a Board of Directors for the Scouting movement in a given area). Again, much of this came from my desire to be involved and my self identification as a Scouting leader and wanting to make a difference in the lives of families and in my community.

In an earlier post I mentioned that I became part of "the movement" that was the burgeoning snowboarding scene in Lake Tahoe during the early part of the 1990's. That was a core part of my identity for many years, and it still is in a lot of ways. While I never had any real thoughts of ever "turning pro", I did align with and aspire to associate myself with the ideals of the snowboarding movement. In fact, one of my first stabs at editorial writing came through this era. I wrote a number of feature article for an online snowboarding magazine called "Cyberboarder" (don't laugh, this was in 1995, *everything* was cyber this or cyber that... OK, go ahead and laugh, it's cool :) ). I started competing in the late 1990's (as a Masters level competitor, i.e. the over 30 division). I won a few medals, placed in a few slope-style events, and podium'd in several races, actually winning a Gold Medal in a regional event in 2004 in the Giant Slalom. I wrote about my experiences and published them in a series of articles (about 30 of them) called "The Geezer X Chronicles" (you can enjoy my early writings for "Cyberboarder" and "The Geezer X Chronicles" via "The Way Back Machine" if you so choose :) ).

What's my point with these examples? Feeling a kinship with these communities, and actively seeking that kinship, gave me identity. Identity gave me purpose and a mission. Mission gave me drive. Drive helped me produce.

For many testers, I believe that the problem is that there is not quite a sense of belonging, or that many testers are unaware of a broader community. Those who will read this are probably already aware of this broader community. The greater challenge is getting those who are not aware to become aware of it and encourage them to be a part of it.

For me, looking back, the real and true motivator in my life, and the one that has had the most sustaining power, has been kinship with a group of people. Continuous communication and involvement has helped me develop drive and passion, and with that drive and passion, opportunities appear. Other opportunities are spawned by how we respond to the initial opportunities we are offered. If you are wondering how to develop your test mojo, my answer is to find other testers who similarly want to improve. Become a community, or attach yourself to the broader testing community via Twitter, LinkedIn, Forums, Weekend Testing, Associations, whatever. The sooner you become part of a community that you care about and cares about you, the sooner you will kick your own motivation into overdrive. Or at least, that's what did it for me :).

BOOK CLUB: How to Reduce the Cost of Software Testing (20/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry. Ths entry covers Appendix C.

Appendix C: Rapid Test Augmentation by Jon Bach

Jon starts this chapter with the analogy that we outsource things in our lives all the time. When we call a plumber or go to a restaurant for dinner, we are actively “outsourcing”. Why do we do it? There’s usually two big reasons (though there may be several others). First, we need someone to do a job that we can’t easily do. Second, we get someone to do something that we could do, but they can do it much faster. So why do so many have a visceral reaction when we hear the word “outsourcing” in testing? It’s because we often equate the word with “offshoring”; using a different country’s labor (usually overseas from the United States) to do work we either won’t or can’t do without great expense. Man, the arguments I’ve had over the years regarding that topic could take up several posts, and really, the question is “is the argument even fair?”. We have no idea if offshoring our outsourcing is really lowering costs and raising value… or do we? Jon seems to think he does. shall we find out :)?

Prior to working for eBay, Jon worked for QuarDev, which meant he was a manager for hire. Why would you call Jon? Probably because you need something done quickly, with a solid expertise, and a budget to allow you to do that in a way your current team couldn’t at that immediate time (not that they were not capable, but they just couldn’t do what was needed in that sphere at that time under the circumstances that necessitated hiring Jon, right?). One of the things Jon is famous for doing is called Rapid Testing (his brother James Bach gets the credit for inventing the approach, along with active help from tester like Michael Bolton and Cem Kaner. Rapid testing is the “skill of testing any software, any time, under any conditions, such that your work stands up to scrutiny.” (James Bach, Satisfice)

Note, Jon’s team is not cheap. Some offshore serices can do “the job” for 5 times less. The real question though, and Jon is willing to stake his reputation on this, is that they can’t, not with the level of ability he an his team can do. Their philosophy is to make the sales conversation about value, not price.

As we’ve seen in this book, cost is entirely context based. One context doesn’t hold up to another. In this case, Jon is comparing labor costs and the fallacy that all testers are created equal, and there’s really little skill involved and one tester is synonymous with another (editors note; hang out with Jon for about an hour and test with him. I promise you will see the interchangeable testers fallacy is exactly that!).

An interesting development was taking place before the downturn in 2008 called “re-shoring”; projects that went overseas because time differences, language barriers, skills levels and other differences were significant enough that the cost alone was no longer the deciding factor. Many times, it made more sense for a value perspective to keep the work local. Sadly with the downturn, the flow seems to be going the opposite direction again.

To be a consultant means to go beyond the stereotypes and the bad jokes everyone has about consultants (jokes that tend to have a sting of truth to them). Jon’s strategy is to:

1. Charge a fair price.
2. Have some ideas about how to execute what you recommend.
3. Write reports that tell the story you need to tell.
4. Write simply.
5. Tell them what you found out and call them "findings."
6. Talk about what you hope to do next or how you see yourself being involved.
7. Discuss your code of ethics.

Questions to ask clients to help them get the best value out of the engagement:

1. "What's important to you?"
2. "What does success look like?"
3. "What's the worst that could happen?"
4. "What would you like to do less of?"
5. "What would you like to do more of?"
6. "What do you wish were different?"
7. "What problems are you trying to solve?"
8. "What's working?"
9. "What would you start doing, stop doing or keep doing?"

Jon likes to speak to the very top level of management, but also likes to spend time side-by-side with the testers to see how testing actually gets done. A lot is learned at both ends as ell as in between. Does the company value "heroism," and does that mean they they *rely* on heroes to solve major problems every time? Are there processes in place that prevent the heroes from becoming casualties? If so, are they actually applied? There’s a happy medium between heroism that’s excessive, and process that’s stifling.

Rather than polishing up a PowerPoint presentation, take out a sheet of paper during the pitch and ask: "Can we work right now on one of the problems you're having?" The way you probe for values - thoughtfulness, enthusiasm, and an earnest interest in the work – will help lead you to find the best fit for who you partner with.

Some Sample Questions FROM Clients 
Your bug database or mine?
Can I get the same tester as before?
How do you train?
Can I get resumes from your staff?
To what associations do you belong?
Can I see the templates you’ll use?
Can I customize the status reports you give?
If I need to postpone or cancel, what’s the penalty?
Can I talk to a tester in the lab?
What are your working hours?
Will you work overtime or weekends?
What’s your hiring process?
Will we be billed for the hours we don’t use?
How do you measure test coverage?
Why didn’t you catch that bug?

Some Sample Questions FOR Clients 
Your bug database or ours?
Can I talk directly to a developer?
What are your working hours?
Do you work overtime or weekends?
What’s your triage process?
When do you plan to ship?
Will we work onsite or here in our lab?
Do you use any existing tools that would be of help?
Can we see your existing bug database?
Did you want to devote time to regressions?
How often will you be giving us builds?
What are the minimum hardware requirements?
What kinds of users is this targeted for?
Has this been tested before?
Would you be a reference?

The “cost” of something isn’t just about price. It’s all of the context and values and weighting and considerations that go into the computation. Think of the average ink jet printer. They’re actually really inexpensive… the printer’s themselves, that is. The ink cartridges, though, can cost up to half the price of the printer!

List all of the things that matter to you – all of your hopes and ideals and values. Put them into a list and give a “gut-feeling” ranking to each of them on a scale of 1 to 10. Share it with the people on your team who are charged with making a decision. Add, delete, modify. But also stay alert to new context that emerges once the decision is made.

To know “the cost of testing”, you have to know the context of testing, the value of testing, and the questions of testing. You can buy a cheap car for $500 and maybe it’s just what you need. You can buy a new car for $50,000 and maybe it “pays for itself” in the benefits you get from it.

For the question of hiring a test lab to do rapid test augmentation, you may be able to reframe conversation to your stakeholders and decision-makers – helping them realize that you’re not hiring them to find bugs, you’re hiring them to assist you in gaining more visibility about the value you are offering to your customers.

More Tech of TWiST

So we have now done three shows in the new "forum" format, and this has brought to light some interesting challenges.

As we keep working with the format, we discover new details and methods that can be used to help keep the sound as clean as possible. Since this is still an all volunteer deal, we are not currently investing in dedicated voice lines. Skype is still the tool we use, but the more people we have on the call, the greater the odds of breakup in the call.

When in doubt, it's a good idea to see what the "pros" are doing. My pro in this instance is Dan Benjamin, who runs the exceptional podcasting empire over at 5by5.tv. One of the things I like about Dan is the fact that he openly discusses the challenges he faces in producing the shows, and he runs them free and loose a lot of the time, leaving the big edits for bleeps or long delays that need to be tightened up. In any event, during "The Talk Show" with Jon Gruber a couple of weeks ago, he talked about how to get the best consistent vocal quality, even if you are using Skype. The secret? There's two of them. The first is to have everyone actively using their mute buttons on the call when they are not talking. This keeps the cross-talk way down, and also frees up the pipe. Giving each speaker about five to ten seconds to transition also helps keep the cross talk and additional traffic way down. Even more than that, though, is to encourage everyone to not use their wireless connections, and instead to plug their Ethernet cable in. This helps eliminate a lot of problems with connectivity and really helps keep the voice drop outs to a minimum. We realize not everyone can do this, but if possible, please plug in :).

So this week, we continue our discussion with Scott Barber, Dave Limbaugh, and Adam Yuret, with me making some comments as well. While last week was more about discussing the Open Jam sessions being held during STP-CON, this week's episode is more about how to leverage what you learn at a conference and how to come back and be productive immediately. Another challenge is that, even when we get back from a conference with a bunch of fresh ideas, we may find resistance from those who are back home, and a less than enthusiastic response to your renewed vigor and desire for change, so in many ways we have to find things that we can do immediately, and better yet, do things we don't have to necessarily ask permission to put into place. All of the contributors this week offer suggestions in that regard (me, too :) ).

So if I've piqued your interest, please have a listen to Episode #65 of TWiST right here :).

Twisto Changeo: TWiST is Changing :)!!!

As man of you know, one of my side things that I do is I produce the TWiST Podcast along with Matt Heusser. Tomorrow or Friday we will post the last of the "old format" shows. Wait, did he just say the "LAST"?!!

I did, but don't worry too much, TWiST is still active, in the works and Matt is still behind the microphone and I'm still behind the production board. We did, however, decide it was time for a change in the format and the approach we took. Rather than just having Matt talk with a single tester in an interview format, we decided to experiment with a panel format and a designated topic to guide the podcast.

So what does this mean? It means that there will be multiple guests on a typical show, usually four or five. It also means that I'm making a bit of a move out of the production booth and onto the microphone more. So far, we've recorded six shows in the new format, and I'm part of four of them, so I'm looking forward to this new approach. we hope you will, too.

Remember, TWiST is still going to be coming out on its usual schedule, which is weekly, either Thursday afternoon or Friday mornings. Our goal is to make the best software testing podcast out there. We hope you'll enjoy listening to the new format, and if you'd like to be a guest on the show, let me know and we can see about arranging a forum and topic to include you :).

PODCAST FRIDAY: Performance Test Conference, Web Security and Landing a Job

Hey everyone! It's Friday (late) and it's time to give a nod to the podcasts that may be of interest to testers some of them not necessarily testing related, but I hope worth your listening attention. So if you are out and about this weekend and want some quality conversation and learning, these may be right up your alley.

First, of course, I'm not going to lead off a podcast roundup without mentioning the one I produce ;). Episode #51 of "This Week in Software Testing" has Abbie Caracostas as our guest. Abbie is the director of training at Redwood Collaborative Media, which is the company that owns STP (and by extension, this podcast :) ). the primary topic of this podcast is the upcoming STP Performance Summit happening July 26-28, which will be delivered entirely online.

Second, Trish Khoo and Bruce McLeod return with the 3rd installment of TestCast, called "Do I Get the Job?" and as you might guess from the title, this is all about interviewing and discussing how to effectively interview for a testing position. A lot of the approach and terminology feels different to me, again, this show originates in Australia, so there's some local vernacular, but the ideas and thoughts are good to be considered by any tester regardless of locale (and don't wait until you are looking for a job to consider these things ;) ).

Over in 5by5 land, Dan Benjamin and Merlin Mann deliver a somewhat disjointed show this week, but even a disjointed "Back to Work" has lots of gold to mine, and "Assistant to the Regional Monkey" definitely delivers some quirky gold. there's a lot of tangents in this episode, and a fair amount of Buddhist philosophy, but some great advice about working your way through your trouble spots and realizing when you are being a hypocrite and dealing with it (blog post this weekend on that topic by your truly coming soon, by the way).

Dan Benjamin and John Gruber deliver a briefer version of The Talk Show this week. In "Hard Stop", John and Dan discuss various new mobile technologies (tablets and phones and how each market is striving to keep the mobile revolution going mainstream. the FBI raid of a data center and the woes of Dropbox also makes its way into the show, as does Stanley Kubrick and a note to projectionists (John Gruber and Stanley Kubrick, who'da thunk? ;) ).

Finally, for those who are not familiar with Scott Hanselmann and his podcast Hanselminutes, I'd recommend putting it on your watch list. a lot of his podcast topics are Microsoft centric, but before you tune out at that (all you hip Rails and old school LAMP folks out there ;) ), many of the ideas and topics do have crossover appeal, and this week, his discussion of "Web Security Basics with Barry Darrans" provides definite crossover appeal. what do we need to know today to be in the game when it comes to Web Security? What threats are out there and actively attacking sites, and what makes them different?

That's it for this week. Looking forward to a new batch of shows next week. Stay tuned and keep listening!

PODCAST FRIDAY: TWiST #49 w/ Adam Yuret (Part 1)

One of the fun things about getting more involved in the testing community is that you get to know those people making a point of being players in the game. It's much like any scene, really. At first you are the neophyte looking to others as your inspiration and sources of information, but you don't have any direct connection to them. Over time, those names may become people you correspond with or otherwise interact with. As time progresses and you get involved, you work more collegially, and a relationship develops that's more one-to-one, and at some point, you then become one of those players yourself that others look to for inspiration. It's a neat feeling.

Why do I bring this up? Mainly because that's been the way that this podcast has gone for me. Most of the people interviewed at first were people I had no knowledge of other than their names. Later, there were a few people that I'd interacted with briefly and had a chance to do some testing related correspondence with. Through Weekend Testing, I've had the opportunity to learn more about and work directly with a lot of the testers that I've come in contact with, and now we are featuring an interview with someone I've met, come to know and collaborate with directly, and yeah, that's a really cool thing to experience.

So this week Matt sat down with Adam Yuret from VolunteerMatch.org. Adam's another of the "life experiences" school of testers, and he's also another tester who's spent a fair amount of his career in the lone tester camp. He also adds to that in the fact that he's a remote tester working with a team 1,000 miles away. Oh, and he took a three year sabbatical to sail the Sea of Cortez with his wife. Adam was one of the participants in the eBay mini-conf I wrote about a few weeks ago, and he's genuinely a great guy, but seriously enough from me, why note go to the site and listen to and listen to Episode #49 and Adam for yourself :).

Standard disclaimer:


Each TWiST podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to the podcasts and the entire library whenever you want to :). In addition, Pro membership allows you to access and download to the entire archive of Software Test and Quality Assurance Magazine, and its issues under its former name, Software Test and Performance.


TWiST-Plus is all extra material, and as such is not hosted behind STP’s site model. There is no limitation to accessing TWiST-Plus material, just click the link to download and listen.


Again, my thanks to STP for hosting the podcasts and storing the archive. We hope you enjoy listening to them as much as we enjoy making them :).

YT9MF65CTP4Z

PODCAST FRIDAY: TWiST #48 w/ Eric Jacobson

So for those who are interested, I've set up a little tracker in RescueTime because I wanted to see just how much time I spent on a typical interview each week.

Part of this comes down to my own personal tick; I have tried and I just can't get past the idea of releasing a talk or an interview with "um's" or "uh's" or stammers in it. If I could, I could cut the time way down, but since I won't, well, it takes me on average four hours of editing to produce a 30 minute show each week. Yeah, that's super nerdy, I know, but I think it adds to what I hope is a "polished" podcast. I don't know about the rest of you, but I treat many of my favorite podcasts as reference material; I go back and listen to many of them over and over again. It's my hope that these podcasts are not seen as "throwaway" weekly blurbs, but as real meat and potatoes stuff that people will want to hold onto and listen to over and over again as well.

Serious question, does the flow of the conversation feel normal and natural to you, the listener? I hear every click and pop, as wells every overly inflected point when I edit. Some of that is just not fixable without sounding really artificial, though I have joked with SideReel's resident audio editor that I have become the master of the "poor man's cross-fade", which is where you drop the volume on the last word and fade in on the first word where the edit takes place. Do you hear it? Is it enough to notice or comment on? If so, I'm genuinely curious.

So this week Matt sat down with Eric Jacobson. Eric works with Turner Broadcasting, and manages a team of testers. I liked this interview because he talked about what can be a challenging topic, and that's dealing with motivating testers. I know full well how it feels to be a tester who is just going through the motions, without any real purpose or focus, and I like what he has to say about helping testers get engaged and involved (it inspired an article I'm hoping will be published soon; I'll let you all know if it gets picked up). Eric also explains what it was like to be "invaded" by the Rebel Alliance at a conference last year, where he found out he didn't get the room he had hoped for, but they gave him a suite with a fold down bed, but a big space. This led to a meeting and talk session of the Rebel Alliance and a number of lightning talks by the participants (and video footage, too :) ). So anyway, if you're tired of my blathering, by all means go to and listen to Episode #48 for yourself :).

Standard disclaimer:


Each TWiST podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to the podcasts and the entire library whenever you want to :). In addition, Pro membership allows you to access and download to the entire archive of Software Test and Quality Assurance Magazine, and its issues under its former name, Software Test and Performance.


TWiST-Plus is all extra material, and as such is not hosted behind STP’s site model. There is no limitation to accessing TWiST-Plus material, just click the link to download and listen.


Again, my thanks to STP for hosting the podcasts and storing the archive. We hope you enjoy listening to them as much as we enjoy making them :).

PODCAST FRIDAY: TWiST #47 w/ Randy Rice

So I'm excited about this week's interviewee.

I can't say for sure, but I think Randy Rice may be the first tester to produce a regular podcast. Over at TestingPodcast.com, the oldest entries are Randy's, so if there are older regularly created podcasts, then I don't know about them. Randy did several podcasts early on in an interview format with Mickey O'Neill that were influential in the way that TWiST is formatted today, so it felt really good to be doing an interview with one of my "testing heroes" and a true podcasting pioneer :).

Randy has been in the game awhile and has seen many changes in the industry over the years. I've also read his "Surviving the Top Ten Challenges of Software Testing" book (my review of which is here), have regularly visited his site over the years and read his various articles, and was really excited to hear about his new book, Testing Dirty Systems.

One of the really valuable pieces of this particular interview is Randy's experience and evaluation of literally hundreds of testing tools over the years. One of his presentations that he gives is on "Cheap and Free Test Tools", which he shares more than a few really good examples of here. Bur really, don't take my word for it, go and have a listen to Episode #47.

Standard disclaimer:


Each TWiST podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to the podcasts and the entire library whenever you want to :). In addition, Pro membership allows you to access and download to the entire archive of Software Test and Quality Assurance Magazine, and its issues under its former name, Software Test and Performance.


TWiST-Plus is all extra material, and as such is not hosted behind STP’s site model. There is no limitation to accessing TWiST-Plus material, just click the link to download and listen.


Again, my thanks to STP for hosting the podcasts and storing the archive. We hope you enjoy listening to them as much as we enjoy making them :

Twist #46 - with Steve Alexander

It's been a bounty of interviews the past couple of months, so much so that I'm starting to lose count how many are in the pipeline. I've set up a little section in Google Docs so that those of us who actively work with these files can keep track of the schedule and what's coming up, where and when. Seriously, it's a nice problem to have :).

This week’s interview comes to us courtesy of New Zealand and my fellow producer Farid Vaswani. Farid goes to great length to line up and talk to people "down under" that would be good potential candidates for interviews, and this week is no exception. Steve Alexander with Purple Shirt is our guest, and he discusses his specialty area, with is Usability and User Experience. More to the point, it's the intersection between what the user wants and what the organization needs, and trying to strike a balance between providing a product in a timely manner that's effective and providing a quality product that users want to use. If you’d like to hear the interview, then please listen to Episode #46.

Standard disclaimer:


Each TWiST podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to the podcasts and the entire library whenever you want to :). In addition, Pro membership allows you to access and download to the entire archive of Software Test and Quality Assurance Magazine, and its issues under its former name, Software Test and Performance.


TWiST-Plus is all extra material, and as such is not hosted behind STP’s site model. There is no limitation to accessing TWiST-Plus material, just click the link to download and listen.


Again, my thanks to STP for hosting the podcasts and storing the archive. We hope you enjoy listening to them as much as we enjoy making them :).

Podcast Friday: TWiST #45 w/ Grand Rapids Testers and Selenium Conference

I've been a little quieter this week than usual. the reason is because I've spent some quality time with Audacity, and you will be getting not just one, but *FIVE* podcasts from me this week.


The first, of course, is the TWiST podcast. This week we have Episode #45, and as a change of pace, we decided to record the Grand Rapids Testers Group as they discussed the ideas related to "Is Complete Testing Possible?" This is more than just a regular TWiST podcast. This is an experiment we are undertaking with the Association for Software Testing in which we use some podcasts that specifically relate to their material in their Black Box Software Testing courses. This is the first one that we have dedicated to a specific topic for that purpose.  We are looking to do more in the upcoming months.


In addition to TWiST, I also recorded about 30 hours of audio from the Selenium Conference that was held in SanFrancisco April 4-6, 2011. I have a number of pieces that I'll be presenting, but I will start out with the Keynote presentations first (Jason Huggins, Patrick Lightbody, Bret Pettichord and Simon Stewart).


All of the podcasts that I have presented (or at least an index for them) are listed on this site's Podcasts page. The Selenium Conference podcasts are currently being hosted on a temporary site, and will be moved to a permanent location as soon as we determine where that will be and when we can move them over. Like all material on TESTHEAD, these podcasts have been produced under Creative Commons copyright. If you want to embed them, comment on them, review them, spread them around, etc. go ahead, but please preserve the CC license details at the end (I'm not asking for money, I just want credit for the things I produce :) ).

TWiST #44 with Stephen Copp

After my time in the game industry, I always wondered how other game companies handled their testing needs. Much of the requirements for testing the games at Konami were live action and person intensive processes. Much off our testing was done with consoles attached to VCR's that captured all of our play times and we had a library of what was recorded to review later. NOw that nearly a decade has come and gone since those days, I'm curious to see what the process is like today.

Thus it was definitely cool to work on this week's interview with Stephen Copp. Stephen is a QA Architect with Electronic Arts, and he shared some details about what he does and how he approaches testing within a multi-located organization (not to mention multi-national). Stephen also has the benefit that he's able to work remotely from Colorado (And for those wondering how he landed a gig like this, he talks a bit about that, too :) ). But really, enough of me talking about it go to SoftwareTestPro and listen to Episode #44 for yourself.

Standard disclaimer:


Each TWiST podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to the podcasts and the entire library whenever you want to :). In addition, Pro membership allows you to access and download to the entire archive of Software Test and Quality Assurance Magazine, and its issues under its former name, Software Test and Performance.


TWiST-Plus is all extra material, and as such is not hosted behind STP’s site model. There is no limitation to accessing TWiST-Plus material, just click the link to download and listen.


Again, my thanks to STP for hosting the podcasts and storing the archive. We hope you enjoy listening to them as much as we enjoy making them :).