BOOK CLUB: How to Reduce the Cost of Software Testing (20/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry. Ths entry covers Appendix C.

Appendix C: Rapid Test Augmentation by Jon Bach

Jon starts this chapter with the analogy that we outsource things in our lives all the time. When we call a plumber or go to a restaurant for dinner, we are actively “outsourcing”. Why do we do it? There’s usually two big reasons (though there may be several others). First, we need someone to do a job that we can’t easily do. Second, we get someone to do something that we could do, but they can do it much faster. So why do so many have a visceral reaction when we hear the word “outsourcing” in testing? It’s because we often equate the word with “offshoring”; using a different country’s labor (usually overseas from the United States) to do work we either won’t or can’t do without great expense. Man, the arguments I’ve had over the years regarding that topic could take up several posts, and really, the question is “is the argument even fair?”. We have no idea if offshoring our outsourcing is really lowering costs and raising value… or do we? Jon seems to think he does. shall we find out :)?

Prior to working for eBay, Jon worked for QuarDev, which meant he was a manager for hire. Why would you call Jon? Probably because you need something done quickly, with a solid expertise, and a budget to allow you to do that in a way your current team couldn’t at that immediate time (not that they were not capable, but they just couldn’t do what was needed in that sphere at that time under the circumstances that necessitated hiring Jon, right?). One of the things Jon is famous for doing is called Rapid Testing (his brother James Bach gets the credit for inventing the approach, along with active help from tester like Michael Bolton and Cem Kaner. Rapid testing is the “skill of testing any software, any time, under any conditions, such that your work stands up to scrutiny.” (James Bach, Satisfice)

Note, Jon’s team is not cheap. Some offshore serices can do “the job” for 5 times less. The real question though, and Jon is willing to stake his reputation on this, is that they can’t, not with the level of ability he an his team can do. Their philosophy is to make the sales conversation about value, not price.

As we’ve seen in this book, cost is entirely context based. One context doesn’t hold up to another. In this case, Jon is comparing labor costs and the fallacy that all testers are created equal, and there’s really little skill involved and one tester is synonymous with another (editors note; hang out with Jon for about an hour and test with him. I promise you will see the interchangeable testers fallacy is exactly that!).

An interesting development was taking place before the downturn in 2008 called “re-shoring”; projects that went overseas because time differences, language barriers, skills levels and other differences were significant enough that the cost alone was no longer the deciding factor. Many times, it made more sense for a value perspective to keep the work local. Sadly with the downturn, the flow seems to be going the opposite direction again.

To be a consultant means to go beyond the stereotypes and the bad jokes everyone has about consultants (jokes that tend to have a sting of truth to them). Jon’s strategy is to:

1. Charge a fair price.
2. Have some ideas about how to execute what you recommend.
3. Write reports that tell the story you need to tell.
4. Write simply.
5. Tell them what you found out and call them "findings."
6. Talk about what you hope to do next or how you see yourself being involved.
7. Discuss your code of ethics.

Questions to ask clients to help them get the best value out of the engagement:

1. "What's important to you?"
2. "What does success look like?"
3. "What's the worst that could happen?"
4. "What would you like to do less of?"
5. "What would you like to do more of?"
6. "What do you wish were different?"
7. "What problems are you trying to solve?"
8. "What's working?"
9. "What would you start doing, stop doing or keep doing?"

Jon likes to speak to the very top level of management, but also likes to spend time side-by-side with the testers to see how testing actually gets done. A lot is learned at both ends as ell as in between. Does the company value "heroism," and does that mean they they *rely* on heroes to solve major problems every time? Are there processes in place that prevent the heroes from becoming casualties? If so, are they actually applied? There’s a happy medium between heroism that’s excessive, and process that’s stifling.

Rather than polishing up a PowerPoint presentation, take out a sheet of paper during the pitch and ask: "Can we work right now on one of the problems you're having?" The way you probe for values - thoughtfulness, enthusiasm, and an earnest interest in the work – will help lead you to find the best fit for who you partner with.

Some Sample Questions FROM Clients 
Your bug database or mine?
Can I get the same tester as before?
How do you train?
Can I get resumes from your staff?
To what associations do you belong?
Can I see the templates you’ll use?
Can I customize the status reports you give?
If I need to postpone or cancel, what’s the penalty?
Can I talk to a tester in the lab?
What are your working hours?
Will you work overtime or weekends?
What’s your hiring process?
Will we be billed for the hours we don’t use?
How do you measure test coverage?
Why didn’t you catch that bug?

Some Sample Questions FOR Clients 
Your bug database or ours?
Can I talk directly to a developer?
What are your working hours?
Do you work overtime or weekends?
What’s your triage process?
When do you plan to ship?
Will we work onsite or here in our lab?
Do you use any existing tools that would be of help?
Can we see your existing bug database?
Did you want to devote time to regressions?
How often will you be giving us builds?
What are the minimum hardware requirements?
What kinds of users is this targeted for?
Has this been tested before?
Would you be a reference?

The “cost” of something isn’t just about price. It’s all of the context and values and weighting and considerations that go into the computation. Think of the average ink jet printer. They’re actually really inexpensive… the printer’s themselves, that is. The ink cartridges, though, can cost up to half the price of the printer!

List all of the things that matter to you – all of your hopes and ideals and values. Put them into a list and give a “gut-feeling” ranking to each of them on a scale of 1 to 10. Share it with the people on your team who are charged with making a decision. Add, delete, modify. But also stay alert to new context that emerges once the decision is made.

To know “the cost of testing”, you have to know the context of testing, the value of testing, and the questions of testing. You can buy a cheap car for $500 and maybe it’s just what you need. You can buy a new car for $50,000 and maybe it “pays for itself” in the benefits you get from it.

For the question of hiring a test lab to do rapid test augmentation, you may be able to reframe conversation to your stakeholders and decision-makers – helping them realize that you’re not hiring them to find bugs, you’re hiring them to assist you in gaining more visibility about the value you are offering to your customers.

The Human Test Case, Revisited

So I first want to say that I had a great weekend talking with and collaborating with the group of testers and participants that are part of the Association for Software Testing's Board of Directors. They are a committed group of people, and I am happy to be counted among them. I also see that we really have our work cut out for us, but again, that's not really the point of this.

As I got to the airport, I thought "Hmmm, there was a bunch of weirdness when I took off the boot cast after going through the gate in San Francisco... I'll just make it simple this time and take it of before I go through". Which I did. I ran my bag and all of my systems through (they actually commented in Madison that it was unusual to see someone traveling with two laptops... I really am starting to think I'm somewhat weird for doing that, maybe it is time to consolidate to one, but anyway). As I hobbled up to the metal detector, I walked through... and nothing. Apparently, that chunk of titanium in me is not significant enough to set of the detector. The boot cast? Definitely! Maybe they know this, maybe they don't. Maybe their scanners are set so that they don't ring on titanium. I honestly don't know, but this time I did not get the special advanced screening, I just put on my cast and walked away.

Isn't life and the things you notice around you so much more entertaining when you are a tester (LOL!)?

BOOK CLUB: How to Reduce the Cost of Software Testing (19/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry. This entry covers Appendix B.

Appendix B - 25 Tips to Reduce Testing Cost Today by Catherine Powell

OK, wait a second... didn't I review a batch of "25 tips" yesterday? What gives?! Well, Appendix B is a strikingly similar set of 25 tips. In fact, they have exactly the same titles. The difference? Many of Matt's tips had to do with things that cut waste, but were organizational, and possibly out of the realm of the Individual Contributor Tester (i.e. the Tester doesn't have the authority to do many of those things). In other words, if I were an individual Tester, what could I do to lower costs where I don't have to ask anyone's permission to make an impact? That's where this "list of 25" comes in :). Look at them from the perspective of how any Tester can put these into play and make them effective. Also, don't try to do all of these things at one, do one at a time, practice it, master it, then move on to something else. Think of these like a pebble going into a jar. It may not seem like very much, but with enough pebbles, you'll ultimately fill the jar ;).

Tip 1: Cut your documentation to a minimum

Let's do what we can to make our documentation more DRY ("don't repeat yourself"). Have you been copying and pasting things into your docs? Stop that. "Incorporate by reference". Replace those copies with a simple note: "See XX", and then put it - just once - at that location.

Tip 2: Make the cost of changing the documentation cheap

Source control, wikis, whiteboards, they all allow us to share documentation and change it quickly and from multiple sources. Leverage that. Make internal reporting as quick as possible, and let the QA Manager or lead handle the more formal report (which should also be easier to do with this collaborative apprach). Start with your own team, then once you've shown the benefit by collaborating on the documentation within your team, share it with other teams and encourage them to follow suit.

Tip 3: Never be blocked

Make a wish list of the things you wish you could do but you don't have time for on your whiteboard, a sticky note, or in your Smartphone. Spend just ten minutes on this, you don't want a massive list. Then put the list aside. Next time you're blocked, pull out your list. Now's the time to do something on it!

Tip 4: Eliminate Multi-Project-ing

Individual Testers may not be able to choose if they work on multiple projects or not, but you can choose how you deal with working on multiple projects. Pick a single project and create a meeting in your calendar for you and that project - and spend that block of time working on that project only. The more often you can do this, the better. Minimize task switching. Turn off email, IM, phone, put a “do not disturb” sign on your chair while you are testing. Whatever it takes :).

Tip 5: Automate entirely redundant processes

Automation freaks some people out, because it sounds so huge and nebulous. Try some "computer aided testing" instead. Make a shell script or a batch file. Find a repetitive task that you've done at least twice in the last month, and create some "computer aided testing". Then check into the source repository so everyone can use it. Start small, get some ideas together, and build it out as you do more of it. Even a list of commonly used commands in a single file can be a huge boost.

Tip 6: Start with Soap Opera Tests

Soap opera tests are where you throw everything, including the kitchen sink, into your tests. Load them suckers up and let 'em run. You might find some spetcular issues :).

Tip 7: Start with quick attacks

Quick attacks are finding ways to do something hairy and over the top to inputs or outputs. Search for eveything. Load Hamlet into a text field (thank you, QAHatesYou, for my ultimate favorite Quick Attack ever :) ). If you feel fancy and want to go a little white box, then find every error message that a program is supposed to display, and force the program to show it. That's one of my all time favorite tests; lots of time is given to the happy path. Have they spent nearly as much time and effort on error handling? This is a great way to find out :). You get the idea.

Tip 8: Test Early

If you're in an agile environment, go through your last iteration and identify how long it took you to start testing each story after it was done, and come up with one idea to start each one faster. If you're in a more traditional environment, go find your friendly local developer and offer to spend half an hour working with him to "pre-test" his feature together. If the developer looks at you funny, offer to bring donuts (or beer, some people are just motivated differently ;) ).

Tip 9: Develop a regression-test cadence, and keep it light

Creating a "testing-oriented system diagram" so you can figure out what components relate to what other components. Use it to identify what needs to be regression tested and what you can more safely not run. Ask your team to expand on the drawing, discussing what fixes or feature changes might break other things. Have an architect validate your diagram. Again, if necessary, bribe with donuts or beer ;) ).

Tip 10: Test the things that actually yield bugs

Pick one area that is on the top of your "must regression test" list, and one area that is on your "we don't have to regression test" list. Then do it, with just one area and with just one release. Prove to yourself that it's okay to not regression test *everything* every time. Focus your testing efforts on areas that are at higher risk of having new problems.

Tip 11: Elaborate - and communicate - a test strategy or triage strategy

Create a status update that you can publish frequently. Publish it at least twice a week for the rest of the release. If your release is in under a month, publish the status update daily. Be sure to include the "must fix bugs" and a summary of new bugs found.

Tip 12: Decrease your time spent reproducing the problem

Next time you find an unreproducible problem, set a timer (an hour or two max). Now attempt to reproduce the problem and gather information. When time is up, send the unreproducible bug over to development with two or three ideas for extra logging or information that will help track down the problem if it happens again.

Tip 13: Do a low-cost beta program or pre-release

First, do a beta test of one. Get an account rep to bring a friendly customer to your offices. Let them use the newest software for an hour or so. Sit next to them and watch what they does and what questions they ask. Have a frank conversation afterward (thoughts, likes, dislikes, what's missing, etc.). Next, do a ride-along for half a day or a full day w/ an account rep to a friendly customers site, and shadow the customer(s) for half a day or a day. Keep quiet and take notes about how the customers actually uses the software. Share this information with your team. Learn from your actual end users. If the customer is open to it, record them with your phone, or at least take pictures. Pictures are worth a thousand words.

Tip 14: Recognize and eliminate project wankery

This might be hard, but give it a try anyway. Think of the meeting that you feel is the biggest waste of your time. Then make plans to skip it. Provide whatever informstion you need to satisfy those who feel you miust be there, but explain you can't make it for a pressing reason (no need to elaborate). The goal is to prove the necessary work *can* get done and that the product won't suffer if you don't sit in this meeting Swapping a meeting for a brief written report is a good bargain in service to your overall goal. You may not be able to do this very often or for many other meetings, but even if you can get out of just one of them on a regular basis, over time that can add up to a lot of quality testing time.

Tip 15: Stop fighting to manage the project

#1 Rule: NO WHINING. If you don't want to manage a project, don't complain about how it's being managed. It's probably not part of your job. Don't let the things other people choose to do interfere with the things that only you can do. Let it go.

Tip 16: If you're going to fight, win

Sometimes the decision really does matter! If it's truly important, you need to be able to sway the decision your way. Your reputation, your argument, and how much support you’ve garnered to your side will have a huge impact on the outcome. The next time you make a decision, sit down and write out why you're making that decision, along with the pros and cons of the choice you are making. Then give that explanation to your boss and ask for feedback. Make clear this is *your* decision, you're not asking your boss for permission. This lets you see if you argument is persuasive and if others will back you up on your decisions, and if you are really ready to "fight to win".

Tip 17: Walk around and listen

If you're a test lead or a test manager, take fifteen minutes and walk around the office in the morning. Repeat your 15 minute walking circuit that afternoon. Don't interrupt anyone, and don't talk with them. (It's okay to say "hi" back if someone greets you.) Just watch and listen, and notice who doesn't seem to be doing anything either time you walk by. After your circuit, sit down for 10 minutes with everyone who was not accomplishing anything and ask what you can do to unblock them or help them find a project to do.

If you're not a test lead or manager, see what you are doing with your time... no really, what you are really doing with your time? You may find that you are not nearly as focused as you think you are. Be brutally honest with yourself, and ask, OK, what can I eliminate? What can I minimize, and how can I get the necessary project time beefed up and finish what needs to get finished? Aim to decrease the distractions each day and boost the project stuff each day until you get to a realistic level of external tasks that have to be done, and as much project focus as you can reasonably perform. Seriously, nobody's going to be able to be 100% during an 8 hour day, but try to maximize where you can.

Tip 18: Write test automation that attacks the business-logic level

If you are using GUI tools and automating through the GUI, detemine how you might use another tool or rewrite your test to do a similar job from under the board. This might take some coordination with the development team but it certanly can be done. Determine what internal tools they use to get to the console or other hooks in the program, and then start working out tests that utilize them.

Tip 19: Develop a test automation library

This may be tougher for non coders in the test group, but think of the scripts and the automation you may already have. How much of it is duplicated in other tests? If you see a bunch of duplication, here's a chance to make a library or somehow centralize those items so that the can be called in a simpler manner. Personal experience: a neat trick in Cucumber is to take commands that get run a bunch of times and make "macros' of them by using the %Q|[cucumber line]| syntax. This lets you group various Cucumber commands into a one line call. DRY in action :).

Tip 20: Develop or hire expertise

Spend 15 minutes and figure out the expertise of each member of your team. Then take another 15 minutes to figure out where you need experts that you don't have. These are the areas you should be considering for training or hiring: to fill your expert gaps.

Tip 21: Get a return from conferences or don't go

Play "find an expert" the next time you go to a conference. Find someone with experience in an area your team lacks. Follow up with the expert - a quick email is fine - to establish a relationship. When you have a question about [fill in the blank expertise], you will be able to reach out to the expert. They don't have to be the famous testers. Often the ones that aren't are totally down with helping you and will often bend over backwards to help you get where you want to be. Note: don't abuse this relationship. They won't do your work for you, but if you do your homework and come to them with specific questions and showing you've already given your all, they may prove to be surprisingly helpful :).

Tip 22: Build a social network

Pick your favorite testing site (Software Testing Club, StackExchange, whatever) and answer one question. Next time you have a question, go to the same site and ask it. Do this regularly (maybe once a week), Over time, you will build a network of testers you can rely on. You'll also build a reputation and have public evidence of your own competence, which can be huge when it comes time to look for a new job :).

Tip 23: Examine invalid bugs carefully

Do some data mining in the defect tracker. Create two reports: (1) number of bugs marked invalid by reporter; and (2) number of bugs marked invalid by feature/module. Take that report and go talk to the reporter with the most invalid bugs. Pick a handful to walk through with him and spend 30 minutes brainstorming how to decrease that rate. Then take the report to the owner of the feature/module with the most invalid bugs. Pick a handful to walk through with him and spend 30 minutes brainstorming how to increase team-wide knowledge of his module/feature so the number of invalid bugs decreases [I'm not going to paraphrase this one, I think it's perfect as is :) ].

Tip 24: Build a test model that deal with the natural up and down staffing need for test resources

If you're already a tester on a team, spend 15 minutes brainstorming "what do you do when you're idle". A list of things testers can do when testing on the product is relatively light should be the result. Make a list of concrete actions you can perform. Next time you're idle, grab something from the list, and do it.

Tip 25: Ask the team to identify opportunities to drive out waste ... and implement them

Is there something in your processes that seems totally stupid and a complete waste of time? Kill it! Are there documentation steps that no one ever reads or does? Remove them. Tests that add little or no value? Weed them out. This is a gift that keeps on giving. As soon as you get good at this, you will find other areas you can trim away, too, and the better you get at it, the more areas you will discover. This is also a cycling list. As you cut wasteful activities, you will find room for more fruitful ones, and that fine, but remember, no fruitful idea will remain 100% fruitful forever. New processes develop waste of their own accord, and soon you'll be fining ways to "refactor" there, too.

So You Wanna' Be a "Rock Star"?!

There's a phrase that those of us who hang out on twitter, or go to conferences, or participate in forums see a lot, and many of us aspire to be it. I'm talking about being a Rock Star Tester. Oh, come on, admit it, I wanna' be a rock star tester! In fact, part of my wanting to be a rock star tester comes from my earlier life when I wanted to be a rock star in music. Why am I bringing that up? Because I think the two have a lot in common.

First, ask yourself, why do you want to be a rock star? Let's take the musical example. There's lots of reasons. Maybe you enjoy songwriting, and want to have lots of people hear your material. Maybe your a performer who loves being on stage and you want to perform in front of hundreds or thousands (or tens of thousands) of people. Maybe you want to be in magazines. Maybe you want to have people recognize you and say "hey, that's [fill in the blank]". Maybe you want to have the recognition of your peers and be seen as being part of a movement. Maybe you want to travel the world and have an impact on the lives of a lot of people.  Maybe you want to party like a maniac.  All of those are valid reasons, an they are why lots of kids all over the world strap on guitars or basses, hit drums, play keyboards or grab microphones in the hope that they will one day hit the big time and live their dream.

In reality, becoming a rock star is not easy. A lot of it is tremendously hard work over a period of many years. It's hit and miss. It has a lot to do with who you know and how you can interact with them. It has a lot to do with the circles you run around in. It has a lot to do with how much of your life you are willing to put on hold so you can live that dream. Oh, and generally speaking, you have to be really good at what you do, as in orders of magnitude better that the average guy who plays an instrument (there are exceptions, some can point to the punk rock movement as a goal for a more everyman type of music, but even that morphed into groups that could play together tightly and preform sets regularly, which required a level of skill beyond just the basics). And even with all that, there's no guarantee you will be able to light it up and go all the way to the top. Luck, timing, and a fickle marketplace have a lot to do with that, and sometimes an act that has all of the pieces needed for superstardom just never gets that shot, while bands that make you scratch your head and ask "huh?" seem to catapult to the top of the charts.

A local band in the Bay Area that we knew called T-Ride (don't worry if you've never heard of them, if you weren't an adherent of the early 90's scene, you probably wouldn't know of them, but if you do know who they are, props to ya' :) ), said something in an interview that I thought was great, and offered some great advice to anyone looking to become performers. Not necessarily rock stars, but solid performers. I'm paraphrasing here, it's been two decades since I read it:

"Bands go through lots of things to try to be successful. They try to find the right agent, they get stylists to make sure they look the part, they hire photographers to make them look great, they go out and promote their shows like mad, they work to get the right producer so they can get the right sound, but all of that doesn't matter if you don't spent the time to get good! We spent four years in our rehearsal room practicing week after week, learning from each other, practicing playing together, getting our ability to work together as tight as posible. All those other things matter to a point, but first, you've got to get good!"


Testers, the same goes for us. We need to get good at what we do, and that needs to be the first thing we focus on. Like songwriters who want to become rock stars, just as they need to learn how to write songs that have impact, we need to learn how to do testing that has impact. Only then will be have the chops necessary to start exploring the avenues that can get us out there. The cool thing? It's a lot easier to become a rock star tester than it is a rock star musician! There's very little that involves luck. There's a lot that involves skill. Develop solid skills, and then develop a desire to share those skills with others. Really, there's no reason to be a rock star if you have no songs to play to people; that's the reason they buy your music and pay to see you perform. Likewise, develop a testing ability that makes people stand up and take notice, so that they want to know what you know.

From there, you can start writing about what you now (and sometimes about what you don't) under the right circumstances. A blog is the perfect rehearsal studio for this. Your costs are very low, and the changes of messing up big time are also really low (unless you are going to plagiarize other people's stuff. Then you can mess up right out fo the gate). Generally, you can develop a following, get feedback, and perfect your writing and skill development in a medium that people are interested in. Grow it from there. Other opportunities to write, present and spread your message come from there, and with dedication and effort, we can all grow and develop to the point of eventually being  Rock Star Testers.

BOOK CLUB: How to Reduce the Cost of Software Testing (18/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry.

Afterward

The editors of the book give us here a charge by way of a description of a game manual set up with a glossary. The point made is the last entry doesn't have a , it ends with Y, in fact, it ends with the word "You".

You: Have reached the end of this manual. We've taken you as far as we can; the rest is up to you.
Not get out there and play!

This is the official end of the book, except that it isn't. Over the next few days I'll be covering Appendix material that likewise deserve their own entries (see below), and of course, the conversation continues with the writers of the book. Most of us are very active on our own blogs, on Twitter and other mediums, such as the Software Testing or SW-IMPROVE discussion lists -- and on LinkedIn. If you try the ideas in this book, please let us know what you think. The important thing is that it's now time to put the ideas to work, so get out there and test!

Appendix A: Immediate Strategies to Reduce Test Cost by Matthew Heusser

So the book is finished, yet you may be thinking to yourself "OK, that's all great, but I need stuff I can do "right now" that can provide immediate payback. do you have any suggestions for that?"

Matt offers the following:

While testing less is an easy cost cut, it also introduces risk. The best way to help reduce the cost in an immediate way is to reduce or eliminate waste. The following strategies may help that, as well as provide additional value added benefits from trying a few things (25 of them, in fact):

1) Cut your documentation to a minimum
First you have to write it, then maintain it. Plus it only has value when people read it.
Instead of a comprehensive breakdown, focus on removing details that are not important to the testing effort. Keep those that would need to be known if you needed to hand it off to someone so you could leave town for a week. Is there enough there that the tester can read it and to the job? If so, aim for just enough documentation to do the job, but not so much that it becomes a time drag.

2) Make the cost of changing the documentation cheap
Perhaps use a shared doc, or on a network drive,  put it in a wiki, or hey, why not just keep it on a whiteboard?

3) Never be blocked
If a tester feels they are blocked or they can't do anything, they are generating waste. Instead, ask the following:

(a) Can I interview customers for test ideas
(b) Can I pair with the developers to learn the system
(c) Can I pair with another tester on a different piece of functionality in the same project

4) Eliminate Multi-Project-ing
Focus on a task at hand, and if you are blocked on it, see #3. Taking on side-projects and having to frequently context switch takes a lot of time. Focus on one area until you are done, then move on to something else if necessary.

5) Automate entirely redundant processes
While automation is sold as a panacea to all testing troubles, we know it's not. There's much more a real thinking brain can do at interesting points in the software that automation can never do. However, there's a lot of steps that are horribly repetitive and don't add any real value to the testing knowledge (set-up, navigation to key places, etc.). Absolutely automate these sets if you can.

6) Start with Soap Opera tests
Instead of testing for individual factors in a test, start out with testing the largest group possible, and if you see errors, whittle it down instead of testing each factor in isolation (which will take much longer).

7) Start with quick attacks
"Quick Attacks"are all about overwhelming the software with invalid input, too much input, and input that is out of range. If the software handles the obvious exception conditions well, it is likely in good shape. If the developers left holes and errors in the exception conditions, they likely left holes and exceptions in the main business logic, too. Quick attacks allows the tester to find bugs early, learn business rules, and perform a quick assessment of the software -- all at the same time!

8) Test Early
If you are use to working with software where an entire build is ship to you to test a new feature, see if there is a way you can get access to an environment where the feature is added and you can test it without having to wait for the entire build. The sooner you can test a feature in isolation, the sooner you can provide meaningful feedback the developer.

9) Develop a regression-test cadence, and keep it light
The days of shipping software once a year are disappearing fast. Most products are moving to incremental updates over the web, or much more frequent updates if its a web property entirely.

Regression testing, therefore, needs to be modified so that we don't have to run a years worth of test changes in a week,  or less. While there is a chance that any change could cause a problem anywhere else in the code, this is not universally true. By compartmentalizing tests that actually have a relation to other functionality, the regression tests can be made more efficient and focus on the areas where testing will actually yield results, which leads to...

10) Test the things that actually yield bugs
If you have regression tests you run for every release that never seem to find bugs, find a way to limit how many of those run. Maybe run them every third release, or one third of them per release, or just enough of them that would find a major defect if it were introduced in that code.

11) Elaborate - and communicate - a test strategy or triage strategy
List features by how critical they are, then go down the feature list in order for a "first pass". If you complete that "first pass", go in more depth. Make sure everyone knows how the decision was made, what the decision was - and the decision-makers feel like part of it.

12) Decrease your time spent reproducing the problem
If your team is spending a lot of time on bug reproduction, look for ways to lower it. Have the server store a log of every action. Find a tool to record exactly what the tester is doing and make a screen-cast that can be played back.

13) Do a low-cost beta program or pre-release
Can you segment your users and send some to a managed beta? If so, you could release early and engage the users in helping to test. You'll need some infrastructure (which users get which builds), and on web-based projects, some sort of production monitoring.

14) Recognize and eliminate project wankery
Getting better requirements, having more time to test, doing architecture and code reviews - those can all be good things. They can waste a lot of time, too. Experiment with these practices, but view them as experiments. After you've tried them once or twice (or if they are currently "mandated") ask if there will be less defects down the line, do we add value to the project, do we decrease overall cost by doing these things? If the answer is no, change the format, or drop them.

15) Stop fighting to manage the project
Instead of arguing over if the software is ready to release, make the defects visible to everyone and let senior management decide if it should be shipped. Imagine saying something like this:
"We're going to stop fighting you over issues a, b, and c. We yield to a, b, and c. We're going to focus on testing. If a, b, or c fail, don't complain to me. The decision is yours." Consider how liberating that might be.

16) If you're going to fight, win
If you do give on some issues, you may just want to pick a few battles. If you pick those battles, fight to win. Otherwise, you're just wasting time.

17) Walk around and listen
It's s good bet that there's a lot of other stuff going on that may look like work, but may just be busy-work leading to little value. How can you tell if that's happening? Walk around and listen. See what the testers are actually doing. They may think they are focusing on something important, or they may be "blocked" and not know how to proceed, but are too embarrassed to say anything. By doing this, you can then help coach or guide the testers to getting out of the rut, or determine what the blocker actually is. Is it their process? is it another person on the team? Are they just "socially loafing"? Don't wonder, find out :).


18) Write test automation that attacks the business-logic level
GUI testing is very expensive, and often GUI testing is really "change detection" testing, and can be very brittle if UI elements change. With setups and scaffolding and test hooks, testers could write tests that exercise business-logic level functions. These tests will general run much faster and be less brittle than GUI tests. Automated tests verify those dozens (hundreds? thousands?) of possible inputs for a given function. Test automation is an investment -- in the short run, testing costs always go up. Starting at the business-logic level might make it possible to see returns in days and weeks, not months or years.


19) Develop a test automation library
If you do want to test at the graphical level, or want to have more powerful business-logic tests, you may want to develop re-usable functions. For a GUI, that might be login (taking to parameters and pushing the login button), search, tag, etc. At the business-logic level these will probably be object-oriented functions.

20) Develop or hire expertise
As a test manager, you can foster expertise with brown-bags and pairing; when you look to expand your team, look for skills that would round out the same. Help develop your team so that there is redundant expertise where possible. Avoid the information silo.

21) Get a return from conferences or don't go
Expect employees to write two 'what I learned' documents: Things for us to do on Monday, and strategies to pursue over the next year. Use conferences to attract talent. Send your employees with business cards and a list of open positions. Use conferences to retain talent. Build the conference into each employees annual professional development goals, and he'll be more likely to stick around to attend it. Use conferences to grow your network of friends with specific expertise. If your staff comes back with ideas to implement on Monday, let them actually try them out!

22) Build a social network
Getting involved in local user's groups, attending conference, collaboration over the internet, all of these can provide an increasing list of people with skills you do not have, people willing, even eager to share ideas if you tap them on the shoulder. Instead of being "blocked" by a performance testing problem for a month and calling a consultant, a few questions to the social network, some calls, some questions, and a couple of days of exploration might solve the problem directly.

23) Examine invalid bugs carefully
How many bugs are being marked invalid, or 'wrong' for various reasons? Each of those means a tester and a developer both invested time for no benefit. If enough bugs are invalid, take a look at the reasons why and try to prevent it in the future.

24) Build a test model that deal with the natural up and down staffing need for test resources
Test projects have an ebb and flow. If testers remain constant, you'll either be stuck with too small a team or too big. Build a staffing model that allows you to scale up and down quickly. Perhaps bring in tech support to test, hold a beta program, bring in an outsourcer, or consider crowdsourcing like  uTest or volunteer a project to Weekend Testers.

And finally ...

25) Ask the team to identify opportunities to drive out waste ... and implement them.
A good tester is inquisitive, curious, and critical. The tester is likely to think "Why do we waste so much time doing X? I would think we could get by without it, or get the same benefit from doing Y."
So ask your testers what they would do to eliminate waste.

What If...: New E-Book On Software Testing

This will be a short entry, but I hope a worthwhile one.

Ajay Balamurugadas, founder of Weekend Testing, active proponent of context-driven testing in India, and an all around great guy (I've met him, I can say that :) ) has written an E-Book about some of the things he has learned in 5 years of active and passionate testing. The E-Book is called 'What if..." and more about it can be seen at this link.

Many of us have great ideas but we have a limited reach in how to get that information out to people who are interested. We have our individual blogs, but I think it's great that w have these additional ways to communicate our ideas and our passions to others, especially in situations where we can earn a little return for our investment of time and energy. For those who have read Ajay's blog, you already know how much he loves the topic of testing. Help support testers writing more and getting their ideas out there. Who knows, maybe some larger publishers will take notice and help more testers spread their messages. You never know :).

BOOK CLUB: How to Reduce the Cost of Software Testing (17/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry.

We are now into Section 3, which is sub-titled "How Do We Do It?". As you might guess, the book's topic mix makes a change yet again. We have defined the problem. We've discussed what we can do about it. Now let's get into the nuts and bolts of things we can do, here and now. This part covers Chapter 16.

Chapter 16: Rightsizing the Cost of Testing: Tips for Executives by Scott Barber

Scott makes the point early on that, unlike the other chapters in the “How Do We do It?” section, he’s not specifically writing to testers. He’s aiming for The CEO, CIO, or CFO and other senior execs that pull the trigger on decisions that they may really not understand. Even with that, the message here is still valid to front line testers. You may find yourself having a conversation with a senior exec, and you may be able to bend their ear and discuss this with them. Also, some day, you might be a high level executive, faced with the same issues. Wouldn’t it be cool to already know many of these details so you don’t make the same mistakes so many organizations do? Forewarned is forearmed.

For those who want to desperately believe otherwise, unless you sell testing services, there is no revenue in testing. There isn’t. It’s an operating expense like so many others, but it’s one that has an unfortunate detail going against it. Unlike some expenses that are quick to notice when they are changed, it may take several testing cycles to notice what happens if you cut testing from the budget. It may be months before the support calls increase, or before the cancellation notices start coming in, but in more circumstances than not, cutting testing is a case of “penny wise, pound foolish” and the result is usually a back tracking to real testing, and an even greater expense, until things normalize again. Testing has a disadvantage in the sense that, to most senior executives, unless they have spent time in the customer support area or in software development, testing is completely of their radar. We often joke that there are two experiences that come with being a tester. Either you are grudgingly appreciated because you find issues, or you are ignored when you don’t (even when the code is well developed and there’s little to really make headlines discovering. We are not celebrated when things are going good. We’re really only visible when things aren’t. Not to mention we’re often seen as the bearer of bad news, and really, who wants to hear that?

For me personally, I’ve been lucky in that many of the companies I have worked have been small enough where I had easy access to senior executives. I’ve also worked with companies that were significantly larger and where they indeed didn’t seem to have any idea what their testing organization was doing, and made decisions that seemed logical at the time but came back to bite them later on. If you are the executive reading this, perhaps a change in approach can help.

Executive Tip #1: Change the Question

Rather than asking “What value are we getting for what we are spending on testing?” instead ask “What value do we want to achieve through testing and how much are we willing to invest to see it actually happen??” Trying to calculate the ROI on testing is a lot like trying to figure out if paying for auto insurance is worth it. It may not be for years and years, but al it takes is one big wreck without insurance and the numbers change immediately! Instead of waiting for a “testing actuarial table” to appear, identify the areas you want to see your business increase value related to testing, prioritizing the areas and then determine an acceptable cost to meet those goals.

Executive Tip #2: Focus on Value to the Business

Most of the time, testing has been defined with the value to the project, rather than the value to the entire organization. How do we do that? We help the company pass regulatory audits, if they are relevant. We can provide defense against clams of negligence, faulty or bad advertising, or perceived breaks in service agreements. In short, good testing can help us not get sued (never a guarantee, but the likelihood of getting sued goes way down with vigorous testing). Testing can help the support staff deal with issues quickly and effectively. Testers can help provide stability between releases. Testing can help identify risk and areas where rick can be mitigated. Testers also work well as product trainers, as they tend to know the product better than just about anyone not on the customer support desk (and yes, that often means developers, too) Testers find discrepancies between requirements and what is actually being delivered. Testers detect issues earlier in the product lifecycle (especially if they are discovered before customers see them). Testers free up time for developers to focus on new feature development. Testers act as the advocate for the end user regarding usability issues and prioritizing issues from a customer’s perspective. The likelihood is that, seen in this light, there should be plenty of value identified in testing efforts.

Executive Tip #3: Distribute Testing Costs Carefully

I’ll confess I’m not an accountant, so much of this is outside of my area of expertise to comment on. Suffice it to say that, by spreading the costs of testing around among an organizations divisions and having the divisions be financially held accountable for the areas that they are responsible for, then testing costs can be placed in the areas that make the most sense for the organization and not impact other cost centers needlessly, and at the same time, each area can provide the support and financial backing necessary for testing to be effective across all divisions.

Executive Tip #4: Demand Accountability from Managers of Testing Programs

Executives have a need for information that they can quantify. Testing is often difficult to fit into that paradigm, but it can be done. Executives need to make clear what information they need, and why they need it. Testers need to make clear what the information they can provide actually means. Instead of asking for a particular measure or metric, executives that understand the value that testing can provide will focus on talking about, and trying to determine what value they are hoping to receive and by collaborating with testers and test managers, they can develop a metric or measurement that accurately represents that value. From there, the managers of the testing program (whoever they may be) need to know that they will be held accountable to that mutually determined metric, and that the testers know that they will need to modify their approach to ensure they can provide that metric.

Executive Tip #5: Keep Tactics at a Tactical Level

We’re all aware of the fact that we are often expected to do more with less, and how well we do more with that less determines if we get promoted or if we get sent packing. Scott points out that in his experience, more times than not, test programs are often cut by people who have little or no involvement or understanding of what the test team actually does. The only ones who can really determine what costs can be reduced without significantly degrading the value of the testing program is, well, the test team! How can they help this process? Well, the rest of the chapters in this book might certainly be a good place to start :).

Being a Human Test Case

As I write this, I am sitting in the waiting area at San Francisco International Airport. I'm waiting for my flight to be ready to board to go to Madison, Wisconsin, where I will be participating in the Association for Software Testings board meeting, and officially get sworn in as one of the Board of Directors and take on my role as the organizations Treasurer.

That's all well and good, but that's not the point of my post this afternoon. Instead, I wanted to share my experience of being a human test case. For sake of context for those who read this weeks or months later, I have a broken leg with a titanium plate helping hold it together. Thus I knew I'd be a piece of entertainment for the security detail today.

I arrived early enough so that I would have time to get through the line. Turns out that if you are gimpy like me, there's a special line to go through. Much faster, but that may also be because they know I'm a guaranteed fail. I offloaded all my stuff, emptied my pockets took of my shoe but left on my boot. Net result, no big surprise, I lit up the metal detector and went over to the special area to receive an advanced screening. While I was there, I started taking of my boot. The security detail looked at me like i was nuts. I was actually asked why I was taking it of, and I said, well, because I can, and I have a titanium plate in my leg, and a simple wave of a wand will tell you that. I was being helpful... maybe too helpful (LOL!). In any event, I was screened much more thoroughly and then when everything was shown to be in order, they let me retrieve my things and go on my way.

Now here's the interesting thing... that portable metal detector they could have waved over my leg to verify the metal detector went off because of the titanium in my leg? They never checked. Part of me was walking around with a bit of smug superiority, thinking "silly security people, a wave of a wand over my leg would have shown what caused the alarm to go off", but as I was thinking that, I realized that, really, I may not even be thinking on the same wavelength as they are. People with medical conditions probably come through all of the time (hip replacements, knee replacements) with considerably more metal in their bodies than I have, so they might look at the gimpy man as being a Trojan Horse. Identify the obvious issue (the titanium in the leg) but somehow ignore something else (like a hypothetical switchblade or a pack of C4 inside of the boot). Note, I have no idea what their objectives were, and I didn't stop to ask (they were plenty busy), but it was amusing to contemplate.

As I'm often fond of saying, testing is all around us, and if your not careful... you just might miss it. Actually, I think that's a bastardization of something Ferris Bueller said, but still, it's apropos ;). And with that, I hope you have a Happy rest of the Friday afternoon and evening. I have a plane to catch. Here's hoping we talk soon :).

Updated: Landed in Denver with no issues, but really, did the connecting flight have to be on the total other side of the airport? Maybe it's karma for writing this earlier (LOL!).

BOOK CLUB: How to Reduce the Cost of Software Testing (16/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry.

We are now into Section 3, which is sub-titled "How Do We Do It?". As you might guess, the book's topic mix makes a change yet again. We have defined the problem. We've discussed what we can do about it. Now let's get into the nuts and bolts of things we can do, here and now. This part covers Chapter 15.

Chapter 15: Clean Test: Suggestions for Reducing Costs by Increasing Test Craftsmanship by Curtis Stuehrenberg

Curtis opens up this chapter with the idea that we can learn a thing or two (or more) from the Software Craftsmanship movement. Software craftsmanship is the idea of an experienced coder developing their skills and techniques over years of study and practice. Curtis expands this description by saying:

“A software craftsman is a skilled artisan apparently able to balance immediate pragmatism with a longer term focus on reducing the amount of work they or (more likely someone else) must do tomorrow. The craftsman coder knows a small investment in time today can save days or weeks later on, but more importantly they’ve developed a sense of which small investments will reap the greatest rewards.”

Software Craftsmanship emphasizes:

• Not only working software, but also well-crafted software
• Not only responding to change, but also steadily adding value
• Not only individuals and interactions, but also a community of professionals
• Not only customer collaboration, but also productive partnerships

That is, in pursuit of the items on the left we have found the items on the right to be indispensable.

Notice that the above highlighted areas are not specific to software developers. These ideas and ideals apply just as much to testers as they do to developers. Having to fix someone else’s code, or even your own code, over and over, prompted the inspiration and the growth for the “Clean Code” movement. The idea of Red-Green-Refactor is part of the view and ideal of “always leave the campground cleaner than they found it.” The ideas behind Software Craftsmanship are appealing to many developers. They are an underpinning of the Agile movement. Organizations world over are trying these ideas out and making them a core part of their work. Yet where are the testers in this paradigm shift?

Craftsmanship should matter to testers every bit as much as it should matters to developers. There are numerous benefits to well crafted tests; less debugging, maintenance, and rewriting. The function of a software tester is to communicate the experienced behavior software when compared against the expected behavior at a specific point in time.

That’s it.

The tester’s primary role is communicating. Information provided by software testing are reports showing the behavior of a product compared to its expected behavior at the time it’s observed. This information is then used to inform decisions with regards to management and budgeting of the project. This information ultimately helps the decision making process on whether or not to release a product to the customer. A good test plan provides information on what the team thinks is valuable and what constitutes a risk to a project.
Test cases are a conversation the test team has with itself. The danger is that we often fall into a form of short hand to describe what we do and create test cases that satisfy document standards rather than realistic testing needs. Bad test cases often come from being rushed, needing to fill details for documentation or compliance needs, and bad test cases don’t just fade away. They have to be deliberately removed or reworked in most cases.

We can look at the cost of performing an activity and the cost of not performing an activity. This is referred to as “Activity Based Costing”. Ultimately, there’s just one cost, the cost of the entire system. Activity-based costing integrates several procedures—value analysis, process analysis, quality management, and costing—into one analysis. Activity Based Costing includes both “Active Costs” and “Passive Costs” into the “Total Cost” of a product. Managing the whole value for software mean we focus on the entire lifespan of a product or company.

We move beyond the individual release and look long term to the life of the product (or the company itself). Software craftsmanship advocates designing and writing features that are easy to understand, support, adapt, and use both now and later when someone else needs to. Test craftsmanship is the art of doing the same for test cases and testing properties. Historically, testers have been at the end of a project, and usually under tight time constraints. For that reason, the idea of test craftsmanship often takes a back seat to the pressing need to just get stuff done and get it done fast! The net result, of course, is bad and poorly crafted tests.

Good software tests, when implemented, help reduce the various activity costs associated with testing. Good software tests aid communication between team members, help new testers get involved quickly, and have many opportunities for reuse. They do of course take time and analysis to make.

Ultimately, testing craftsmanship can be summed up in the following list (for detailed analysis of each, hey, read the book ;) ).

1. Plan the appropriate testing for the task.
 2. Know your audience and keep them in mind at all times.
 3. Don’t rely on someone else to clean up after you … even yourself.
 4. Refactoring and redesigning are not tools for excusing halfway effort.
 5. There will be no second chances to get it right due to the escalating technical debt you’re currently compounding.

Bad test case craftsmanship and bad code craftsmanship are probably the two most unnecessary costs incurred as part of doing business by software companies today. The software craftsmanship movement is taking on the developers in this regard. Testers, how about we pick up the charge and agree to be held to the same standard?

Three Cheers for Second Chances

So as you all know, the Pacific Northwest Software Quality Conference for 2011 just ended yesterday. Many of you also know I was scheduled to be there as a speaker. I had to drop out of the conference due to the condition of my leg, or the anticipated condition of my leg. As I look at it right now, I probably could have muddled through, but it would have impacted my delivery style seriously, so I still think I made the right decision. Besides, it's not the bones that are the issue, it's the circulation. If I walk, I'm OK. If I stand still, then I'm in trouble; circulation is not working well at the moment, and it starts to hurt after awhile. Anyway, what's done is done, and good or bad, I have to stand behind the decision.

Well, there were some people who were bummed about my not being able to give my talk. A good friend who shall remain nameless (unless they want to out themselves :) ), contacted me and asked me if I knew Lee Copeland, who chairs the STAR conferences (STAR-East and STAR-West). When I told them that I didn't, they said they'd get in touch with Lee and tell them about my situation and send them a copy of the original talk I had prepared. Which they did.

I'm happy to say that Lee got back to me and said that the committee for STAR-East reviewed my paper and proposal and thought it would be a good fit for the Star-East Conference in April. Of course, having just come off of a month's working from home, I felt strange asking for the time to go to the conference, but a commit was needed from me by a certain date, so I put it in and figured, well, we'll see what happens. After some back and forth, I got the clearance to go, and if I promote SideReel while I'm there, I'm on the clock. I think I can do that :).

Thus, I will get another chance to present the talk that I prepared for PNSQC in a live venue, and that live venue will be the STAR-East Testing Conference, to be held April 16-20, 2012 in Orlando, Florida. I've never been to a STAR-East event (OK, before 2010, I'd never been to a testing conference at all, so I think there may well be many "firsts" in this list ;) ). Nevertheless, there's a number of testers I've grown to know and appreciate over the years and several of them will be at STAR-East, so I'm looking forward to meeting and interacting with them, as well as getting the change to present the talk I've been itching to deliver now for several months. And huge thanks to a friend willing to help a guy out when he's down, too. So I guess my last question is... will I see YOU there :)?