Concentration on Information Radiation

Ever have an experience at a conference that just sticks with you, that one thing that seems like such a little thing and then, as you consider it more and more, you just slap your forehead and say to yourself "for cryin' out loud, why didn't I think of that?!"

On Monday while I was at CAST, I was the room helper for Dhanasekar Subramaniam's tutorial about "Using Mind Maps for Mobile Testing". Much of the session was around heuristics for mobile testing and the ability to capture heuristics elegantly inside of mind maps. as part of the process, we spent a bit of time creating mind maps in XMind to use later with our chartered test sessions. I've done this before. I've even created a full mind map of the entire James Bach Heuristic Test Strategy Model (yes, one mind map and yes, when fully expended it is massive. Probably too massive). As we were working to create nodes and sub-nodes, Sekar pointed out that there were many labels that could be applied to the nodes, and that the labels were additive. In other words, each node could have several labels applied to them. 

As I was looking at this, and seeing labels such as pie chart fill, green check boxes, people silhouettes of several different colors, red x's, green yellow and red explanation points, and many others, I started thinking about how, by color and proximity, we could gauge how much coverage we have given a particular mindmap (or in this case, how completely we have applied a heuristic to testing) and what the results were for that. Instead of stopping to write down lots of notes, each node we were testing would get a label placed, and each label would have a semantic meaning. Green check box meant good, red X would mean failed or something wrong, a quarter pie chart would mean a quarter done, a yellow square would mean something that was a warning, but maybe not an error. Different color people icons would mean the person who performed that set of steps, and so on.

As I was looking at this, I joked with Sekar that we could tell the entire testing story for a feature or the way we applied a heuristic to a story in one place with one relatively small map. We both chuckled at that, and went on to do other things.

The more I thought about this, though, the more I liked the idea. In a previous company, we set up a machine and had a couple of flat screen monitors attached. These flat screens were placed in the main area and left on, cycling the images that were shown, only in this case, they were graphs and pages of results that were relevant to us. In short, they were acting as information radiators for our team. At a glance, we could know if the build had failed, if deployment was successful or not, and where the issue would be if there was one. We could also use this technique for information radiation. Imagine a charter or set of charters. Each one had their own mind map. Each mind map could be cycled through presentation on the monitor(s). the benefit would be that, at a glance, the team would know how testing was going fort that area, and we could update it all very quickly. I kept experimenting with it, and the more I did, the more I became convinced this just might work.

To that end, I am holding a Weekend Testing session this coming Saturday, August 8, 2015 at 10:00 a.m. PDT. We will look at mind mapping in general and XMind in particular, and we will develop a small heuristic for a feature (within XMind itself) to test and to update. I really like this idea, but I want to see if it can be tinkered with, and if it might be a useful approach to others.

If you think this might be a fun way to spend a couple of hours, come join us on Skype. Contact "weekendtestersamericas" and add us as a contact. On Saturday, get on Skype about 20 minutes before the session and say you want to be added to the session. Prerequisite, if you want to follow along and actually do the exercise, would be to download the XMind app for your platform.

 Again, I apologize for the short notice, but I hope to see you Saturday.

BOOK CLUB: How to Reduce the Cost of Software Testing (19/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry. This entry covers Appendix B.

Appendix B - 25 Tips to Reduce Testing Cost Today by Catherine Powell

OK, wait a second... didn't I review a batch of "25 tips" yesterday? What gives?! Well, Appendix B is a strikingly similar set of 25 tips. In fact, they have exactly the same titles. The difference? Many of Matt's tips had to do with things that cut waste, but were organizational, and possibly out of the realm of the Individual Contributor Tester (i.e. the Tester doesn't have the authority to do many of those things). In other words, if I were an individual Tester, what could I do to lower costs where I don't have to ask anyone's permission to make an impact? That's where this "list of 25" comes in :). Look at them from the perspective of how any Tester can put these into play and make them effective. Also, don't try to do all of these things at one, do one at a time, practice it, master it, then move on to something else. Think of these like a pebble going into a jar. It may not seem like very much, but with enough pebbles, you'll ultimately fill the jar ;).

Tip 1: Cut your documentation to a minimum

Let's do what we can to make our documentation more DRY ("don't repeat yourself"). Have you been copying and pasting things into your docs? Stop that. "Incorporate by reference". Replace those copies with a simple note: "See XX", and then put it - just once - at that location.

Tip 2: Make the cost of changing the documentation cheap

Source control, wikis, whiteboards, they all allow us to share documentation and change it quickly and from multiple sources. Leverage that. Make internal reporting as quick as possible, and let the QA Manager or lead handle the more formal report (which should also be easier to do with this collaborative apprach). Start with your own team, then once you've shown the benefit by collaborating on the documentation within your team, share it with other teams and encourage them to follow suit.

Tip 3: Never be blocked

Make a wish list of the things you wish you could do but you don't have time for on your whiteboard, a sticky note, or in your Smartphone. Spend just ten minutes on this, you don't want a massive list. Then put the list aside. Next time you're blocked, pull out your list. Now's the time to do something on it!

Tip 4: Eliminate Multi-Project-ing

Individual Testers may not be able to choose if they work on multiple projects or not, but you can choose how you deal with working on multiple projects. Pick a single project and create a meeting in your calendar for you and that project - and spend that block of time working on that project only. The more often you can do this, the better. Minimize task switching. Turn off email, IM, phone, put a “do not disturb” sign on your chair while you are testing. Whatever it takes :).

Tip 5: Automate entirely redundant processes

Automation freaks some people out, because it sounds so huge and nebulous. Try some "computer aided testing" instead. Make a shell script or a batch file. Find a repetitive task that you've done at least twice in the last month, and create some "computer aided testing". Then check into the source repository so everyone can use it. Start small, get some ideas together, and build it out as you do more of it. Even a list of commonly used commands in a single file can be a huge boost.

Tip 6: Start with Soap Opera Tests

Soap opera tests are where you throw everything, including the kitchen sink, into your tests. Load them suckers up and let 'em run. You might find some spetcular issues :).

Tip 7: Start with quick attacks

Quick attacks are finding ways to do something hairy and over the top to inputs or outputs. Search for eveything. Load Hamlet into a text field (thank you, QAHatesYou, for my ultimate favorite Quick Attack ever :) ). If you feel fancy and want to go a little white box, then find every error message that a program is supposed to display, and force the program to show it. That's one of my all time favorite tests; lots of time is given to the happy path. Have they spent nearly as much time and effort on error handling? This is a great way to find out :). You get the idea.

Tip 8: Test Early

If you're in an agile environment, go through your last iteration and identify how long it took you to start testing each story after it was done, and come up with one idea to start each one faster. If you're in a more traditional environment, go find your friendly local developer and offer to spend half an hour working with him to "pre-test" his feature together. If the developer looks at you funny, offer to bring donuts (or beer, some people are just motivated differently ;) ).

Tip 9: Develop a regression-test cadence, and keep it light

Creating a "testing-oriented system diagram" so you can figure out what components relate to what other components. Use it to identify what needs to be regression tested and what you can more safely not run. Ask your team to expand on the drawing, discussing what fixes or feature changes might break other things. Have an architect validate your diagram. Again, if necessary, bribe with donuts or beer ;) ).

Tip 10: Test the things that actually yield bugs

Pick one area that is on the top of your "must regression test" list, and one area that is on your "we don't have to regression test" list. Then do it, with just one area and with just one release. Prove to yourself that it's okay to not regression test *everything* every time. Focus your testing efforts on areas that are at higher risk of having new problems.

Tip 11: Elaborate - and communicate - a test strategy or triage strategy

Create a status update that you can publish frequently. Publish it at least twice a week for the rest of the release. If your release is in under a month, publish the status update daily. Be sure to include the "must fix bugs" and a summary of new bugs found.

Tip 12: Decrease your time spent reproducing the problem

Next time you find an unreproducible problem, set a timer (an hour or two max). Now attempt to reproduce the problem and gather information. When time is up, send the unreproducible bug over to development with two or three ideas for extra logging or information that will help track down the problem if it happens again.

Tip 13: Do a low-cost beta program or pre-release

First, do a beta test of one. Get an account rep to bring a friendly customer to your offices. Let them use the newest software for an hour or so. Sit next to them and watch what they does and what questions they ask. Have a frank conversation afterward (thoughts, likes, dislikes, what's missing, etc.). Next, do a ride-along for half a day or a full day w/ an account rep to a friendly customers site, and shadow the customer(s) for half a day or a day. Keep quiet and take notes about how the customers actually uses the software. Share this information with your team. Learn from your actual end users. If the customer is open to it, record them with your phone, or at least take pictures. Pictures are worth a thousand words.

Tip 14: Recognize and eliminate project wankery

This might be hard, but give it a try anyway. Think of the meeting that you feel is the biggest waste of your time. Then make plans to skip it. Provide whatever informstion you need to satisfy those who feel you miust be there, but explain you can't make it for a pressing reason (no need to elaborate). The goal is to prove the necessary work *can* get done and that the product won't suffer if you don't sit in this meeting Swapping a meeting for a brief written report is a good bargain in service to your overall goal. You may not be able to do this very often or for many other meetings, but even if you can get out of just one of them on a regular basis, over time that can add up to a lot of quality testing time.

Tip 15: Stop fighting to manage the project

#1 Rule: NO WHINING. If you don't want to manage a project, don't complain about how it's being managed. It's probably not part of your job. Don't let the things other people choose to do interfere with the things that only you can do. Let it go.

Tip 16: If you're going to fight, win

Sometimes the decision really does matter! If it's truly important, you need to be able to sway the decision your way. Your reputation, your argument, and how much support you’ve garnered to your side will have a huge impact on the outcome. The next time you make a decision, sit down and write out why you're making that decision, along with the pros and cons of the choice you are making. Then give that explanation to your boss and ask for feedback. Make clear this is *your* decision, you're not asking your boss for permission. This lets you see if you argument is persuasive and if others will back you up on your decisions, and if you are really ready to "fight to win".

Tip 17: Walk around and listen

If you're a test lead or a test manager, take fifteen minutes and walk around the office in the morning. Repeat your 15 minute walking circuit that afternoon. Don't interrupt anyone, and don't talk with them. (It's okay to say "hi" back if someone greets you.) Just watch and listen, and notice who doesn't seem to be doing anything either time you walk by. After your circuit, sit down for 10 minutes with everyone who was not accomplishing anything and ask what you can do to unblock them or help them find a project to do.

If you're not a test lead or manager, see what you are doing with your time... no really, what you are really doing with your time? You may find that you are not nearly as focused as you think you are. Be brutally honest with yourself, and ask, OK, what can I eliminate? What can I minimize, and how can I get the necessary project time beefed up and finish what needs to get finished? Aim to decrease the distractions each day and boost the project stuff each day until you get to a realistic level of external tasks that have to be done, and as much project focus as you can reasonably perform. Seriously, nobody's going to be able to be 100% during an 8 hour day, but try to maximize where you can.

Tip 18: Write test automation that attacks the business-logic level

If you are using GUI tools and automating through the GUI, detemine how you might use another tool or rewrite your test to do a similar job from under the board. This might take some coordination with the development team but it certanly can be done. Determine what internal tools they use to get to the console or other hooks in the program, and then start working out tests that utilize them.

Tip 19: Develop a test automation library

This may be tougher for non coders in the test group, but think of the scripts and the automation you may already have. How much of it is duplicated in other tests? If you see a bunch of duplication, here's a chance to make a library or somehow centralize those items so that the can be called in a simpler manner. Personal experience: a neat trick in Cucumber is to take commands that get run a bunch of times and make "macros' of them by using the %Q|[cucumber line]| syntax. This lets you group various Cucumber commands into a one line call. DRY in action :).

Tip 20: Develop or hire expertise

Spend 15 minutes and figure out the expertise of each member of your team. Then take another 15 minutes to figure out where you need experts that you don't have. These are the areas you should be considering for training or hiring: to fill your expert gaps.

Tip 21: Get a return from conferences or don't go

Play "find an expert" the next time you go to a conference. Find someone with experience in an area your team lacks. Follow up with the expert - a quick email is fine - to establish a relationship. When you have a question about [fill in the blank expertise], you will be able to reach out to the expert. They don't have to be the famous testers. Often the ones that aren't are totally down with helping you and will often bend over backwards to help you get where you want to be. Note: don't abuse this relationship. They won't do your work for you, but if you do your homework and come to them with specific questions and showing you've already given your all, they may prove to be surprisingly helpful :).

Tip 22: Build a social network

Pick your favorite testing site (Software Testing Club, StackExchange, whatever) and answer one question. Next time you have a question, go to the same site and ask it. Do this regularly (maybe once a week), Over time, you will build a network of testers you can rely on. You'll also build a reputation and have public evidence of your own competence, which can be huge when it comes time to look for a new job :).

Tip 23: Examine invalid bugs carefully

Do some data mining in the defect tracker. Create two reports: (1) number of bugs marked invalid by reporter; and (2) number of bugs marked invalid by feature/module. Take that report and go talk to the reporter with the most invalid bugs. Pick a handful to walk through with him and spend 30 minutes brainstorming how to decrease that rate. Then take the report to the owner of the feature/module with the most invalid bugs. Pick a handful to walk through with him and spend 30 minutes brainstorming how to increase team-wide knowledge of his module/feature so the number of invalid bugs decreases [I'm not going to paraphrase this one, I think it's perfect as is :) ].

Tip 24: Build a test model that deal with the natural up and down staffing need for test resources

If you're already a tester on a team, spend 15 minutes brainstorming "what do you do when you're idle". A list of things testers can do when testing on the product is relatively light should be the result. Make a list of concrete actions you can perform. Next time you're idle, grab something from the list, and do it.

Tip 25: Ask the team to identify opportunities to drive out waste ... and implement them

Is there something in your processes that seems totally stupid and a complete waste of time? Kill it! Are there documentation steps that no one ever reads or does? Remove them. Tests that add little or no value? Weed them out. This is a gift that keeps on giving. As soon as you get good at this, you will find other areas you can trim away, too, and the better you get at it, the more areas you will discover. This is also a cycling list. As you cut wasteful activities, you will find room for more fruitful ones, and that fine, but remember, no fruitful idea will remain 100% fruitful forever. New processes develop waste of their own accord, and soon you'll be fining ways to "refactor" there, too.

BOOK CLUB: How to Reduce the Cost of Software Testing (18/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry.

Afterward

The editors of the book give us here a charge by way of a description of a game manual set up with a glossary. The point made is the last entry doesn't have a , it ends with Y, in fact, it ends with the word "You".

You: Have reached the end of this manual. We've taken you as far as we can; the rest is up to you.
Not get out there and play!

This is the official end of the book, except that it isn't. Over the next few days I'll be covering Appendix material that likewise deserve their own entries (see below), and of course, the conversation continues with the writers of the book. Most of us are very active on our own blogs, on Twitter and other mediums, such as the Software Testing or SW-IMPROVE discussion lists -- and on LinkedIn. If you try the ideas in this book, please let us know what you think. The important thing is that it's now time to put the ideas to work, so get out there and test!

Appendix A: Immediate Strategies to Reduce Test Cost by Matthew Heusser

So the book is finished, yet you may be thinking to yourself "OK, that's all great, but I need stuff I can do "right now" that can provide immediate payback. do you have any suggestions for that?"

Matt offers the following:

While testing less is an easy cost cut, it also introduces risk. The best way to help reduce the cost in an immediate way is to reduce or eliminate waste. The following strategies may help that, as well as provide additional value added benefits from trying a few things (25 of them, in fact):

1) Cut your documentation to a minimum
First you have to write it, then maintain it. Plus it only has value when people read it.
Instead of a comprehensive breakdown, focus on removing details that are not important to the testing effort. Keep those that would need to be known if you needed to hand it off to someone so you could leave town for a week. Is there enough there that the tester can read it and to the job? If so, aim for just enough documentation to do the job, but not so much that it becomes a time drag.

2) Make the cost of changing the documentation cheap
Perhaps use a shared doc, or on a network drive,  put it in a wiki, or hey, why not just keep it on a whiteboard?

3) Never be blocked
If a tester feels they are blocked or they can't do anything, they are generating waste. Instead, ask the following:

(a) Can I interview customers for test ideas
(b) Can I pair with the developers to learn the system
(c) Can I pair with another tester on a different piece of functionality in the same project

4) Eliminate Multi-Project-ing
Focus on a task at hand, and if you are blocked on it, see #3. Taking on side-projects and having to frequently context switch takes a lot of time. Focus on one area until you are done, then move on to something else if necessary.

5) Automate entirely redundant processes
While automation is sold as a panacea to all testing troubles, we know it's not. There's much more a real thinking brain can do at interesting points in the software that automation can never do. However, there's a lot of steps that are horribly repetitive and don't add any real value to the testing knowledge (set-up, navigation to key places, etc.). Absolutely automate these sets if you can.

6) Start with Soap Opera tests
Instead of testing for individual factors in a test, start out with testing the largest group possible, and if you see errors, whittle it down instead of testing each factor in isolation (which will take much longer).

7) Start with quick attacks
"Quick Attacks"are all about overwhelming the software with invalid input, too much input, and input that is out of range. If the software handles the obvious exception conditions well, it is likely in good shape. If the developers left holes and errors in the exception conditions, they likely left holes and exceptions in the main business logic, too. Quick attacks allows the tester to find bugs early, learn business rules, and perform a quick assessment of the software -- all at the same time!

8) Test Early
If you are use to working with software where an entire build is ship to you to test a new feature, see if there is a way you can get access to an environment where the feature is added and you can test it without having to wait for the entire build. The sooner you can test a feature in isolation, the sooner you can provide meaningful feedback the developer.

9) Develop a regression-test cadence, and keep it light
The days of shipping software once a year are disappearing fast. Most products are moving to incremental updates over the web, or much more frequent updates if its a web property entirely.

Regression testing, therefore, needs to be modified so that we don't have to run a years worth of test changes in a week,  or less. While there is a chance that any change could cause a problem anywhere else in the code, this is not universally true. By compartmentalizing tests that actually have a relation to other functionality, the regression tests can be made more efficient and focus on the areas where testing will actually yield results, which leads to...

10) Test the things that actually yield bugs
If you have regression tests you run for every release that never seem to find bugs, find a way to limit how many of those run. Maybe run them every third release, or one third of them per release, or just enough of them that would find a major defect if it were introduced in that code.

11) Elaborate - and communicate - a test strategy or triage strategy
List features by how critical they are, then go down the feature list in order for a "first pass". If you complete that "first pass", go in more depth. Make sure everyone knows how the decision was made, what the decision was - and the decision-makers feel like part of it.

12) Decrease your time spent reproducing the problem
If your team is spending a lot of time on bug reproduction, look for ways to lower it. Have the server store a log of every action. Find a tool to record exactly what the tester is doing and make a screen-cast that can be played back.

13) Do a low-cost beta program or pre-release
Can you segment your users and send some to a managed beta? If so, you could release early and engage the users in helping to test. You'll need some infrastructure (which users get which builds), and on web-based projects, some sort of production monitoring.

14) Recognize and eliminate project wankery
Getting better requirements, having more time to test, doing architecture and code reviews - those can all be good things. They can waste a lot of time, too. Experiment with these practices, but view them as experiments. After you've tried them once or twice (or if they are currently "mandated") ask if there will be less defects down the line, do we add value to the project, do we decrease overall cost by doing these things? If the answer is no, change the format, or drop them.

15) Stop fighting to manage the project
Instead of arguing over if the software is ready to release, make the defects visible to everyone and let senior management decide if it should be shipped. Imagine saying something like this:
"We're going to stop fighting you over issues a, b, and c. We yield to a, b, and c. We're going to focus on testing. If a, b, or c fail, don't complain to me. The decision is yours." Consider how liberating that might be.

16) If you're going to fight, win
If you do give on some issues, you may just want to pick a few battles. If you pick those battles, fight to win. Otherwise, you're just wasting time.

17) Walk around and listen
It's s good bet that there's a lot of other stuff going on that may look like work, but may just be busy-work leading to little value. How can you tell if that's happening? Walk around and listen. See what the testers are actually doing. They may think they are focusing on something important, or they may be "blocked" and not know how to proceed, but are too embarrassed to say anything. By doing this, you can then help coach or guide the testers to getting out of the rut, or determine what the blocker actually is. Is it their process? is it another person on the team? Are they just "socially loafing"? Don't wonder, find out :).


18) Write test automation that attacks the business-logic level
GUI testing is very expensive, and often GUI testing is really "change detection" testing, and can be very brittle if UI elements change. With setups and scaffolding and test hooks, testers could write tests that exercise business-logic level functions. These tests will general run much faster and be less brittle than GUI tests. Automated tests verify those dozens (hundreds? thousands?) of possible inputs for a given function. Test automation is an investment -- in the short run, testing costs always go up. Starting at the business-logic level might make it possible to see returns in days and weeks, not months or years.


19) Develop a test automation library
If you do want to test at the graphical level, or want to have more powerful business-logic tests, you may want to develop re-usable functions. For a GUI, that might be login (taking to parameters and pushing the login button), search, tag, etc. At the business-logic level these will probably be object-oriented functions.

20) Develop or hire expertise
As a test manager, you can foster expertise with brown-bags and pairing; when you look to expand your team, look for skills that would round out the same. Help develop your team so that there is redundant expertise where possible. Avoid the information silo.

21) Get a return from conferences or don't go
Expect employees to write two 'what I learned' documents: Things for us to do on Monday, and strategies to pursue over the next year. Use conferences to attract talent. Send your employees with business cards and a list of open positions. Use conferences to retain talent. Build the conference into each employees annual professional development goals, and he'll be more likely to stick around to attend it. Use conferences to grow your network of friends with specific expertise. If your staff comes back with ideas to implement on Monday, let them actually try them out!

22) Build a social network
Getting involved in local user's groups, attending conference, collaboration over the internet, all of these can provide an increasing list of people with skills you do not have, people willing, even eager to share ideas if you tap them on the shoulder. Instead of being "blocked" by a performance testing problem for a month and calling a consultant, a few questions to the social network, some calls, some questions, and a couple of days of exploration might solve the problem directly.

23) Examine invalid bugs carefully
How many bugs are being marked invalid, or 'wrong' for various reasons? Each of those means a tester and a developer both invested time for no benefit. If enough bugs are invalid, take a look at the reasons why and try to prevent it in the future.

24) Build a test model that deal with the natural up and down staffing need for test resources
Test projects have an ebb and flow. If testers remain constant, you'll either be stuck with too small a team or too big. Build a staffing model that allows you to scale up and down quickly. Perhaps bring in tech support to test, hold a beta program, bring in an outsourcer, or consider crowdsourcing like  uTest or volunteer a project to Weekend Testers.

And finally ...

25) Ask the team to identify opportunities to drive out waste ... and implement them.
A good tester is inquisitive, curious, and critical. The tester is likely to think "Why do we waste so much time doing X? I would think we could get by without it, or get the same benefit from doing Y."
So ask your testers what they would do to eliminate waste.

BOOK CLUB: How to Reduce the Cost of Software Testing (13/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry.

We are now into Section 3, which is sub-titled "How Do We Do It?". As you might guess, the book's topic mix makes a change yet again. We have defined the problem. We've discussed what we can do about it. Now let's get into the nuts and bolts of things we can do, here and now. This part covers Chapter 12.

Chapter 12: A Nimble Test Plan; Removing the Cost of Overplanning by David Gilbert

As I posted in another blog post a couple of days ago, there is no more frustrating experience than having to put together a way too specific test plan, one that is under most circumstances never going to be completely run, and even in the best circumstances, doesn't even really provide the appropriate coverage needed (but it sure looks impressive and feels ever so hefty as it weighs in at several ounces if not closer to pounds).

These test plans are not just cumbersome and unwieldy, but they are also shackles that lock us in place and become more inefficient the longer the project progresses. Any later discovery and re-calibration is often discouraged in favor of covering the tests already defined, regardless of whether or not they have any real effect. By creating a "nimble" test plan, we focus on the test cases that are the most relevant, and we adapt it over time as the conditions determine.

Plain and simple a "plan" is what you believe you will do, at the time you devise the plan. Things change over the course of time. Being able to adapt and change is much more effective than a plan that was created earlier and locks the tester into a set approach. Much like a map that's never been made, a plan is only the best ideas an estimates of what may be happening. You won't know for sure until you actually try to follow it, and hey, your assumptions may prove to be wrong!

A nimble test plan will include direct communication and feedback, and it will be a living document that changes as new information comes to light. Nimble is defined as:

• quick and light in movement; moving with ease; agile; active; rapid: nimble feet
• quick to understand, think, devise, etc.: a nimble mind.
• cleverly contrived: a story with a nimble plot.

Nimble testplans are small, lightweight, adaptable to change and easy to read and understand. it is allowed to change and evolve as time progresses based on customer’s needs. They uses good ideas from many different disciplines.

To creating a nimble test plan, it's important to keep some key criteria in mind:

1. Keep It Light

The test plan should provide proper guidance for the scope of testing, but should not dictate the act of testing. Keep it Simple. Anything not adding value to the customer should be considered waste.

2. Make All Content Goal Oriented

Useful heuristics can be a big aid in this process, such as San Francisco Depot (SFDPOT): Structure, Function, Data, Platform, Operations and Time. Covering just these areas in a system and actually executing a testplan with just this heuristic can provide a huge advantage to testers over those who do not structure their approach along a similar heuristic.

3. Commit to as Little as Possible

We cannot know all the tests in advance. Ideas will come as we explore. If the time for testing gets squeezed, the scope of testing also gets squeezed. Thus, testing that was planned may not get done. Flexibility becomes vital in these situations. Higher risk areas will be grouped and they will likely be performed first.

4. Keep It Practical

Our test strategy should be product specific. Start each test plan with a blank sheet of paper. If you must conform to some corporate standard, try to limit it to a simple outline. Focus on empowering the team to do good testing and returning good information to the customer. Incorporate Risk in your planning to focus on the most important parts of a system. Prioritize which areas need the most attention.

5. Empower the Team

If we do not directly and explicitly empower the test team the likelihood of their being empowered during testing will go way down. Focus on People and rapidly create value for the customer, and understand what the customer actually wants. The primary goal of a test plan is to empower good testing and communicate to other stakeholders how we do that.

6. Create Fast Feedback Loops

Do a little testing, get the results to the development team and decide what further testing is needed and in what order. Get these results quickly, and have the envelopment team require feedback quickly. Face to cafe communication is critical at this stage, and a quick turnaround for testing and associated feedback is essential.

7. Iterate Often

While this often happens by default anyway, it's critical that the testing proves be allowed to work closely within development. If issues are discovered, quickly stop testing. Have the project go back to development, and have them make necessary changes. Get the project back into testing. Include any changes and modifications to the underlying code into your test plan.

8. Communicate Face to Face

Face to face communication is essential. Non-verbal cues such as body language, facial expression, breathing patterns, nervous ticks, etc. come through in face to face communication in a way that email or IM never will capture.

Testing is never able to flow on a perfect flight plan. For that matter, neither do planes. We may be off course on an airplane trip 90% of the time. The reason we get to our destination is that the flight crew consistently adjust the flight plan. A Nimble test plan takes this same consideration to heart; we can't know everything, so let's start with what we do know, and through the process of discovery, fill in the blank areas of our map. If we do this, we will be more likely to test that which is important, and jettison that which is not.

BOOK CLUB: How to Reduce the Cost of Software Testing (12/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry.

We are now into Section 2, which is sub-titled "What Should We Do?". As you might guess, the book's topic mix makes a change here. We're less talking about the real but sometimes hard to pin down notions of cost, value, economics, opportunity, time and cost factors. We have defined the problem. Now we are talking about what we can do about it. This part covers Chapter 11.

Chapter 11: You Can't Waste Money on a Defect That Isn't There by Petteri Lyytinen

Ideas such as technical debt and the need to be faster to market and to get the product released faster are always consideration we have to contend with. In some ways, these can be anywhere on the spectrum of benign to truly dangerous. The amount of time and the proximity to release tends to determine where on the spectrum your organization or project may fall. Make no mistake, though, technical debt and chasing ways to cut corners as the release gets closer happens. Focusing on immediate needs often cause the technical debt to grow, not shrink. While it's possible to enhance testing as a standalone process, why should the testers have all the fun? Petteri suggests that developers have a chance to contribute to the decrease of the cost of software testing as well, by focusing on techniques like Test Driven Development, Continuous Integration, and Lean principles of Software Development.

Here's a heretical thought. Want to reduce the cost of software testing? Bring up the skill of your developers. More to the point, encourage your developers to develop software from the approach that Uncle Bob Martin refers to as 'the Software Craftsmanship Movement". Central to that is the idea of Test Driven Development, and the ccircular process of developing software with tests in mind first. Having the test fail first, then coding to get the tests to pass, and then refactoring and repeating the process.

It should be noted that TDD does not resolve all of the issues; there's still plenty to test. What TDD does do, however, is it takes many of the solidly boneheaded issues out of the picture. My work with SideReel is an example. Yes, there are occasional issues that I find or details that may not be specifically implemented the way they should, but I rarely come across a truly bone-headed omission or a really truly "broken" implementation. the developers has mostly resolved those issues through actual TDD processes, so I can vouch for them being effective :).

Continuous integration (CI), is the process where all new code committed gets immediately built and deployed to a server and where new features and functionality is immediately tested. Along with TDD, this helps developers commit small or large scales changes and quickly see how their changes effect the rest of the environment and application. Coupled with TDD unit tests, and a smoke test run from the testers side, testers are freed up to focus on exploring the new changes and seeing if the changes have additional issues or if they are slid enough to be deployed. Another benefit of CI is that developers can quickly see where the changes "broke the build" and can back them out, make fixes, and then resubmit/retest. this helps with the ever present issue of finding issues late in the game. While it will never be completely eradicated, the odds of finding a problem that has never been tested or examined in conjunction with other components goes way down. Still, even with these enhancements, testers must never get complacent and think their work is all done. It's not. As E.W. Dijkstra noted: "Program testing can be used to show the presence of bugs, but never to show their absence".

The biggest benefit to using processes like TDD, CI and automated smoke tests is the hope and goal of eliminating needless time waste. As time and skills grows with the development team, downtime because of issues diminishes. It also helps to diminish the inevitable downtime between "bug discovery" and "re-test" with an updated module. Petteri suggests having the team sit closely together so that, when issues are discovered, the need to enter issues in a defect tracking system is not the bottleneck to a fix being made. rather, leaning over and saying "hey, developer person, check out the issue I just found here!" While tracking issues is not in an of itself a bad thing, it can be if the workflow is specific to alerting each member of the next step by changing states in the issue tracking system. Direct verbal updates are much faster. If immediate personal interaction is not possible, use Instant Messaging as the next best thing.

It's common to think that just having automated test scripts will solve all of the testing team's problems. They can help, up to a point, but as more and more tests roll in, and need the be run, the quick and dirty smoke test often grows into a more extensive set of feature tests, and their completion time grows longer and longer (why yes, I have experience with this :) ). it's impossible to test everything. Even simple programs could have millions of paths through the programs and testing all of them would be physically impossible, even with automation running day and night. thus the goal is not comprehensive testing, but targeted and smart automation testing. Combinatorics (pairwise testing being a popular version of this and a term known to many testers) can help trim down the number of cases so that the tester can focus on the ones that give the most coverage in the least steps.

A disadvantage to up front test cace development is that we just plain don't know what the test cases are going to be. we can guess, but it takes time to develop everything to get the true picture of the requirements and the coded features. Reworking these test cases later is a pain. Rather, Petteri describes a process called Iterative Test Development (ITD). When reading a user story, a use case, or part of a technical spec., write down a few brief lines about what needs to be tested. As developers start coding features, flesh out each test case in a simple format. When the feature is finished, fill in the precise details for the test cases and start testing with the full requirements s son as they are ready.

These examples (TDD, CI, and ITD) all point to the same goals and focus; they are meant to help make sure that the craft of developing software is first and foremost a sound one. ITD is the testers step to help make those development steps come into focus quicker and make sure that, as the developer focuses on the craft of software development and the processes that bring testing to their sphere, we likewise also develop our tets as the code is being developed, so that we do not waste time creating test cases that do not address the real code in question. Ultimately, this all comes around to helping answer Petteri's initial goal... you can't wast time on a defect that isn't there. Rather than focus on finding bugs, let's focus on preventing them in the first place.

BOOK CLUB: How to Reduce the Cost of Software Testing (8/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?


Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry.


We are now into Section 2, which is sub-titled "What Should We Do?". As you might guess, the book's topic mix makes a change here. We're less talking about the real but sometimes hard to pin down notions of cost, value, economics, opportunity, time and cost factors. We have defined the problem. Now we are talking about what we can do about it. This part covers Chapter 7.


Chapter 7: Test Readiness: Be Ready to Test When the Software is Ready to be Tested by Ed Barkley


Ed starts the chapter with a painfully familiar refrain. Many times, we are ready to start testing, but our environments are not, or they are not in sync with the development machine, or needed patches have to be applied, or other server requirements have to be rolled out so that testing can begin. after a much protracted comedy of errors, the test team is ready to hi the ground running... a week and a half after testing was supposed to begin.


Sound familiar? It sure does to me, I've lived this more times than I care to admit, at more than a few companies. There is a beautiful symmetry to the Boy Scout Motto and a test environment actually being really and truly ready to go. "Be Prepared" is more than a slogan, it's really the best way to hit the ground running when you really need to be doing so.


Good planning can save you money! It's one thing to plan your work and work your plan. It's another to anticipate issues and be ready with a contingency plan at the moment an issue arises. Having backups, servers in reserve, virtual environments that can be quickly spun up in the event of a crash, etc. will help you to be able to stay ahead of the curve. Cool. Can we actually do that? Ed thinks the answer is yes :).


Having test cases and being ready to test is just the tip of the iceberg. To get ready to "catch the wave", you'll need a little more in your quiver.


Areas of Concern to be addressed first are your Test Environments themselves. Are they up to date? Do they have the latest OS patches, application patches or anything else they need to do what they need to do? Next comes your Configuration and Change Management. Is the hardware, software, firmware, data and databases, documentation, test tools/fixtures, and test documentation in sync with what the developers are using? Are you sure? Is all your Documentation up to date? That means Requirements docs, Test Plans, Traceability Matrices, Models, Use Cases, User Stories, Test Scenarios, Cases and Scripts, whatever docs you use. Is your Data up to date? Is your "Golden Database" really golden? Is your test data you'll be using in sync with the Golden Database? Are your Test Tools up to date? Have you validated your automated scripts, your macros, or other computer aided testing tools? Do you know who your Testers are? Have they been trained in the tools and scripts they will be using? Is your issue Issue Management tool up to date and ready for use?


Think of any of the above areas. Is there a possibility something could happen in any of those spheres? If so, what would the impact be? Knowing this can help you plan for contingencies or any other number of options to get you back to up and running.


It's likely that anything happening to the Environment, Configuration and Change Management, Data & Testers will have a high impact on a project. Issues with documentation may warrant a moderate impact, depending on when the change or disruption occurs. Earlier in the process is better, later can eat up a lot of time for things that are not needed, or not allow for time for things that are important but unknown. Issues with Test Tools & Issue Management may be moderate to low. Although losing the speed of automated testing, we can still test, and sometimes we can cover more ground manually (if we are using exploratory methods) than our automated tests can in their well worn grooves :) ).


Ed recommends developing a checklist to help you determine if everything is ready to go. Just like I as a Scoutmaster use a checklist to make sure I have all the information that I need for my Scouts any time we do an outing like summer camp, the checklist can be just as important for the start of a major software test cycle (or any test cycle for that matter). To see the full list of checklist options for each area, check out the book :).

Book Review: Testing Dirty Systems

Wow, two book reviews in one day? Well, this one's been waiting in the wings for awhile. I've wanted to finish it, but reality keeps getting in the way. It's also a fairly lengthy and detailed book, so it took me some time to get through it and consider some of its ideas.

Testing Dirty Systems is the latest book by Randy Rice and William Perry. Both collaborated together to bring out "Surviving the Top Ten Challenges of Software Testing". That was a fun book to read and it was fairly quick to absorb its main messages. Testing Dirty Systems is a much bigger work and covers lots more ground. The title alone ("Testing Dirty Systems") might raise some eyebrows. Perhaps a better title would be "Testing Real World Systems That Actually Exist and are in Production Today", but that's not nearly as snappy a title. Besides, those of us who have spent any time in the testing trenches already know what a "dirty system" is. The trick is to find out how the average tester actually works with these systems and brings value to the testing efforts when they can't realistically have a truly clean test environment. Does this book help with that?

The book starts out by defining what Randy and William consider a dirty system, and that dirty systems fall on a continuum. They start with a checklist that will help you determine where your project falls. I've been on projects that range everywhere from Spotty to Soiled (I've yet to be on a truly Clean project, but thankfully I've never been on a project that was so bad it would be deemed Filthy). The benefits to seeing this is that you get a clear idea what your limitations and issues are even before you start testing.

While Dirty Systems are a challenge, Randy and William make the case that these systems can be tested. They can even be tested in ways we are familiar with, but they need some extra "filtering", to borrow from the coffee metaphor they use in the book :).

Testing Dirty Systems uses a six-point process to help guide the tester through this environment. The steps are as follows:

Step 1: Diagnostic Analysis: get a bearing on the system and find out just how dirty it is, utilize this information as you devise your test strategy.

Step 2: Test Planning: A two-pronged black box and white box approach to testing. Watching the behavior and evaluating the software using heuristics to determine if the system is working as described, in concert with an active white box testing approach to really see if the code does what requirements say it does.

Step 3: Perform the Tests: Fairly obvious, but goes into great depth about the various considerations to be worked through when testing a production system. Exploratory and session based test techniques from both black and white box testing approaches would be effective in this space.

Step 4: Interim and Final Analysis: Bug reporting, of course, but there’s more than that. Code coverage, test case coverage, and requirements coverage fall into this area. It’s important that more than just defect finding and defect fixing happens here. This is a clear way to indicate whether or not the system is responding in a way that makes sense based on the known requirements, and to help clarify those requirements based on real world behavior.

Step 5: Test Reporting: Randy and William recommend two reporting systems, one that focuses on fitness of each component and another for cumulative reporting. The details that are included in the test reports are meant to help direct system maintenance, they are not meant to inform new development (though they certainly can). Also specific to this is the idea of being as effective with our communication of issues as possible (I can recommend a great class called “Bug Advocacy” that does exactly this, but that’s outside of the scope of this review ;) ). To borrow from “Testing Computer Software”, the true mark of effective testing in not the number of bugs that get reported, but the number of bugs that get fixed.

Step 6: Using Test Results to Clean the Dirty System: This is a compelling idea, in that the whole goal of testing a dirty system, in addition to testing it, is to help put it into a state where it is less dirty than when we started. Over time the system can be "cleaned" as we provide more details and fill in areas that are prone to fail.

This is fairly heavy on process, and some may complain that this book is advocating the "old ways" of documenting everything and doing everything up front. Actually, it's not. Randy has often been quoted as saying there should be "just enough process" and "just enough documentation" to make sure that the team can be effective. The underlying point of the book is that, for many testers, you are going into or are already working in an environment that is sub-optimal in the way that it was developed or maintained. The processes are not meant to be a cookbook to handle all possible scenarios, but rather a series of techniques to help the user get into the mindset of being a detective and working through the challenges each phase will present. Each section also gives the user a case study to help the users actually see the concepts related to each of the six steps applied and explained.

Bottom Line:

This is a dense book, and it covers a lot of ways to perform testing in environments that may be sorely lacking in the way of documentation, requirements or understood expectations (hey, that sounds like a lot of projects I've worked on). Will this be a panacea to the problems we all face? No, but it certainly gives structure to a lot of things we take for granted. Will you need to do everything listed in these chapters to effectively test a dirty system? Again, probably not, especially since every project will have its own set of eccentricities in this regard. The goal is to get the reader into the mindset and develop skills and tools that will work for their particular issues, and that's the key takeaway of this book; it helps the user develop skills in areas that they may not be currently familiar, and help them to come to grips with systems that might stop them in their tracks otherwise. If you are starting with a clean slate or a company that does a great job on documenting their development approach, requirements and standards, this book may not be as helpful, but you may still get some new ideas. For those who have less than optimal environments, there really is a lot to consider here.

Does Simple == Reliable?

In the wake of the issues surrounding the Toyota and Lexus recalls, I have been thinking about cars a lot lately. Specifically, I’ve been thinking about the cars I’ve owned over the years and the various challenges they all have posed (or not posed). During my life, I have owned cars from the 70’s, 80s, 90’s and 00’s. Some have been gigantic vehicles (my 1971 Ford LTD Wagon had an engine that would put to shame most full sized trucks of today), and some have been a bit smaller (my 1975 AMC Hornet was an early example of what would come to be known as the “economy car”). I’ve owned what were considered top shelf cars (my 1978 Saab GLE, a true boutique car with seriously finicky boutique needs for maintenance) and genuinely bare bones vehicles (my 1990 Ford Escort Pony had no power anything in it, and was purchased for $6,000 brand new).


Looking back on all of these vehicles, one thing stood out. All things being equal (equal maintenance, equal attention to the wear and tear and replacing of items), the vehicles with the best longevity and the best overall performance (not referring to speed or handling, but costs to maintain during their lifetime) the simplest of my vehicles has to date gone farther and less expensively than all of the others, and that would be my 1990 Ford Escort Pony. In fact, between me and my brother, we drove that car relentlessly for ten years, averaging 100 miles per day between us, and finally retiring it after it had logged just shy of 300,000 miles (for those who are familiar with American cars prior to the 1990’s, you will understand why this astonished both of us over the years).


My brother, who is a pretty adept auto mechanic , told me why he felt the Escort did so well for so long. Since at the time, I was scraping together whatever money I could, and couldn’t afford an expensive car, I opted to buy a car off the lot that was the absolute base model. Manual transmission, no Power windows, no A/C, no power steering, no power braking (heck, it didn’t even have a radio when I first bought it, though I did put one in after the fact)., all of this added up to the fact that the amount of moving parts and electronic connections were limited to the bare essentials. In short, little went wrong because there was so little that could go wrong.


I’ve often thought about this when I’ve approached testing projects and testing methodologies. Many who have heard me talk about testing have heard me reference my “trusty old pony” as a testing metaphor. For those who haven’t heard it, here it is… just as the Ford Escort Pony had so little that could go wrong, one should make tests that, for the most part, as a simple as required to test the feature and perform the task. Just enough detail to accomplish the task successfully, but no more. Some people have criticized me for this in the past, asking for “more details” and “more explanation” when creating test plans or adding test cases (often times, just for the sake of adding test cases and no other reason). Past experience has shown me that the added documentation was rarely used, and that the added details often didn’t result in better test cases. More maintenance was required to keep the documents and tests up to date, absolutely, but there was not a direct relation to better testing or better performance. The process ran well when things went well, but there was a heavy re-do charge when things had to change (reminds me a lot of my 1978 Saab; a dream of a car when it ran, but very expensive to fix, and it seemed to need to be fixed an awful lot).


For those who find that they are looking at doing a lot of document maintenance, test case maintenance, and a lot of rework because of a lot of tests that promise to do much but don’t really seem to deliver, consider my “trusty old pony”, and try to see how much you can do with simple and focused plans and tests. I believe the time you will save in maintenance will translate to more and better testing, all things being equal. Happy driving :).