SummerQAmp: What is Bias? Intro

This is the next module that I am working on for the SummerQAmp program, and as such, I'm looking for both feedback and a way to see if my ideas make sense. Read into this what you will :). 

Bias is a tricky topic. There are several examples of it, and ways that we manifest it. Each of them is somewhat challenging to describe. One of the ways that I want to direct this is to use a thought experiment using a game many of us are familiar with already. 

I did this as a session for Weekend Testing some time year ago. In it, we used the game SET. Specifically, we used two versions of the game. The first version, which can be seen here, has a basic format and set of rules to play the game. In it, a single player tries to find six matching sets that follow the rules of the game (the matching criteria is that everything has to match or nothing can much for it to be a set; same/different color, same/different number, same/different shape, same/different shade). 

We used this approach to help get the players familiar with the game dynamics and the basic rules of how SET works. From there, we let the players loose on a different version of the game, a multi-player version that followed the rules for matching, but the way to get scores and find sets was different, in that people were competing to see who would find matches first.

What was interesting to see was that a number of the participants were reporting issues they were having with the second game. The issues were legitimate discussion points, and had a lot of validity, but what was more interesting to me was that many of the issues being reported were because they were comparing the second game's dynamics to the first game. Why? They were two separate games, and they had two different sets of rules. One game didn't necessarily reflect on the other… except that it did, at least in the mind of the participants in this Weekend Testing event.

The reason? Playing the first game trained them to consider interactions one way, and then playing the second game didn't match their expectations. For some, that was a frustration point. They wanted the two games to play the same, or similar enough that the dynamics of the first game would be reflected in the second.  In short, I unintentionally "biased" some of the participants by having them play the first game and get used to the "rules" before I had them play the second. 

What if I had said up front "these two games are very different, and you should not compare the two to determine acceptability of play"? Would the outcome have been different? It's hard to say, but the point is, I didn't. By not doing so, I introduced bias into the process.

There are many areas of bias in software testing, and may ways to approach it. This is one I'm planning on using. What do you think? A reasonable way to introduce the topic? Could you think of a simpler way? I'm serious, I am totally open to suggestions. There will be more to come :).

Michael Larsen, TESTHEAD

People May Laugh, Or...

My daughter and her friend outside the 4 Stars Movie Theater in San Francisco

My kids, and especially my girls, are fans of Japanese animation (Anime), and we share a love and an interest in a number of titles. Back in 2011, one of the more inventive and unique titles to make its way to the television and streaming sites  was Puella Magi Madoka Magica. In it, a group of "Magical Girls" does battle with witches, and they make discoveries along the way that have tremendous consequences to themselves and others. The "Magical Girl" is a trope that has been used in Anime for several decades. This story stands the tradition on its head.

My youngest child really grew to love this series, so much so that she decided she wanted to make a Cosplay outfit based on one of the characters, Mami Tomoe. So she had it made, wore it, and I figured that would be the end of it. Well, not quite.

The creators and distributors of Puella Magi Madoka Magica has taken the twelve episode Anime series and have made two movies that they are showing at independent movie houses all around the U.S. and in other countries. This was a chance to see the story in a "director's cut" of sorts, and one that ran for four hours over both films. As a birthday present for my youngest daughter, I decided to spring for it and go see the movies. She invited her good friend... and insisted that she go in her Mami Tomoe costume.

We went to the showing, and Amber quickly noticed something she hadn't expected. She was the only one in costume. Out of the hundred or so attendees, she stood out, dramatically so. I was curious to see wat her reaction would be. Would she feel embarrassed? Would people laugh? I had warned her that, even though this is an Anime showing, it's possible she might be the only one in Cosplay. Was she OK with that? Her reaction was that "people may laugh... or they might think it's fun. I'm OK with either one."

Well, people didn't laugh, other than to say how cute her costume was. People in line came by and complimented her on her outfit, and her being so young and so brave to get into the spirit of the movie. A number of girls walked by and shouted "Mami!!!", and some said "Oh, I want to do a costume like that someday!" Lots of people asked before the movie, during intermission and afterwards if they could get a picture with her. She smiled and said "Sure!" and was very gracious with all the requests.

Why do I mention this? It struck me that sometimes we deny ourselves the opportunities to do things because we fear other people's reactions. Often it comes down to "People Will Laugh At Me".  My daughter reminded me last night that, yes, sometimes we do things that might get laughs, and sometimes  people who don't appreciate what we do will look at us funny and yes, they may even laugh. Her answer, though, was "So what? I'm not doing this for them. I'm doing this for me, and for those who will appreciate it for what it is". I'm proud of her, and for the fact that she had such a great experience, and was willing to do what she wanted to do and not be afraid of what other people might think.

Michael Larsen, TESTHEAD

Use It or Lose It

A couple of nights ago, I had one of those frustrating realizations, where the understanding was there, but the knowing of the rules and the exact steps of what to do escaped me. Was this during some intense testing session? Sort of. Was a lot on the line? In a way, yes. Did I find myself a bit aggravated? I did! So what could possibly be so unnerving and frustrating? Sitting down with my older daughter as we tried to unravel and make sense, to both our satisfactions, multi-variable problems and systems of equations.

First, a little background. Math has never been a really strong subject for me, at least not without a lot of tweaking and playing around. I know how to work formulas. If I can put them into scripts or programs, I can make them work. I can even appreciate a lot of the arcana that goes into numbers and the theory behind them, but some of these questions were just driving me crazy.

The funny thing was, as I was going through some of these problems, and I was breaking free the calcified neurons that probably haven't thought about, much less used, a lot of this stuff in at least fifteen years (seriously, it was back in 1997 that I went on my marathon mathematics quest and made it through Intermediate Algebra up through Calculus 1 in a little under four semesters), I realized that, while we can understand this stuff at a lofty and philosophical level, there just is no substitute to going through and solving problems. You just have to suck it up and work this stuff. It's like going to the gym, training for a  marathon or… learning to code. The more you do it, the better you get. The less you do it, the harder it is to recall what you need. Fortunately, my son, who is much closer to this, was able to come in and check our work, and make sure I wasn't leading my daughter on the wrong path.

We often think we are in a good place, and that we have a good handle on ideas. That may be true, but just understanding the ideas is a small part of the story. Having the skills and being able to flex the muscles associated with those skills is the much bigger part, and yeah, it really does come down to use it or lose it.

Michael Larsen, TESTHEAD

Santi Says… Live Blog from San Jose Selenium Meetup

This is going to be a Live Blog, so please forgive the fast typing and likely typos and bad grammar... I'll clean this up and make it pretty later :). Slides to tonights presentation are at http://bit.ly/test-automation

One of the adjustments I'm currently making is that the time and ability to reach Meetups, and where and when I go, is in transition. One of the reasons is that I'm no longer working in SOMA, which is where a large percentage of the Meetups are held (there or the lower financial district at times). Needless to say, with my now being in Palo Alto, it makes more sense to see what's more "local" to Palo Alto in the evenings. Thus, it was with delight that I discovered that there was a San Jose based Selenium Meetup group. Even more to the point, Santiago Suarez Ordoñez is the main speaker, talking about "Beyond Selenium" and what we as testers should have in our toolkits other than just Selenium. Tempting? Absolutely. That's why I'm here :).

Having to relearn the roads and byways to get to destinations in San Jose is always fun, and fortunately, I made it in time. It was fun to re-connect with some familiar faces, hear some of the scuttlebutt about the upcoming Selenium Conference in Boston (and my seeing if I could offer my services in some fashion to get me there… Ashley, seriously, happy to do recordings/podcasts, let's talk).

The talk tonight is "Automated Testing: It's not all about Selenium". For those not familiar with Santi, he's a regular presenter at both the SF and SJ Selenium meetups, he's with SauceLabs, and has been around the block a few times when it comes to both testing and working with Selenium and small open source projects. Oh, and he's also Employee #1 at Sauce Labs (honestly, I never knew that).

Automation or not?

Some questions need to be considered first... what is the value of quality? We want to make sure that the software will meet our customer's expectations. testing allows to reach some level of standards (by some definition). Test Automation comes into play when we want to do a number of steps repetitively or have a need to do them faster. While computers are lousy at making judgment calls, they will be more than happy to do a step 10,000 times, where a sentient human will balk at the repetition, but we are very good at judgment calls. Put together, lots of cool things can be done.

Types of Automation

Test automation happens a a lot of different levels. they are not all created equal, and they do not all use the same methods and approaches. Unit Testing and Front End Interaction are as different as night and day. The problem is, much of the automated testing ideas and ideals often come from the unit testing focus, and we need to think very differently the farther away from atomic units we get. Database and service tests are likewise different, and the tests necessary to integrate them all and look at the system holistically means we need to use all of the skills together for our automation efforts. Scared yet?

Setting Priorities

Many open source projects start ut as ways to learn, share and have fun. Quality is often a later part of the process (not saying it's not important, but early on, it's hazy and that's OK. With maturity and more focused interest, it becomes considerably more important. Also, making a product make money sometimes takes priority over quality early on. Priorities shift, and again, that's OK, but at some point, you do have to care about quality, or someone else will take your idea and do it one better than you. When you have just a few people, very few specialists, and a rapidly changing code base, automation may just not be reasonable. Outsourced projects also may be difficult to get automated, though it can certainly be done, especially if the codebase is settled.  When a projects gets to the point that automation makes sense depends a lot on who you are, what you are doing, when you want to do it, and ultimately what you and your audience values. Santi shared some of his experiences making CSSify, a tool that converts XPAST values to CSS.


Architect Your Tests

Very often, we jump in and hack and slay and make our tests to meet some immediate need. Do this enough and you have a test suite. Multiply enough times, and you might think that you have a good handle on and have a high confidence in the value of your automated testing... until something unexpected happens and the whole thing comes crashing down. Been there, done that, and more than once. Ideally all tests would be formulated theoretically, would have a clean and well understood interaction model, and tests would be easy to create. Frankly, I don't know anyone who has reached that level, but taking the time to do some architecture for your testing can pay major dividends later on. Tests mature as the codebase matures, or at least they should.

Good Test Practices

Tests are code. Treat them as such. Set up version control and use it, religiously. Get familiar with every aspect of your testing, all the tools, all the configuration options, know what it takes to make tests, load tests, write tests and run tests. Being able to debug failures is essential. For the test manager, do all you can to help make it as easy as possible for your team to do their work. Reproducibility is vital (tests break, it's going to happen, so prepare for it). Even more frustrating is when a test works 99.95% of the time. Performance is very important. Parallelize where you can, use analogues such as mocks to help you make sure your tests are doing what they should. Make your tests as simple as they can be, as often as humanly possible. Can we do all of this, all the time, every time? Of course not, but thinking about these things and giving them attention helps to diminish the possibility of a complete derail of an automation project.

CI and CD

Continuous Integration and Continuous Delivery... simply put, YES!!!  There are tremendous benefits to haviong a system that runs your build process each time a check in is made. With the unit tests being run, the components being rebuilt and the image being created (or the services being set up and restarted, there is a lot that can be done to help flag problem areas when these tests are done regularly, early and often. Putting in multiple changes and then doing a manual build can  certainly show errors, but chasing them down can be vexing, as well as time consuming. Having a CI server like Jenkins to auto-run with each check in can help you know very quickly if your changes will be good, or if they will break the build (and if you will have to have your picture on the wall of shame :) ). Continuous Delivery is a special animal, and it's not for everyone, but for those who are willing to commit to the practice, the principal is the same. The more atomic the changes, the easier it is to tell what causes problems, and where. It's key and valuable if done right; it can be tremendously frustrating if it's done wrong. If you have low test coverage, or a low opinion of your test coverage, do NOT do Continuous Delivery.

Some Common Test Techniques

Santi took some time to discuss TDD, and the fact that he's not really a fan of the approach himself. He feels it's valuable if done with abstractions, such as using tools like Page Object Model or using an abstraction language like Cucumber to do inside out development. It's also important to not discount Manual/Exploratory testing. Yes, it's high cost, but it's also high sapience, and that in turn can be incredibly valuable. Doing Exploratory Testing of a high value is hard. Be aware of that fact. Balancing the exploration aspects with the automation initiative is a fine balance, and knowing when to leverage the value of each can be huge.  


Fundamentally, the Point is...

Organize your efforts, encourage your developers write tests, focus on edge case coverage and maintenance. Don't silo your QA engineers and have them just write tests, they can't keep up and the tests will suffer. Encourage a partnership where developers write, and QA reviews, maintains and enhances. Let the teams work together, and encourage an even partnership. Santi suggests having the developers write and maintain the tests, while the QA does review, writes and works with tools, and develop and grow the frameworks used. Automation needs good architecture. MOre tests mean more complexity. Complexity demands good architecture and design. These are not trivial things, they take a lot of time and commitment.

Michael Larsen, TESTHEAD

SummerQAmp: Calling for "Champions"

The new year is underway, and with that, the plan to make SummerQAmp into a bigger and better opportunity, is likewise underway. I'm exploring a number of new topics, and with that, we have a strategy and a body of knowledge that we want to develop. With this, I want to encourage everyone out there who is interested to come join us and help shape the discussion and the content that will be presented.

We have a number of topics and topic areas that could be built up and developed. The discussion is happening over at the AST Forums. You do not have to be an AST member to participate in the forums, but you do have to be registered with the forums to post there. This serves a couple of purposes, the most important one being "if you contribute, you will get credit". Your being part of the discussion and contributing to the overall text, exercises or activities that we hope to make into the core curriculum can and will have your name on it, if you so choose to participate.

So what does it take? It's simple. Make an account if you don't already have one, then pick a topic in the EdSig section (EdSig is Education Special Interest Group). Most of the topics in that group right now are related to the SummerQAmp initiative. To get started, check out the post Kevin Haggard made a few days ago which spells out the road map that we are intending to follow. Do you think the topic list is good? Are we putting our energies in the right places? Do you think we're missing something important? Is there something that you, as a fledgling tester years back (or hey, maybe even here and now) wish you knew and had someone tell you, encourage you, or warn you about? Well, here's your chance to contribute to exactly that!

One thing to be aware of, this is a compressed schedule this year. We have a hard deadline for materials we will be presenting, and that deadline is March 1st. We need to be ready to go with what we have at that point, so if you want to take part, you need to get in and stake a claim to topics or propose new ones.

To all who have already put their time, talents and energies towards this initiative, thank you. To those who still want to do so, hey, come join us and help us make the best and most relevant software testing training possible!

When You Don't Have "The Perfect Tool"

This past weekend, I took our scouts out to Huddart Park in Woodside. We were going to go and shoot rockets at Moffett Field, but alas, the field has to be used for something else last minute, meaning, yep, the launches were scrubbed. We're going to go again this weekend, but that's not the point of this particular post.

One of the things that our Troop takes pride in is our tradition of being "backwoods cooks". Usually that means Dutch Oven's and fill in the blank recipes, and this trip was to be no exception. I have a few signature dishes I like to make, and they made their appearance, but I really had my heart set on trying something new... Aebleskiver!

For those not familiar with Aebleskiver, they are Danish Pancakes, round, often stuffed with fruit filling or jam/preserves, and then topped with either powdered sugar or dipped into maple syrup. My hope was to find at my favorite local outfitter a cast iron Aebleskiver pan (yep, Lodge makes one :) ). Alas, when I got there, they did not have it. I asked the proprietors and they said that they'd love to carry it, but it's such a specialty item and so few people ask for it, they don't carry it as part of their regular inventory. They said they'd be happy to order one for me, but of course, that wouldn't help me for this campout.

Hmmm, what to do?

Well, as it turns out, necessity is the mother of invention, and in this case, I decided to experiment with a muffin tin. The approach was similar:

• Fill the cups about a third of the way with batter
• Put in the fruit filling (in this case, blueberry preserves) 
• Cover them again with enough batter to seal in the fruit filling, but allow a little room for the dough to expand. 
• Cover the top of the pan with tin foil and make an A-frame roof of sorts, so the dough can expand and still trap the heat. 
• Place on coals for about five minutes or until the tops start bubbling. 
• Take a fork and roll the semi-cooked Aebleskiver so the bottom is now on top, and continue another five minutes.

Was it perfect? Nope, they were a little angular, of course, and they were kind of a cross between pancakes and muffins, with a jelly donut center… but to the scouts that ate all of them without taking a breath, they didn't care. They thought they tasted fabulous, and asked me if we could do them again the next campout.

Testing parallel? Of course!

We often lament the fact that we don't have the "perfect tools" to do our work. Oh, if we only had this item or that item, then we'd be able to really get something accomplished! My Aebleskiver would have been much more nicely formed had I had a proper pan. The texture and appearance would have been much better, instead of being slightly jagged and bumpy, and with the filling slightly off center. At the end of the day, though, no one really cared about that. They had something different to eat and they enjoyed it. End of story.

Likewise, we also sell ourselves short if we think our efforts are not worthwhile without "the proper tools". Sure, the tools make things easier, but the idea is often still the same; sometimes, new discoveries can be made when you have to wing it and do things without the perfect tools. In the future, if I don't have an Aebleskivber pan, two sandwiched muffin tins, with coals on both sides, would do splendidly. Of course, the purist in me wants the right tools, too, and make no mistake about it, by the time we are at a campground again, I will have a proper Aebleskiver pan… but it's nice to know that I can still do a pretty good job without it if I have to.

Free Kindle Tech Books (And Other Genres)

I have a Kindle Fire. It's been a good little tool, and I have to admit, I've used it more than I imagined I would. From being my "thou shalt not copy and paste" window to being my place to go and play with a handful of walled-garden apps (yes, I could break it and load Jelly Bean or something else on it, but frankly, I'm OK with it being a Semi- Android E-Reader for the time being). Also, let me make a note that, right now, this is a standard Kindle Fire, not the HD. I may upgrade in the future, but for now, again, it meets my needs.

What I wanted to do was start looking to see what was available for the Fire and, more importantly, see what I could find for free. Since a lot of my focus is on tech books at the moment, I decided to see what I could find in that avenue, and that led me to the Free Kindle Books and Tips Site. More importantly, it led me to this particular page:

Free Kindle Books and Tips: Free Book LinksMany of these areas are of semi interest to me. Truth be told, I'm not much of a fiction reader. Given a choice, I much prefer reading non-fiction, especially stuff that centers around world history, science, exploration, world cultures and philosophy. Computers and Internet are a sort of essential grab bag for me, so that's where I look most frequently.

Yeah, I know, when you see "Free", very often the response is that "you get what you pay for". This is somewhat true in that a lot of the titles are, well, of questionable quality or are older, but some of the titles I've found have proven to be worthwhile and somewhat interesting. Right now, these are a few I have read or am reading (if there are links, it means that, as of this writing, they were still listed as available for free):

The Past, Present and Future of JavaScript

jQuery: Just the Basics

CSS and Documents

What is Node?

What is HTML5?

Most of these titles are short, like in the neighborhood of 32 pages or so. The jQuery books is full length, and so far, looks to be a useful title. Again, the listings will likely change week to week, and again, some of them are worth what you pay for them, but if you want to get your hands on some quick and potentially interesting reading (not to mention FREE), then this might be a good service to bookmark.

Book Review: Selenium Testing Tools Cookbook

As the experience level with Selenium and other open source automation tools grows, naturally there will also develop more questions. How would I effectively access elements? What's a good way to compartmentalize tests? Howe can I make my tests extensible? What's the best way to parallelize tests? What if my project doesn't use a standard approach like so many others? What if I'm not on Linux, or don't use Java?

The idea and benefit of a "Cookbook" approach is that it allows developers and testers to look at the aspects that they need at that time, and see how they work in example formats, and then see how they relate to other topics. Packt has decided to take this approach with Selenium by publishing the Selenium Testing Tools Cookbook. In it, author Unmesh Gundecha has broken down thirteen areas of interest (eleven in the book itself, plus two additional areas as bonus download sections), and presents small sections and ideas to allow the developer or tester to leverage the ability to create tests, use the Selenium API, and test applications such as browsers and mobile applications. The book demonstrates examples primarily in Java, but several are provided that use C#, Python and Ruby.

For the tl;dr crowd, if you have never used Selenium before, this should not be the book to start your exploration. I would suggest reading David Burns "Selenium 2 Testing Tools Beginner's Guide" or Alan Richardson's "Selenium Simplified" first. David's book is also a Packt Publishing title, and uses many of the same documentation standards, so it would feel very natural looking at section's of David's book, and then coming back to try out more examples in Unmesh's Cookbook here.

As this is a Cookbook style title, I must give a disclaimer that I have not yet tried every single example in the book. For that matter, I haven't gone through even a minority of the examples, and that's OK. Cookbook's are not designed to be read from cover to cover, and neither is this one. It is expected, and intended, that the user pick an area that interested them, and explore that area. Different projects have different needs, and being able to skip around will help focus the reader's time on the areas that matter most to them at that given time.

Each chapter uses a similar structure, with an introduction to explain the tools and calls that will be used in that section. The given recipe is described  and gives the reader an idea as to what can be done (along with sample code for each of the recipes, which can be downloaded from Packt's site if you prefer to see the projects in their entirety, and would rather not have to type out the examples from the book directly. Each item then goes through and describes what you will need to accomplish the project at hand (if you need tools like Firebug or Intellij IDEA, it tells you what you need and where to get them). The recipe concludes with a How it Works section, and in many cases, additional recipes that relate to the one you are working with.

The book starts out with locating elements, and the different ways in which that can be accomplished (CSS, XPath, text values, jQuery, locating table rows and cells, etc.).

Next the Selenium API is explored and the various methods available to examine and determine what you are looking at (text, attributes, CSS values, interacting with the mouse and keyboard, using JavaScript, capturing screenshots, drop downs and menu items, even going in and changing values in the Windows Registry if desired).

Chapters Three and Four will be important to testers especially, since these chapters deal with test flow and ways to create data driven tests. Topics such as waiting for elements, handling pop-ups, determining if elements are the correct state, working with JavaScript alerts, handling frames and iFRAME's, plus using a variety of frameworks to interact with tests and set up data driven tests in these frameworks are explored, with examples for Java, win32, Ruby and Python.

Chapter Five talks about the Page Object model and how to use it when developing your tests in the languages covered in the book (Java especially, but examples also show how to use .NET, Python and Ruby).

Chapter Six gets into some examples of how to extend selenium and shows some in depth examples using Java (examples include making an extension class for web tables, jQueryUI Tab widget, creating an object map for Selenium tests, capturing screenshots and comparing images).

Chapter Seven Shows how to perform mobile testing using Selenium, and some of the tools and ways to automate tests on iPhone an Android devices using RemoteWebDriver and other tools, such as the iWebDriver App to be used on a simulator or real device, or AndroidDriver on an android Simulator or device)

Chapter Eight demonstrates some examples of how to get a handle on the Performance aspects of your sites or applications. Using timers, accessing and using BrowserMob proxy, and integrating with dynaTrace or HttpWatch are likewise covered. For the Ruby users among us, an example with Watir-WebDriver-Performance is demonstrated.

Chapter Nine covers HTML5 and some of the unique tools provided with HTML5, JavaScript and CSS3, such as video widgets, canvas and web storage, and how to interact with those elements.

Chapter Ten shows a number of ways in which a user could Record a Video of a Test (not record and playback of a test, actually recording an instance of the test being run as a video file, so it can be reviewed later. Tools such as the Monte Media Library (Java), Microsoft Expression Encoder (.NET) and Castro (Python) are demonstrated.

Chapter Eleven discusses Behavior Driven Development (BDD). There are several different books that go into this topic at length, so for more on how this works and how to leverage that aspect, you will definitely want to read more than this single chapter. Still, if you are interested in how BDD can be performed in different environments, this will be of interest. The goal of TDD is meant to be a way to write test cases in a more natural language that those who do not program can both read and, in some cases, create themselves. Ruby users will find examples using Cucumber and Cabybara. Java users will see examples using Cucumber-JVM and JBehave. .NET uses will see examples using SpecFlow.NET.

Two additional chapters, Integration with Other Tools and Distributed Testing with Selenium Grid are not included with the book itself, but can be downloaded from the Packt site. these deal with integrating with tools like Eclipse and IntelliJ IDEA, as well as using Ant and Maven for Continuous Integration, as well as parallelization of tests using Selenium Grid.

Bottom Line:

We now have a number of titles to help users get into the world of Selenium and SeleniumWebDrier. Each of these titles, by necessity, can only go so deep and with so many examples. For many, that would be fine. For those who want to know "what's next" or "where could I take these ideas and expand on them" or even "can you give me some ideas as to where I might use the variety of options?", then Selenium Testing Tools Cookbook may be a great next step to explore. One thing's for sure, there's plenty in this book to keep the Selenium enthusiast busy for quite some time.

Does it Really Do What You Think It Does?

Yesterday I had one of those interesting experiences. When I say "interesting", that's often a euphemism for "disquieting discovery that I need to do something better". Yeah, that's exactly what I mean by it.

Case in point. We had a convoluted story with a lot of test examples we had to go through. We raked this sucker over the coals, and we found all sorts of stuff that was fixed, refactored and, by one of our developer's admission, "it took me longer to explain what we did than it took me to write the whole thing". It's a good bet that, in those circumstances, you may find that there are holes. I was prepared for holes. I wasn't prepared for the question I did receive, though...

"So, how can I make a clear and specific choice that shows the effect of this change, and in a way that I can explain it to a customer?"

It seemed so simple, such a clear and easy thing to do... and yet, I realized that I was struggling with this, because our Quality Director, as he walked me through this particular scenario, kept showing me things that would effectively get in the way of doing what we wanted to demonstrate (browsers with the way they cache and store items, the differences between browsers, etc.). Each time, I was starting to feel my confidence fade a little bit. I went from thinking I was very confident that we were on good ground with our testing to feeling rather uneasy. What did we miss?

It wasn't what we missed, but what we didn't articulate. The developer and I did so much back and forth with the story that we, at the time, knew we'd run the story through its paces, multiple times. What I didn't do was give enough of an indication of what, specifically, could be told to our support engineers so that, in the event of explaining the story changes, they would easily be able to explain the pros and cons to our customers.

I mention this because, sometimes, there's a danger we'll lose the balance between being lean and fast, with "just enough documentation" to be effective, and explaining enough so that those who see stories after we have finished them have the right amount of information to likewise be effective. The key takeaway from this experience was to say "can you, in a short but succinct way, explain to anyone who reads this story days, weeks or months after the fact, exactly what it is the story is accomplishing, and how to, on the system, show exactly what a change is doing? If you can do that, then you can go lean and mean with documentation. If you can't, then adding more and more documentation will probably not make the story any more understandable".  This hearkens back to my comments a few days ago about "implicit instructions". We know what they are, the developer may know what they are, and the Product Manager may also know what they are. If, however  any key individual in that line and beyond does not know, then there's a problem, and it can be exacerbated the farther away from the implicit understanding we get. It just goes to show that even veteran testers may have to do a gut check every now and again to make sure that they can simply and directly explain what something does, and be sure  that it really does do what we think it does.

The Best Podcast We Are Going To Put Out This Week (By Some Definition Of Best)

Today marks a milestone. This morning, Episode 131 of This Week in Software Testing has been posted. In it, we conclude the series of Matt Heusser's keynote adress at the Pacific Northwest Software Quality Conference (Part 1 and Part 2 can be obtained here).

This also represents 125 episodes that I have personally produced (I came onto this project back in July of 2010, and my first show, Episode 6, was posted on Aug. 6, 2010).

It's been a lot of fun, I've learned a lot along the way... and you can probably tell with this buildup that there's a change in the air.

There is.

Matt and I have decided that, for now, we are going to put the TWiST podcast on hiatus. No, there's no bad blood between Matt and me (as a musician, I am very familiar with the first question asked whenever a project goes on hiatus; it's assumed that there must have been a fight of some kind). We decided to take stock of what we have covered, who we have talked to and areas that may yet to be covered, and decided that now was a good time to take a breather after two and a half years of weekly production work.

We both love producing this podcast, so for those thinking it's going to go away forever, rest assured, that is not our plan. We both, however, want to take the time to focus on some other opportunities for a bit, and give those projects a chance to get some atmosphere to breathe (there are, after all, only so many hours in a given week that we can devote to those opportunities).

To those who have helped make this a fantastic journey, my thanks. To Matt, first and foremost, for being willing to be in the hot seat week after week, in one way or another, either in interviewing, gathering talent for discussions, or recording "on location" events along with me.

Thanks to our recurring "panel guests" over the past year plus, who helped us make for some great conversations, with special thanks to Benjamin Yaroch and Wade Wachs for their frequent contributions and going above and beyond the call so many times.

Thanks to everyone we have interviewed over the past two and a half years. It's your stories and your experiences that make the show what it is.

Thanks to the people behind the scenes these past two and a half years who have helped with show production, editing, and distribution (Rick Baucom, Rich Hand, Thomas Ponnet, Justin Rohrman, Janette Rovansek, Mark Tomlinson and Farid Vaswani).

Most of all though, thanks go out to YOU, our listeners. You are the reason that we were able to put out "the best software podcast we were going to put out this week (by some definition of best)", week after week. We hope that you have enjoyed these presentations, and will continue to enjoy them well into the future. We look forward to coming back and following the pulse of the software testing world in the not too distant future.

Until then, thanks for listening, and keep discovering :).

Regards,
Michael and Matt