Tuesday, October 9, 2018

Adventures in Modern Testing - a #PNSQC Live Blog From #OneOfTheThree


As a long time listener to Alan Page and Brent Jensen's "A/B Testing" podcast, I consider myself to be an early contingent of being "One of the Three".  For those wondering what this means, when the show first started, they joked that really there were only three or so people at Microsoft that even bothered to listen to it. As the show grew in popularity, the joke grew to mean that there were still only three people listening, just at any one time. Longtime listeners have taken some pride in regularly referring to themselves as "One of the Three" and I'm one of those people. Anyway, that's a long buildup to me saying that I've listened to and read Alan for a number of years (plug: check out his book "The A Word" for some great thoughts on the plusses and minuses of software test automation).

Alan and Brent have been focusing on an initiative dedicated to "Modern Testing Principles" and many episodes of A/B Testing have been dedicated to these principles. Happily, Alan isn't just throwing out the ideas, he's putting it in context and sharing his recent adventures and real-world experiences getting to these principles. Alan was part of the XBox One team in 2011-2013 and he describes those two years as "the best five years of his life" ;). It was in this role that he began thinking about what would become the core of the modern testing ideas, specifically about the test org as a community.

Testers are often not all part of the same group in an organization. Often, they are working on a number of different teams in different places and often, those people are working on similar problems in isolation. I can appreciate this as I work for a small team (used to be its own company) that was acquired by a larger company (with a number of functional teams with testing groups) which in turn has been acquired by another company (with already existing testing groups). How often do all of these people communicate? Simple answer. Outside of my immediate team (three testers) and a handful of conversations with a few other team leads (I think about six total people), that is all of the "testers" I have actually spoken to in my entire organization. I know that we have something in the neighborhood of fifty or so testers throughout the entire organization. Who is willing to take the bet that we have similar challenges and issues that we could probably solve or make movement on if we were all able to communicate about them? I'd be willing to take that bet.

Something else to consider is that we need to overcome the hubris that developers can't test, or at least not as well as a dedicated tester. It's wrong and it's selling developers short. That's not to say there aren't a lot of developers that don't want to test. I've worked with a lot of those people over the years but I've also worked with several developers who were/are excellent testers. By leveraging our interaction with and encouraging the teaching and shared learning with developers,  can help foster their growth in testing skills and abilities and that can help us focus on other areas that we haven't had time to work on or consider.

Like Alan, I tend to get bored easily and I look for new areas to tickle my brain. Often, we need to be willing to dive into areas we may not be qualified for or somehow find a way to get involved in ways we may not have been able to be involved before. Alan's suggestion is.... well, don't outright lie... but if you express at least an interest in an area and show some ability or desire to develop said ability, we may be surprised how many organizations will let us run with new initiatives. That's how I became a Build Manager and the Jenkins Product Owner. Am I a Jenkins expert? Not even close, but I'm working with it all the time and learning more and more about it every day. I wouldn't have been given that choice if I just asked for it.

We should get used to the idea of Testing without Testers? Wait, what?!! Hold on, it's not what you think... or at least it's not as much of what you might think. For a variety of teams, it is possible to do testing and do good testing, without having a dedicated testing team. I get it, why would a tester be excited about that proposition? Overall, if we are not the sole person responsible for testing, we can actually look at areas that are not being addressed or might be less covered. While I still do a lot of testing, I actually spend a significant time as a Scrum Master and as a Release Manager and Build Manager. Neither of those roles is specific to testing but they are enhanced by my being a tester. Both roles allow me to actually exercise some dominion and allow me to address quality issues in a more direct way.

Traditional testing harms business goals by creating unnecessary delays, an over-focus on specification correctness over actual quality, trying to be a "safety net" for the organization, and focusing on testing as a dedicated specialty with no interaction or an isolated approach. Wouldnt we instead want to be more focused on having everyone be focused on creating solutions that are better quality from the start, where everyone understands how to test and make that a part of the process?

In Alan's model of the Modern Tester, we are more of a Quality Coach. We focus on speeding up the team. We do some testing as well but on the whole, we use our expertise to help others tests. In that process, Alan and Brent and the rest of "The Three" have gone through and vetted what the Modern Testing Principles are:

The seven principles of Modern Testing are:


  1. Our priority is improving the business.

    We need to understand where the pain points are so that we can help change the culture and way things are done. We should place priority on delivering quality and customer satisfaction over code or functional correctness.
  2. We accelerate the team, and use models like Lean Thinking and the Theory of Constraints to help identify, prioritize and mitigate bottlenecks from the system.

    If testing is the bottleneck, then it needs to be addressed. If delivery is the issue, that needs to be addressed. Find the bottleneck and figure out how to mitigate those issues if they can't be eradicated.
  3. We are a force for continuous improvement, helping the team adapt and optimize in order to succeed, rather than providing a safety net to catch failures.

    Pair testing with developers, tooling and building good infrastructure can be a big help, as well as serially working on items to completion rather than having a million things in-process that move so slowly that they don't really progress.
  4. We care deeply about the quality culture of our team, and we coach, lead and nurture the team towards a more mature quality culture.

    Test experts will be able to leverage their skills and help others develop those skills. Don't think of this as losing influence or the team losing the need for you. The opposite is true; the more effective coaching testers can provide, the more helpful they can be and as such, more indispensable.

  5. We believe that the customer is the only one capable to judge and evaluate the quality of our product.

    At the end of the day, the decision of good enough isn't ours. We need to be willing and able to let go and encourage the product owner and the customers to let us know if they are happy with what we have delivered.
  6. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.

    We need to get a better understanding of what the data actually says. What features are really being used and are important? How can we know? How can we break down what the customer is actually doing and how positive their experience is?
  7. We expand testing abilities and know-how across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.

    This is the most daring part. If we are really willing to embrace this, we might work ourselves out of a job. If we are truly good enough to do this, this shouldn't be that big of a concern because there are a LOT of organizations out there who are not even close to this. A "Modern Tester "will always be needed, but perhaps not in the same place all of the time.
As I've said, I've listened to these ideas come together over the past couple of years and it's really cool to hear this in this condensed format after all this time. Alan has a bold and interesting take on the transitioning of testing into the brave new world. It's a compelling view and, frankly, a view I'd like to embrace in a greater way.


No comments: