Monsters & Magicians: Testing the Illusions of Generative AI with Ben Simo (a PNSQC Live Blog)

Day 2 of the main program is underway (Day 3 includes the workshop presentations). It's crazy to see/feel how quickly this event goes by. As always, I've had a great time at this event and enjoyed my interactions with everyone. It's especially neat to realize how many people I know in this space, and when keynote speakers are literal friends, such as today with Ben Simo. 

Machines to do things beyond our physical or mental powers have existed for thousands of years. We can go back to the Antikytheria Mechanism for what may quite possibly be the world's first "artificial intelligence," depending on how you want to interpret that term. Over time, as we have come to grips with and developed an understanding of the rules, laws, and repeatability of activities, what was magical once upon a time has become commonplace in our everyday use. 

PNSQChronicles: Brief Interview of "Monsters and Magicians" with Ben Simo on YouTube

We now see Large Language Models and predictive text generation as the current amazeballs part of our reality. Many people are excited about these technologies, but at the same time, there are many risks surfacing, with reports of organizations suffering actual harm or damage because of using AI tools. We have heard of apps that have jacked up rates arbitrarily, published legal documents with no basis in law, fact, or reality, and taken models of "virtual people" who learn from interactions and the biases and inputs trained these models to be incredibly racist and hateful. 

These situations point to an interesting set of questions: how specifically can we as testers benefit from this wild new world of seemingly random query and response systems? How do we test software that produces inexplicable fuzzy outputs? At the end of the day, software deals with patterns and algorithms. We have technologies such as machine learning, clustering, and ways that data can be grouped and sorted. If we give LLM's a closed data set of information and ask it to work with just that information, it does a remarkably good job of transforming or "creating" work and assets. The key here is that we have given it a known and bounded set of information. Because it is bounded, it is working with a known set of information and can be guided specifically as to what to do with it. As we open it to the outer world and give it fewer controls or restrictions, we open the model up to having to look at vague clusters of data with potentially dubious provenience. A great example of this that I saw in practice during one of the workshops was with the idea of creating spec documents in markdown that would reside at the base of your document tree. By making the spec document the oracle of choice, and giving the instructions that the spec document was the arbiter of what the model was to do, we limit the chance of hallucinations and odd reactions considerably. Not completely, but we make it much easier to track what the model is doing.

An example I had fun with recently was when I heard through a podcast a story of the son of Hephaistos who went to Olympus during the waning days of the Greek pantheon's influence and took a remnant of the fire from Olympus (in an homage to the myth of Prometheus). However, something about this seemed "off", as in it was being presented as an ancient myth, but it clearly wasn't. Could we identify where the original story came from? Through various prompts, reviewing the text transcript of the story, and other clues, the LLM determined that, indeed, it was a modern story being told in the manner of an ancient Greek myth, and even noted that the delivery of the story in meter and timing mimicked closely the delivery of Hesiod. I was not able to determine who wrote the story or where to find it on the internet, but the details it did provide were interesting. Many of them felt fanciful, but all of them felt plausible. That's the danger with LLM output. Unless we are diligent, the very plausibility of the output could be accepted easily as though it were fact. Closer inspection found numerous areas that were not accurate (referencing older myths that I was aware of and had history with, but attributing individuals and characters that didn't belong there). People who may not have this knowledge or familiarity might accept what's being presented as fact because it flows so naturally and just "feels right and authoritative". 

In these cases, the Boolean PASS vs FAIL rationale doesn't work. It's not that the tests pass or fail, but that large elements do pass, but there are outputs that are "off". It's not a total fail, but it's also "corrupted" in a way that we cannot simply rely on the output. Additionally, we can run tests multiple times and get slightly different outputs with the exact same information being presented. In my own world of testing AI, we use a variety of benchmarks and monitors that help us determine if the models are behaving in ways that we expect them to. We have a variety of tests and models that allow us to determine things like Performance drift, comprehensive analysis, bias drift and disparity, the currency of the data, and comparing for homoscedasticity (a $10 word that means looking at the variance of the errors in a model and determining if it is constant/consistent across all of our observations). 

A neat tool that we have is the "AI Risk Repository" which helps us identify risks and domains of use of models where the various risks can be found. By looking at the areas that are potential risks, we can better be informed or consider aspects we can apply to our testing efforts.

Ultimately, one of the key takeaways from this talk is the idea that we are at peril of being beguiled by the magic surrounding us. We want to be responsible with our use of AI, and thus, we need to test and consider how best to apply what we learn and spread that knowledge amongst our colleagues. Magic is often sleight of hand, and it's important that we understand and can understand how that sleight of hand is performed.

Into the Depths of Risk Analysis to Improve Your Testing: an #InflectraCON2023 Live Blog

It has been a while since I've done a blog update. Granted, it's been a while since I've been anywhere so reality has been much of the same but I am currently at InflectraCON and taking notes, so you can all come along for the ride if you'd like :).

Bob Crews

CEO, Checkpoint Technologies

Our first talk is with Bob Crews and is covering Risk Analysis to improve testing. Interestingly, we have seen the complexity of software development explode over the past couple of decades. Web sites and apps have matured significantly and what they can do has increased exponentially and continues to do so. By virtue of that, sites and apps are becoming more challenging to test every day. We can't test everything, no matter how delusional we believe ourselves to be. Thus, we have to apply a different metric. We have to consider what is critical and of most importance, and then work our way down from there to "nice to have" long before we ever get remotely close to "we've done verything" (trust me, no one gets to that point, ever).

With this, we need to make sure that we have a clear understanding of what areas are most important, what risks we face, and how we are able to mitigate those risks, to the best of our ability. We can't prevent risk but we can do some mitigation in the process. By analyzing what the potential threats are, we can make sure that we put the most important situations at the forefront. Elisabeth Hendrickson often led with the idea of waking up and seeing your company on the front page of the local newspaper. What would be the most terrifying thing you could see in those headlines? If you can envision that, then you can envision what the potential risks are if your product were to fail. Odds are, we will never face anything that dire but it illustrates the critical elements that we should be alert to. By putting those horrorshow examples front and center, you have done a simple risk analysis of what could go wrong. From there, you can start to consider what would be next in line, and then consider how to mitigate those potential issues.

To be clear, risk assessment is a time-consuming process and can be as formal or informal as you want to make it. It can be an enterprise-level operation and exercise, or it can be a personal and singular effort just for our own benefit. I'm not sure how many people have pipeline CI/CD systems but much of the time, we have created tests that are independent and can run in any order. That's great for parallelization and speed but it may not be the best approach for risk mitigation. In a randomized, parallelized environment, every test is basically considered equal. Every test has the same potential to be a pass or fail and every test can stop the pipeline until it is resolved. How often do we find ourselves working on trivial tests that stop the system while something major doesn't even get run? There are possible ways to set up a prioritized run and make those the tests that get run first and cover the broadest area possible. By doing this, we can schedule and structure our tests so that they run in a criticality order. Think of it as placing your tests in folders, where those folders are rated by priority. We would of course want to run the tests in folder #1 before we run the tests in folder #9. To determine what those tests are based on that kind of hierarchy, we would need to evaluate and assign a risk assessment to each test.

By taking the time to look at a test, giving it a risk impact score, a likelihood that it might happen, and the possible frequency that it might happen, we can determine which bucket an item falls into. Also, high impact is subjective much of the time, but there are places where that subjectivity can rise from annoyance to a critical issue. Over time, we can get to the point where we might assign a weight to these tests, let's say that 99 is a top weight and 10 may be a minimal weight (I'd argue anything less than 10 may not even be worth running, at least not daily or as part of the full CI/CD commitment).

The fact is, we often look at risks as being "Acceptable". For years, Accessibility and Inclusive Design are low priority items unless legal action pushes them to the forefront. Accessibility may be seen as a low-priority item unless a big client demands it to buy your product. Then Accessibility rapidly rises to the top of your risk list. Security is always a top-level and critical area but how much is critical? If everything security related is critical, then nothing really is. Of course, we want to keep the system secure but what level is intelligent and prudent coverage and what level is overkill? To that end, we create a Risk computation, based on the classic four quadrants (urgency and impact, meaning we have at level 1 high risk and high impact, level 2 being low risk and high impact, Level 3 being high risk and low impact, and level 4 being low risk and low impact). Level 1 is of course the most important and arguably Level 3 is the next most important. Level 4 is probably not even worth our time but again, circumstances can move any of these situations into a different quadrant. This is why risk assessment is never a "one-and-done" thing.

There's a phrase called the "wisdom of the crowd" where the idea is a large group of people can determine what is important. If enough people consider an issue to be an issue, it will be addressed. It may or may not make a lot of sense on the surface but if enough people consider it important and make known the fact it is important, best be sure it will be considered and worked into whatever process is necessary to have it be addressed. The crowd is not always right but it is often a good indication of conventional wisdom. Usability often falls into this. While we may decide that a process is logical and rational, if enough users disagree with us and decide they will not use our product because of it, it will become a talking point and possibly a critical one if enough people voice their displeasure. 

Over time, we can get pretty good at looking at the risk areas we face and weigh them in order of how critical they are. We may never get to a perfect level, but we will come closer to a workable risk assessment that will help us address the most needful things and prioritize those areas over just trying to be thorough and cover everything.

From Fear To Risk: An #InflectraCON Live Blog

Next up is Jenna Charlton with a realistic look at the rhetoric of Risk-Based Testing. As many may well be aware, there's a lot of branding and promises that surround a variety of terms. Many of the phrases that we like to use have a certain comforting ring to them. Risk-Based Testing is one of them. Think about what it promises. If we identify the areas of greatest risk and test around those areas, we can deliver the best bang for the buck quality and we can do it so much faster because we are not testing every single thing. 

Sounds great, right? However, what does this ultimately tell us? We have said we care about risk but what actually is risky? We are only alert to the risks if we have thought about them. The biggest fear I have when I think about doing a risk assessment is that I have made risk assumptions from what I know and can anticipate. Is that really a good risk assessment? It's an okay and workable one. However, if I'm not able to consider or understand certain parameters or areas that may be blind spots to me, I cannot really do a great risk assessment, so my risk assessment is incomplete at best and flying blind at worst. 

One of the first things that can help ground us in these considerations is to start with a simple question... "what am I most afraid of?" Understand, as a tester, what I am most afraid of is missing something important. I'm afraid of having shallow coverage and understanding. That's not necessarily something that a general risk assessment is going to focus on. How many of us have said, "I don't know enough about the ins and outs of this system to give a full risk assessment here"? I certainly have. What can I do? Much of the time, it's a matter of bringing up my concerns about what I know or don't know and being up-front about them. "I have a concern about this module we are developing because I do not feel I fully understand it and thus, I have foggy spots here and here". Sound familiar? What is the net result of this? Do we actually get a better understanding of the components and that leads to a more lean testing plan because now we know the items better? Do we double up our coverage and focus model so we can "be sure" we've addressed everything? Here's where risk assessment breaks down and we fall back into the "do more testing, just to be sure" approach.  

Something else that often doesn't get addressed is the fact that what is a risk at one point in time, as the organization matures and they have covered these areas, risk in those areas actually goes down. Still, how many of us have continued focusing on the "riskiest areas" because tradition has told us that they are, even though we have combed through every aspect of this area we consider so risky. If you have made tests for a risky area, you've run them for an extended period, and no problems have been found (the tests pass all the time), what does that tell us? It could tell us we have inadequate tests (a real risk, to be sure) or it could also tell us that this area has been thoroughly examined, we've tested it vigorously and now we have a system in place to query multiple areas. In short, this area has been moved into an area where it might be risky if something blows up but as long as it doesn't, the risk is actually quite low. Thus, we now have the ability and the need to reassess and consider which risks are the current ones, not yesterday's.

We have to come to grips with the fact we will never cover every test possible and as such, we will never fully erase the risk. Also, we will never get it perfect. Still, we often operate under the assumption that we will be blamed if something goes wrong, or that we made bad assumptions, and of course, we fear the retribution if we get it wrong. Thus, it helps to see how we can mitigate those fears we have. If we can quantify the risk and define it, then we can look at it objectively, and with that, we can better consider how we will address what we have found. Are afraid of an outcome (nebulous) or are we addressing the risks we can see (defined and focused)? To be clear, we may get it wrong, or we may make a mountain out of a molehill. Over time, we might get better at that. Our goal is to deal with the molehills effectively but miss the entire mountain. 

Again, there's a chance that we will miss things. There's a chance something that matters to our organization will not get the scrutiny it deserves. Likewise, fear may be making us focus on solidly functioning software over and over again because "it just pays to be safe" only to realize we are spending so much time on an older risk that isn't as relevant now. It's more art than science but both are improved with practice and observation. 

Risk Based Testing - a #PNSQC Live Blog

Its a fact of life. We can't test everything. We can't even test a subset of everything. What we can do is provide feedback and give our opinion on areas that may be the most important. In short, we can communicate risk and that's the key takeaway of Jeny Bramble's talk. By the way, if you are not here, you are missing out on Dante, the deuteragonist of this presentation (Dante is Jenny's cat ;) ).

Jenny points out off the bat that, often, words are inadequate when it comes to communicating. That may sound like unintentional irony but I totally get what Jenny is saying. We can use the same words but have totally different meanings. One of the most dangerous words (dangerous as in its fluidity) is "risk". We have to appreciate that people have different risk tolerances, often in the same team. I can point to my own team of three testers and I can feel in our discussions that risk is often a moving target. We often have to negotiate as to what the level of risk actually is. We get the idea that risk exists, but how much and for whom is always up for discussion.

Jenny points out that risk has a variety of vectors. There's a technical impact, a business impact, and a less tangible morale impact. When we evaluate risk, we have to determine how that risk will impact us. What is the likelihood that we will experience failure in these scenarios? I often have these discussions when it comes to issues that I find. Rather than just come out and say "this is a bug!", instead, I try to determine a consensus of how bad this issue might be. This is often done with discussions with our product owner and asking questions like "if our customers were to see this, what would your impression be?" I likewise have similar discussions with our developers and often, just asking questions often prompts people to look at things or to have them say "hey, you know what, give me a couple of hours to harden this given area".

Risk isn't always limited to the feature you are developing at the given moment. A timetable changing is a risk. Third party interactions can increase risk, sometimes considerably. If your infrastructure is online, consider where it is located (Jenny is from North Carolina and as many are probably aware, we recently had a hurricane sweep through that made a mess of Eastern North Carolina. Imagine if your co-lo was located there.

Ultimately, what it comes down to is being able to perform an effective risk assessment and have a discussion with our teams about what those risks are, how likely they are to happen, and ultimately how we might be able to mitigate those risks.

Jenny has a way of breaking down a risk matrix to make it a numerical value. By looking at the level of likelihood with the level of impact, multiply the two numbers and that gives you the risk factor. A higher number means higher risk and higher efforts to mitigate. Lower values mean lower risk and therefore lower cost to mitigate.

"This feature has been drinking heavily and needs to go to rehab!" Best. Risk. Based. Metaphor. Ever (LOL!).

This is my first time seeing Jenny present, though I see her comments on Twitter frequently. If you haven't been to one of her presentations, may strongly suggest that, should she be speaking at a conference near you, that you make a priority to see her speak? Excellent, my work here is done :)!

Two Great Tastes that Go Great Together :)

StockMonkeys.com

One of the things I enjoy hearing about is when engineering teams give credit to their testing teams and customer support teams. Both play important roles. In fact, it’s common to see software testers and customer support people wear each others hats. I personally spent a few years in my career doing both roles simultaneously, so I appreciate how, in many ways, customer support and software testing are two sides of the same coin.

Imagine my smile when I read "The Incredibly Obvious Secret: Finding (and Fixing) Product Bugs Through a Close Relationship Between Test and Customer Support”, written by Jay Kremer and posted to the Zoozk Engineering Blog. How, might you ask, would I think to be looking at an online dating web sites engineering blog? For that, I have to thank the great group of software testers who attend the Bay Area Software Testers Meetup group. One of them is the Test Manager at Zoosk, and he told me about this article and suggested I have a look. OK, yeah, that test manager is Jay ;).

In any event, I felt a lot of kinship with this post, and I wanted to have a chance to reply to it and say that, yes, there are organizations that do have this kind of relationship in addition to Zoosk. I work for one of them.

At Socialtext, if a  customer reports an issue, the first line of contact is a real human being who walks through the issue with them. The support engineer can often see the the potential value of the bug in question (it’s real, it’s reproducible, it’s potentially impactful, and it has a high probability of affecting a number of people). At this point, that support person can contact us, or we can go over to them and say “hey, anything interesting coming in?”, and they will tell us what they are working on.

Socialtext has the approach that bugs get on the Kanban board as soon as we can identify and reproduce the issue (again, the easier it is to quantify the problem, the easier it is to fix, generally speaking). It is not at all uncommon to have a bug be reported one day, scheduled in the Kanban and picked up on the second day, a fix committed and tested, and then merged to our staging environment in short order thereafter. We are a company that runs most of our businesses processes on our product. This makes us, essentially, the primary alpha and beta testers, which in turn allows us to address bugs quickly, at least most of the time. It seems that Zoosk has a similar philosophy.

I have often said that customer support engineers make for awesome testers, and software testers can often make amazing support engineers. Like Jay, I believe this is a symbiotic relationship that needs to be enhanced and encouraged, and more testing organizations should make that connection and work closely with their support teams, not just for when they report issues. Software testers can learn a lot about the true state of their product and the workflows that really matter by spending some quality time with the support team. I think you will be pleasantly surprised with what you learn.

Test What Matters: 99 Ways Workshop #98

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #98: Test what matters - Rosie Sherry

Ahhh, it all comes down to that, doesn't it? It's simple, elegant, really easy to understand, and yet, try as we might, it's so very hard to actually do (no, seriously, it is). See, what matters is tremendously subjective. Who are we talking about? Are we talking about our end users? Our management team? Our co-workers? Our shareholders? 

We'd love to believe that each and every one of those groups are aligned in purpose and intention, that they would all want the same things, and that what maters to one matters to all. Sadly, that's not true. Thus, to make sense of what matters, we first have to make a solid determination as to "who" matters.

Workshop #98: Get a feel for the five biggest customers that your organization wants to keep happy. Hint: they may not be end users of your product. Once you find them, get to the heart of the matter and discover what really matters to them (ask informally if you can't get direct answers from the one's who call the shots). Then structure your testing regimen to focus on what matters to those people (hint: they will not always be aligned).

A famous phrase that many of us have heard over the years is "Quality is value to someone who matters" (thank you Jerry Weinberg). Therefore, what matters is what we can identify that is important to the person or people that matter. Those people can shift, and they can have significantly different goals. Therefore, what should we do? Do we skip around from person to person and find out what matters most to them, and make sure that we deliver it to them? We could, but I would also hazard that it would make us look schizophrenic, and quite possibly untrustworthy. Quality is, indeed, "value to someone who matters" and the "who matters" part can be hard to pick out at times.

Therefore, rather than a fragmented and manic rush to figure out what is most valuable to any one person at any given time, I'd much prefer to go at it from another route, which is to provide information that will help those people that matter make the best decisions they can. I'm not there to "check off the list that makes my manager happy" or "work on the story that makes the director of development look good" or " deliver under cost or ahead of schedule so that we can maximize sales ahead of the upcoming holiday season". Those things are all valuable, and they are all, in their sphere, important. If I choose to chase any one of those, I will be doing a disservice to everyone else who matters. So what should we do?

It comes down to what I feel is the fundamental thing that testers do, and it's not find bugs, or prove that software is "fit for use". Instead, it's to provide information about the state of the product in ways that are meaningful, and to let those in other positions in the organization make the best decisions that they can based on what we have aggregated, analyzed and synthesized. In the end, the development team really doesn't care how many test cases I ran if I didn't find the issue that is most embarrassing to them. The CEO doesn't care that I was meticulous and covered multiple testing scenarios if, when they stand up and give the demo to customers, the program crashes. The customer doesn't care how many features were delivered if the one that they actually care about still doesn't work. 

We need to be more focused than that, and we need to contribute to more than just working in our predefined box and testing what we are told to test. If we are information providers, than we need to be bold and brave enough to provide information. Even when it isn't convenient. Even when it may embarrass some people. Even if it may mean we have to announce a delay. If we try to please all entities, we will end up pleasing none of them. If we are honest, and show integrity, we may still not make a whole lot of people happy, but we will do one thing for certain… we will be providing the key information for all parties to make the best decision possible. If we truly believe that is what matters, then that is what we need to deliver. That information, the kind that helps make an informed decision.

  

Bottom Line:

So much of what we do is laced with politics, cronyism, and what I often refer to as a "perverse reward system" that tends to honor the short term benefits over long term health. If we focus too much on the short term goals, we can win many battles, but ultimately lose the war. We can paint ourselves into a corner, and have no way to get out without causing a mess. Pick whatever metaphor you want to, but realize that what will please one person may royally irritate someone else. Quality works the same way, and playing sides will ultimately win you few friends. Instead, pledge to make the story, the whole story, the most important thing that you can deliver. By doing so, you can make sure that you are delivering something of real value, and value that will last. Ultimately, that is what really matters, so go forth and do likewise :).

You Won’t Catch All the Bugs, and Not all the Bugs You Raise Will Get Fixed: 99 Ways Workshop #96 & 97

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #96: Be prepared, you won’t catch all the bugs, but keep trying - Mauri Edo

Suggestion #97: Be prepared, all the bugs you raise won’t get fixed - Rosie Sherry

This is really two sides of the same coin, so it pays to focus on them together. In the world of testing, nothing rolls downhill faster than "blame". If everything goes great, the programmers are brilliant. If there are problems in the field, then the testers are marched in and demands made to know why we didn't find "that bug". Sound familiar? I'm willing to bet it does, and if it doesn't sound familiar, then count yourself very lucky. 

This comes down to the fact that, often, an unrealistic expectation is made of software testers. We are seen as the superhuman element that will magically make everything better because we will stop all problems from getting out. Show of hands, has that ever happened for anyone reading this? (crickets, crickets) … yeah, that's what I thought.

No, this isn't going to be about being a better shield, or making a better strategy. Yes, this is going to be about advocacy, but maybe not in the way that we've discussed previously. In short, it's time for a different discussion with your entire organization around what "quality" actually is and who is responsible for it.

Workshop #96 & #97: Focus on ways to get the organization to discuss where quality happens and where it doesn't. Try to encourage an escape from the last minute tester heroics, and instead focus on a culture where quality is an attribute endorsed and focused on from day one. Get used to the idea that the bug that makes its way out to the public is equally the fault of the programmer(s) who put it there, as it is the testers(s) who didn't find it. Focus on maximizing the focus of quality around the areas that matter the most to the organization and the customers. Lobby to be the voice of that customer if its not coming through loud and clear already.

Put simply, even if we were to be able to catch every single bug that could be found, there would not be enough time in the days we had to fix every single one of them (and I promise, the number of possible bugs is way higher than even a rough estimate could give). The fact of the matter is, bugs are subjective. Critical crash bugs are easy to hit home. Hopefully, those are very few and far between if the programmers are using appropriate steps to write tests for their code, use build servers that take advantage of Continuous Integration, and practice proper versioning. 

There are a lot of ways that a team can take steps to make for better and more sable code very early in the development process. Contrary to popular belief, this will not negate the need for testers, but it will help to make sure that testers focus on issues that are more interesting than install errors or items that should be caught in classic smoke tests.

Automation helps a lot with repetitive tasks, or with areas that require configuration and setup, but remember, automated tests are mostly checks to make sure a state has been achieved, they are less likely to help determine if something in the system is "right" or "wrong". Automated tasks don't make judgment calls. They look at quantifiable aspects and based on values, determine if a should happen, or if something else should. That's it. Real human beings have to make decisions based on the outcomes, so don't think that a lot of automated "testing will make you less necessary. They will just take care of the areas that a machine can sort through. Things that require greater cognitive ability will not be handled by computers. That's a blessing, and a curse.

Many issues are going to be state specific; running automated tests may or may not trigger errors to surface, or at least, they may not do so in a way that will make sense. Randomizing tests and keeping them atomic can help with the ability to run tests in a random order, but that doesn't mean that the state that will be met when the 7,543rd configuration of that value on a system is met, or when the 324th concurrent connection is made, or when the access logs record over 1 million unique hits in a 12 hour period. The point here is, you will not find everything, and you will not think up every potential scenario. You just won't! To believe you can is foolish, and to believe anyone else can is wishful thinking on steroids. 

Instead, let's have a different discussion.

- What are ways that we can identify testing processes that can be done early as possible? 

- Can we test the requirements?

- Can we create tests for the initial code that is developed (yes, I am a fan of TDD, ATDD and BDD processes)?

- Can we determine quickly if we have introduced an instability (CI servers like Jenkins do a pretty good job of this, I must say)?

- Can we create environments that will help us parallelize our tests so we know more quickly of we have created an instability (oh, cloud virtualization, you really can be amazing at times)?

- Can we create a battery of repetitive and data driven checks that will help us see if we have an end to end problem (answer is yes, but likely not on the cheap. It will take real effort, time and coding chops to pull it off, and it will need to be maintained)?

- Can we follow along and put our eyes into areas we might not think to go on our own in interesting states (yes, we create scripts that allow us to do exactly this, they are referred to as "taxis" or "slideshows", but again, they take time and effort to produce)?

- Can we set up sessions where we can create specific charters for exploration (answer is yes, absolutely we can)?

- Are there numerous "ilities" we can look at (such as usability, accessibility, connect-ability, secure-ability)?

- Can we consider load, performance, security, negative, environmental, and other aspects that frequently get the short end of things?

Even with all of that, and even with the most dedicated, mindful, enthusiastic, exploratory minded testers that you can find, we still won't ferret out everything. Having said that, if we actually do focus on all these things early on, and we actually do involve the entire development team, then I think we will be amazed at what we do find and how we deal with them. It will take a team firing on all cylinders, and it will also take focus and determination, a willingness to work through what will likely be frustrating setbacks, lots of discoveries and a reality that, no matter how hard we try, we can't fix all issues and still remain viable in the market. 

We have to pick and choose, we have to be cautious in what we take on and what we promise, and we have to realize that time and money slide in opposite directions. We can save time by spending money, and we can save money by spending time. In both circumstances, opportunity will not sit still, and we have to do all we can to somehow hit a moving target. We can advocate for what we feel is important, sure, but we can't "have it all" No one can. No one ever has. We have to make tradeoffs all the time, and sometimes, we have to know which areas are "good enough" or which areas we can punt on and fight another day.

Bottom Line:

No matter how hard we try, no matter how much we put into it, we will not find everything that needs to be found, and we will never fix everything that needs to be fixed. We will have plenty to keep us busy and focused even with those realities, so the best suggestion I can make is "make the issues we find count, and maximize the odds that they will be seen as important". Use the methods I suggested many posts back as relates to RIMGEA, and try to see if many small issues might add up to one really big issue. Beyond that, like Mauri said at the beginning, just keep trying, and just keep getting better.

Try to Find Problems and Victims: 99 Ways Workshop #95

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #95: Do not try to find errors or bugs - try to find problems and victims. Testing is more than checking. - Thomas Lattner

Wow, that's a provocative statement, and it's one I agree with 100%. One of the things that happens a lot is that we end up couching "issues" and "glitches" into safe words like that. We don't talk about "problems" or "bugs" or "catastrophic failures" because they might have the potential to hurt someone's feelings. We are encouraged to report dispassionately, to stick with facts and to couch words in ways that are not inflammatory.


For the long term survival of our jobs and sanity, that may well be appropriate. However, when we test, drop the polite shtick. As Steve Martin quoted many years ago, 'Comedy is Not Pretty". Neither is testing, if it's done right. We don't have to speak to others like murderous pirates, but we sure had better do so to the code we hope to test, otherwise we will miss something important.

Workshop #95: Exercise the product you are testing with an air of "what's the worst thing I could do here?" Think of a person that could be harmed, and how. Do your absolutely diabolical worst to see if you could expose the most sensitive aspects of their data and exploit it. Then write it up (in whatever dispassionate verbiage you choose) and lay it on the table. See who stands up and reacts.

Yeah, these are getting a little more tricky to write here at the end, I will admit it. I already feel like I've said this several times in other workshops, but I'll say it again here. We do our best persuading when we can make the bugs we find personal, when the programmers and product team can most directly empathize with the pain. Therefore, set yourself up so you can really bring the pain (or give it your all trying).

- Create a persona.

- Make it as detailed and as data rich as you can.

- Give this person a back story, and as much "dirt" as you would want to keep hidden.

- Then do everything you can to expose that dirt (or have one of your teammates try to do it).

Some of you are saying I'm taking "bug hunting" and I'm creating an inversion. In a way, yes, that's exactly what I am doing. I'm approaching the application from the aspect that I want to learn all I can about that person, and I want to do so with whatever restraints I can configure. The more restraints, the more aggressive I want to try to overcome them. 

Can I determine a password? 

Can I sent HTTP requests that will send me back raw data in clear text? 

Can I get to their credit card information? 

Can I order things on their behalf? 

One of the oft heard phrases is "well, no user would do that". They are right; no normal user we have ever envisioned that would be friendly to our product would aim to do such things. We're not testing for nice people. We are testing to help us thwart truly rotten, obnoxious and dangerous people. If we could put a human face and emotion to the issues we find, we will get much more attention than if we have some abstract, corner case feeling bug. Think about it, what's going to draw your attention more, someone saying: 

"in this obscure case, where I entered in the same password 700 times, I was able to throw an exception and leave the machine in a bad state" 

or 

"hey, check this out! Running this looped bash script and cURL, I was able to clobber the machine, get to the database prompt, and I now have the credit card information of all our customers. Yee Haw, who wants a Harley?!!" 

Bottom Line:

Sometimes it's a little to easy to get into the mindset of focusing on the features we are testing, but neglect the risks that might reside around those features (and yes, that's a point I made in another post a few days ago). For many, though, as Roland Orzabel* so aptly put it in "Goodnight Song", "Nothing ever changes unless there's some pain." It's up to us to help our customers see the potential pain, and  prove that it is possible. Not an abstract pain, but one that has a person's name, face and anguish behind it. The more you can make the hair stand on end for those discussing your findings, the better :).

* Tears For Fears, Elemental, 1993