Is There Room for Test Strategy in Agile? (#PNSQC2021 Live Blog)

Suffice it to say, today is going by quickly. We are already up to the lunchtime keynote and immediately following, it's go-time for me! However, before that, we get to hear Mirjana Kolarov talk about "Is there Room for Test Strategy in Agile? It's funny to hear how Mirjana, like so many people in software testing, came to it from an oblique angle. Are there any testers that actually chose it as a first career ;)?

So let's start with the base question... what do I think when  I hear the words "Test Strategy"? 

Honestly, I think of pre-year 2000 testing projects with the need for large documents that nobody actually reads to sign off on testing in test pans that no one actually does to meet standards that no one actually checks. 

Sounds cynical, huh? 

Well, that heavy-duty documentation by its very existence made testing more difficult than it needed to be Let's be clear, though, I'm complaining about the documentation and the heaviness of it by some arbitrary mandate. A test strategy on its own I'm actually very much in the "let's do this" camp.

 

So what is the actual benefit of having a test strategy? It's not to meet that arbitrary "fill in the template" need of middle managers. Instead, it should be looked at as:

• What will guide our testing? 
• Do we have a common understanding of our testing processes? 
• Do we know what our deliverables are? 
• What are the expectations of the given test strategy?

Like Mirjana, I would encourage limited test strategies for given projects and areas of influence. rather than have some overall encompassing test strategy for an organization or a project, how about we focus on a given project or instead have a strategy for a specific story? Mirjana points to a larger overall strategy, so if I were to gather multiple test strategies into one document, then yes, I could imagine them being the same size (or roughly the same). 

An Example Test Strategy outline 

Mirjana points out that writing a test strategy in isolation is a task doomed to failure. It needs to be informed by the needs and the goals of the group, the project, and the over-arching organization. Having a single test strategy for an organization may or may not make sense. In some cases, a large test strategy is irrelevant to incremental Agile development. That's not to say there is no need for a strategy but instead, having a modular test strategy that can be addressed for each area both as needed and as defined can come into play. Again, having twenty different test strategies for multiple areas of interest may be a little different compared to one large document in volume, time, and effort but if a test strategy for a particular area is one or two pages, it's much more likely that those pages will actually be read and reviewed. 

Additionally, there are other ways to represent a testing strategy beyond a standard document, table, or spreadsheet. Mirjana makes the case that simple mind maps may be able to communicate the same level of information that a more dense document would. 

Ultimately, the level of trust and understanding of the team will determine how these test strategies are both communicated and documented. Information Radiators can be as simple as a page in a wiki or post-it notes on a whiteboard. The key takeaway, at least to me, is not the formality of the document but the ability to communicate it effectively and quickly. The more closely knit and actively the team works, the less formal or voluminous the test strategy or any documentation needs to be. The more distributed the team, the more necessary those same strategies and guidelines be codified and visible (they still need not be overly long or arbitrarily wordy). Overall, are we trying to create an effective method for testing that we can communicate to the entire organization? If the goal is to have testing happening all the time, everywhere, in every aspect, the more people who understand and know the testing goals, the better :).

Soft Skills of Automation (#PNSQC2021 Live Blog)

 

Yay!!! It's a Jenny talk :)!!!

If you have ever been to any of Jenny Bramble's talks, you know what I mean by saying that. I always look forward to seeing Jenny do these presentations. I miss seeing her in person and hope to remedy that after the current unpleasantness is more under control.




Seriously, if you've never seen Jenny give a talk in person, she is very engaging and fun, a little bit irreverent, and always thoughtful.




The talk this go-around is about the "Soft Skills of Automation"... wait, what? How does automation have soft skills? Perhaps this is better stated as developing a mindset to deal with and talk about automation. While there are lots of coding frameworks, what is our personal mental framework? Part of the process is trying to look at how we would make something work to be automated in the first place. My approach has often been a bit of brute force:

1. Start with processes that I might need to be done repeatably.

2. Set up and identify those areas that can actually be repeated.

3. Once I have those areas mapped out, now let's think about the variable items I need to deal with.

4. Now that I have a decent idea of what I need to run, now how do I actually put it into place?

I have often joked that I am a good scientist and a good tactical individual but I'm not really lazy enough to be a good automation engineer. I feel like I always have to go through these four steps and I have to get to the level of "sick and tired" before I actually get something working. Does that mean my method is bad? Not necessarily but it does mean that I pretty much have to get to a state of "fed up" before I really strive to change what I am doing. However, once I have a sequence down, then I will refine it all day long :).

Key to Jenny's talk is the fact that automation is not all or nothing. There are variations and a spectrum as to where it is appropriate/not appropriate and necessary/not as necessary. If we are doing CI/CD, it's essential for all steps. In the Exploratory phase, it's less so.

We are moving more towards machine-assisted testing and I think that's a better starting point to talk about automation. It can be seen as this complex set of algorithms but, as I am fond of saying, if you find there are multiple commands you run together and you decide that putting them in a file and running them from one command rather than five or ten is absolutely effective and usable automation.

Jenny brings up a neat idea called the Code Awareness Scale. How much knowledge of code do we need to have? The truth is, it's a sliding scale. the areas I work within my immediate area of influence, I know quite a bit. However, when it comes to peripheral areas or apps and how they interact, I don't really know what's happening nor do I really need to.

 


The more comfortable we are with the code, the better prepared we are to test the code. This is certainly true in my world. A lot of the things that I work with are specific to looking at how data comes in and how data goes out. again, as I currently work with data transformations, the mechanical process of delivering files, setting up parameters, and verifying that the processes are run is relatively easy to automate. Processing and comparing files to see that what we started with and what we ended up with for its destination usage... that's a little bit more daunting and the area I'm mostly concerned about improving.

'One of the cool things that we can do and that I talk about in my Testability talk is getting familiar with log files and "watching them" in real-time. By watching them and seeing what comes across is that we get used to seeing the patterns, as well as what causes error conditions to surface. One of the neat skills I learned some years ago (though I still have a ways to go to really make it automated) is to see wherever we have created an error control condition and see what it will take to make it surface. Logs can help you see that. So can unit tests.

So let's say someone is already code-aware. How can we work on other areas? If we are focused on White-Box testing, what would it take to focus on true Exploratory testing sessions? How about getting involved with Usability or Observability? accessibility or Responsive Design? All of these can help us look at software in a way that is less code-dependent. In short, get into the humanity of the application in question.

Jenny highlights some principles of Automation we should consider:

- emphasize reliability, value, speed, and efficiency

- collaboration helps us determine what kinds of tests we need

- testability is a huge factor. DEMAND IT!!!

- everyone should be able to review and examine code or participate in code reviews

- automation code is production code... treat it as such!


It's important to think about what/why we want to automate. Do we want to recapture time? Do we want to be confident our releases are solid? Do we want to be sure our deployments will be successful?

Sometimes, you may find that there are areas and steps that are easier to automate than others. It's also possible that some steps will require breaking out of what you might normally do. Let's take my example of data transformation. If I have a piece of middleware that is doing the transformation steps, I may find myself spending a lot of time automating interactions with an application that isn't even my application to be testing. Sometimes, the best thing to do is to step back and see if there's another way to accomplish what I want to do. Does it make sense to interact with the middleware's UI if making REST API calls will accomplish the same task? If I need to be dealing with file comparisons, I don't necessarily care about the steps to document the transformations (don't get me wrong, at times I care a lot about that) but often all I want is the starting and ending files for comparison's purposes. Thus, a lot of the in-between steps can be removed and I can focus on performing the necessary steps to actually get the before and after files.

One of the key differences is to be able to identify the aspects of a test the system needs. Computers are very literal, so they need things like locators, labels, and actions that associate with them correctly. This is why I tend to focus on the literal steps to see what is required to get where I need to. some years ago I used the metaphor of automation less as a train track and more as a taxi route. Taxi route automation is a lot more involved than a standardized train track route but there could be some neat things to discover if you can get there. 

Often, what I will do is I will use something like Katalon and actually record my steps to get to a particular place and I will see if there are other ways to do those steps (literally creating a folder full of driving directions). Once I have those, I will run them to get me where I want to go, and then I get out and poke things manually or identify other areas I might be able to automate. Again, a lot of automation doesn't need to be as formal as specific tests. A lot of it could just be simple data population or state changes to get to interesting places.

A final statement Jenny makes is to "don't over-automate". Automating everything sounds good on the surface but over time, too much automation can add unnecessary steps and time for little benefit. Don't automate everything. Automate the right things If you get interested in some automation for the sake of it, that's okay but perhaps only check it in if it adds a tangible benefit. A lot of my automation isn't really testing, it's set-up, it's data population, it's state change. It helps me but it doesn't necessarily help the flow of testing itself. rather than automate all of the things at a user level, perhaps make background steps that set up an environment that will be effective to test and then spin it up ready to go.

To add to this, also take a look at Test Automation University. Definitely worth digging into :).

Crafting a Quality Organization Through POC’s (#PNSQC2021 Live Blog)

 

A talk with someone from The Pokemon Company?! I'M IN!!! :). I'm not sure I even care what the talk is about (LOL!).



Okay, no, that's not true, but I will confess that the nature of the company did sway me considerably to attend, so well played, Paul Grimes. Well played :).

So what's the purpose of this talk? I'm going to literally borrow from the description on the PNSQC site to set the stage here:

"Organizations often make changes without a way to evaluate if they are getting the results they want. This can be true when incorporating a new tool, making changes to communication routines, or changing management structure.

The Product quality and Insights team at the Pokemon company international uses Proof of Concepts (POC’s) direct changes we want to make. Anyone on the team can write up a POC proposal for an idea they believe will make the team better. The proposal includes what they want to change, how long the POC will take, and what questions the submitter hopes to answer. The POC is reviewed and refined to determine the criteria for success. If accepted the POC is then executed and evaluated and if successful the change is adopted."

Ultimately, a Proof of Concept (POC) takes a hypothetical idea and puts it into practice so it can be interacted with and operated. The key here is a POC is a chance to experiment before committing. It also allows people to see if the idea will actually work or if it may actually work.

The key here is that we are already pretty familiar with technical POCs. We may not be as familiar with Cultural POC's (New to Paul and honestly, new to me :) ). this is an interesting idea in that instead of focusing on the progress of tasks, it focuses on the goals and progress of people. The key to this is reflection, goals, and tasks. Thinking about it, it does make sense from a familial or relationship perspective, where we would try an experiment to see if a potential change might be beneficial before absolutely committing to it. Why wouldn't we consider it for the corporate and cultural aspects of a company?

One of the obvious questions about Cultural POC's is that they seem to be a lot more "squishy" compared to a Technical POC. That doesn't mean that it can't be done but it does mean it will have some definitely different aspects.

A POC has five specific elements:

• Description
• Purpose
• Timeline
• Hypotheses
• Measurement

I like this template and explanation as it makes it relatively easy to get a handle on what the POC is meant to be for, what the benefit is, how long the POC will run, and to see if the hypotheses we started with actually holds up as the process runs its course.


Okay, so what's an example of a Cultural POC? evaluating a tool might be seen as a Technical POC. Employing Mobbing as a programming and testing effort would be a Cultural POC. Odds are, if we just said "we are going to do mobbing starting in November" we might be setting ourselves up for failure. Instead, perhaps we might want to approach this from:


"We've seen that mobbing brings a fair amount of increased communication and better software quality outcomes. As an experiment, during the Month of November, we will be devoting Wednesdays to practice the mobbing concept and see if this helps us with our development and testing efforts. We will run this for the month of November and December, and then we will regroup and see if this is indeed beneficial or if it's not something we want to continue exploring."


The benefit to this approach as described is that it's not all-or-nothing. It's time-bounded. It allows for people to work in a more traditional way for several days and it allows for an experiment to see if the new approach will be beneficial. It's boxed on a single day a week so that it can be planned for, executed, and reflected on in a small experiment approach rather than committing whole-hog and then it becoming a new normal with no reflection or considering if it actually made sense.

I agree with Paul that  Cultural POC's should be small and focused efforts. they shouldn't be all-encompassing and they probably shouldn't be handled a bunch at a time. POC's should probably be handled one at a time to see how they fit into an organization and if they will ultimately be worth considering beyond their POC period. Also, I think that POC's are better when formed from the ground up rather than dictated from the top down. that's not to say that Cultural POC's can't be proposed from the top but I'd guess the really effective ones are going to bubble up from the people doing the actual work.

Simplify Your Life With Static Testing (#PNSQC2021 Live Blog)

 

I confess, the title alone is what caught my attention here and I decided I wanted to jump in and see what this is all about. 

"You're Working Too Hard! Can You Simplify Your Life with Static Testing?"

Okay, what exactly is this? That's what Chris Cowell is describing and how he uses it.

So let's get down to it... what specifically is Static Testing and how do you do it?

Well, the easiest way to think bout this is to get to what static testing is going to be counter to? Is this like how we used "Lint" as a tool for finding issues in C code (and yes, I am that old ;) ). Static testing looks for problems in the source code without actually running it. Thus, yes, Lint was/is a static testing tool. But static testing today goes beyond that. We can use static testing to look for problems in configuration files, XML or markup files, etc. In some cases, these issues could be fixed automatically, or at least suggest fixes and let you, as the user, apply them. 

The real question here is, of course, "can static testing replace functional testing, performance testing, security testing, or other kinds of testing?" No, not really. However, it can complement dynamic testing and as Chris puts it, static testing can help identify risks rather than bugs. 

So where can we use this? Many IDE's have static testing built-in. Any time you see some underlining or alert icons pop up in your code and as we address them as they come up, that's static testing in action. Cool, I never considered it that way! Build tools like Maven, Gradle, and Rake have static testing plugged in. GitLab and GitHub also have this built into their tools (Chris works with and instructs for GitLab, hence why he's mentioning this :) ). There are even Chrome extensions that can do this (Code Climate, never heard of it, but intrigued now).

When static test tools look at code, the tools identify ways that the code might "work" but maybe not the way we intend it to. We could call this a "code smell". Perhaps we might have a coding approach that "works" but may end up causing us problems as things get more complex. 

Static testing can't tell you if you will meet performance goals, but it can/could tell you if there are processes and methods that might add extra time or be wasteful if run as-is.

Static testing may also be able to identify where you may have security issues such as file injection or the potential for Cross-Site Scripting attacks. Code Complexity can also be examined to see if there are overly complicated methods or areas that can be refactored.

Static testing can even help with your documentation. Heck, there are even tools that can help with grammatical issues in text that we have put into the code. as a Grammarly user, I definitely think that's a cool and neat thing to have.

So what does static testing not help with? It can't tell you if your algorithm is effective. It can't really correct iffy implementations, it can't actually help meet performance goals or integration issues, so those are areas where dynamic testing will be needed. Also, Static testing might be a little bit too helpful. Many of the areas that it suggests may not be relevant for our implementation. 

Again, this is meant to be a complementary approach, not a complete replacement for dynamic testing. I can see, however, that there are some definite benefits to looking at additional static testing options. I have some homework to do :).

We are Underway: Building on Success, Beyond the Obvious (#PNSQC2021 Live Blog)

 

It's exciting to be back at the Pacific Northwest Software Quality Conference (PNSQC). Granted, we are remote and virtual for a second year but I have enjoyed being a participant at this conference for the past ten-plus years. At this point, I think I have attended and presented at PNSQC more than at any other conference. I've gone the whole route from writing papers, presenting posters, giving full talks, being an invited speaker, and being a finalist for best presentation. I've learned a lot here and I look forward to continued participation. Yes, I have a talk I will be presenting here as well, but for obvious reasons, I can't live blog my own talk ;).

Monday is starting off with a keynote being given by Eric Van Veenendaal discussing "Building on Success, Beyond the Obvious". It's important to start with a simple question, "What is success? What happens and what makes a difference?" In other words, what can we do to actually be successful without necessarily being "THE BEST". 

Eric pivots here and takes on the "No More Testers" rhetoric. Will we indeed see a day where there will be no more testers? I'd say that the real answer is "it depends". I think testing will always be relevant and testing will need to be performed by skilled people. I also do not believe that testing has to be performed by a dedicated tester. Fortunately, many of us who have come up in the trenches as software testers are also not only doing testing. We may do a lot of it, but odds are we are involved in many other initiatives that help the success of the company. 

Having been in or around testing for thirty years now, I can say that the testing landscape and the focus of my efforts have changed many times over those three decades. The basic methods I learned in many ways are still relevant but the overall approach and HOW I test, much less WHAT I test, has changed dramatically. 

An interesting comparison is made with the idea that Agile adoption has grown but the achievement of the benefits from it looks to be elusive. Doing Agile has benefits but it does not guarantee better quality software. 

As a teacher of the BBST courses, I realize that there is a large gap between what we say and espouse testers to focus their efforts on and what actually gets focused on in the real world. Even when I train and work with people to use these skills and techniques, often they fall apart in a real-world application. I'm a Senior Automation Engineer by title but I'll say that my day-to-day testing efforts are still a large majority manual and exploration based. More to the point, I get to do a number of other things that go beyond just handling testing initiatives. this may sound bleak and yet, we release, we fulfill obligations, we get software out to our customers, and they make it work for their purposes. Ideally? No, certainly not. Effectively and usably? Yes. Thus a lot of what we do and focus on answers to the various Agile rituals, truth is we make do and make it work to be effective. It's not a perfect process at all but it does work, for some definition of "work" ;).

It's one thing to "do the things that make the difference" and that's really the key here. Getting to that place can be difficult because there are a lot of turfs that are often defended vigorously. In many cases, what we do is not nearly as important as how and why we do it.  

Set Clear Priorities and Understand The Risks

In many cases, testing comes down to communicating the risks. What would happen if the worst-case scenario occurred? How would we effectively approach our testing if we placed a serious focus on the risk assessment and taking a risk-based approach to testing? This doesn't necessarily mean "more time testing" but "testing the areas that would have the greatest risk were they to go wrong". This means we need to be focused on setting priorities related to risk identification, analysis, and mitigation. Additionally, how many areas do we have specific control over, and what areas are peripheral to us yet still important? Will we be dealing with things that would have a high impact if they went sideways? How likely is that sideways issue to happen? The better we can get a handle on these two values, the better we will be at developing our risk-based test approach.

I like what Erik says here to the effect that a test plan should reside on a single page where possible. We should have a test approach that deals with where to focus and determine what matters the most and what may not matter anywhere near as much.

Review Processes and Methods to Determine what is Actually Working/Important

Shift-left is a popular term these days and a part of that is looking at ways to be involved and focused on issues that are discovered and how they may be mitigated in the future. Part of the shift-left approach is increasing visibility on testing efforts as soon as possible in the process. Reviewing past efforts and learning what we can do to be effective EARLIER  in the process is a helpful approach. To be clear, this doesn't mean review everything all the time. That's all we would be doing. The key to effective reviews is to look at specific areas, scope them so as to not be overwhelming, and work on them a little bit at a time.   

Use More Effective Unit Testing to Help Find Issues Earlier

This may seem obvious but this is not to say "do Unit Testing". Hopefully, most organizations are doing Unit Testing as an active practice. This is taking a development methodology and thinking about how and where to use it to help with the development process. The potential issue here is that many developers have not been trained in testing techniques or skills. Thus, while there may not be a need for a dedicated tester in many circumstances, that means that there needs to be a developer(s) who are well versed and experienced in testing techniques and can bring that back to their software development. To say that developers don't know how to test is ludicrous, as many developers are very good at testing. The challenge is many other developers are not. How do we bring them up to speed?

Introduce Test Design to the Development Process

One area where testers can be helpful early on is identifying test conditions and test situations early on. In many cases, testers will not be able to address every area upfront or early in a development cycle. Over time, though, we will develop familiarity and we will have an understanding of the areas we have already tested and worked with, reviewed, and evaluated with the team. This will allow us to consider testing, testability, and risk areas early in the development process and ultimately make it possible for us to address testing in an effective manner. Exploratory testing is important in the early learning stages. It's important at many other times but it can be make-or-break in these early phases of development. this is the time where we can address areas like testability, accessibility, inclusive and responsive design.

Build Experienced and Skilled Testers, Whoever Those "Testers" Happen to Be

When we talk about training the testers, realize that there may be many different people testing at any given time. Dedicated testers, sure, but also developers, support engineers, operations people, stakeholders, customers (indirectly but they will test whether we like it or not). No one is going to be able to be excellent at everything but everyone can and will develop broad skills and knowledge that will be helpful to the process and projects. This also includes a variety of soft skills. Testing and bug hunting is important but just as important is advocacy and developing the ability to present cases on behalf of others and speak to the risks and challenges and what they may face. Additionally, think about the members of teams and what they are doing. If the attitude of "no more dedicated testers" is a potential reality, as what the plan is with those people above and beyond testing. Do we plan on having them add value in development? In Operations? Assisting support? Each organization will have different goals and needs but again, investing in people will net great results down the road.  

QA Open Season w/ panel people (@Xpanxion #QASummit 2021) : Live Blog

All right, here comes the last formal activity of the day. We are all gathered for a Q&A shootout with a panel of six participants:

• Rachel Kibler
• Carlos Kidman
• Greg Paskal
• Jason Bryant
• Marcus Merrell
• Matthew Heusser

The questions for this session have been generated via Sli.do and we've covered a number of questions, such as:

"What is the path for software testers going forward if you may not specifically be aiming towards being a technical tester?"

The general consensus is that there are so many possible avenues to explore and get involved in that worrying if you are not suited for automation, you need not worry that your career is over or that you will be replaced. If you apply your brain effectively in an organization and bring value with your efforts, you will run circles around any computer and script. Maybe not fast circles, but circles nonetheless. 

What is the difference between the hype and reality behind AI and ML?

In general, the hype around replacing the human with a computer seems to inspire investors much more than it inspires organizations. AI and ML should be focusing on the data science so that we can actually learn from the data we already have accumulated. Now that could be valuable (and I very much agree :) ).

How does your organization demonstrate the ROI on the testing investment?

The consensus is that if you lead with risk, the odds are that the C suite will start paying attention. Many ideas may take precedence at random times but talking about the actual risks, lead with risk and the C Level folks will year you. 

What are some ways to get testers to think more about quality?

Rachel voiced that she has a quality coach on every team but not necessarily a tester on every team. IOW, the role of testing may or may not be as critical as the role of quality but the role of quality itself certainly is.

A question that I will in no way be able to repeat because it was too verbose...?

Learn to ask better questions and learn when to avoid useless/needless buzzwords.

We interrupt this program to have a company jingle breakdown (you had to be there ;) ).

Can Unit Testing Be used as Integration Testing?

Seems the consensus is they are two different things. It's what you do with them that matters. Add two and two in your head. That's a Unit Test. Check the time... let me grab my watch... that's an integration test. Works for me :).

What is the #1 issue facing the QA world currently? What is the hottest trend?

The biggest issue is not testing the right thing. This extends to testing on devices people actually use (including mobile devices and Internet of Things devices). The biggest trend is ignoring failures and moving on as though there are not issues, and that is problematic, to say the least. Observability is a hot property and we are just at the beginning of what might be possible. 

How do we get our management teams to focus on iOS testing (or mobile testing)?

It seems that iOS in several organizations is not being actively tested or distantly compared to other infrastructures. the answers tend to range around risk and the fact that things break. Quantify how bad things could be if iOS interactions would be compromised or made unusable. My guess is a lot of users would be locked out and would effectively stop a revenue stream and that should light a fire under some people.

And that's a wrap.... oh, and Marvel (LOL!)

How Holistic Testing Affects Product Quality with @janetgregoryca (@Xpanxion #QASummit 2021) : Live Blog

 We're down to our final keynote and it's a pleasure to see Janet Gregory, if only virtually, this year. Since the border situation between USA and Canada is still in question (and considering the situation with outbreaks we are seeing, I don't blame it in the slightest), We're still getting to hear Janet talk about the value of DevOps and the fact that it genuinely works when the teams in question genuinely put in the time and energy to make sure that the teams can work. 

Quality is always a vague and odd thing to get one's head around. What makes something good to one person may not be so excellent to someone else. In some areas it is objective but much of the time it is subjective and not even related to the end product itself. Janet uses the example of a cup of coffee. For some, the best coffee is experienced black, so that every sense of the flavor of the beans can be examined. For others, the best crafted iced frappuccino with all of the extra flavors makes the experience a quality one. Does one approach replace the validity of the other? It really doesn't but it matters a lot to the person in question at that point in time. Quality is what matters to a person experiencing the item in question and in the way that they want to experience it.

So, how do you build quality into your product? In many cases, quality is not just one figure but many that come together. Some may argue that Lamborghini sports cars are of high quality. I may or may not agree but the cost for a Lamborghini puts it well out of the range where I will ever find out. Is the level of quality a consideration if you can't consider paying for it? If it is super affordable, does that automatically mean the product is of low quality? Not necessarily. I'm reminded of the app Splice, which is a video editing app that I use on my phone. Granted, I pay for it (about $3 a week) but their regularity of updates and their method of continually improving the product makes it worth that expense for me. It's not s much that it is going to discourage me but it also provides me a value that makes me willing to keep paying for it.

Holistic Testing focuses on the idea that testing happens all the time. To that end, Janet is not a fan of the terms shift-left or shift-right testing. The real question is, "what do you mean you are not doing active testing at every stage of the process?" It does help to know all areas where testing makes sense to perform and why/when we would do it. It may honestly have never occurred to people that monitoring and analytics after a product is released fits into testing and that testing can actually learn from these areas to help improve the product. 

One of the best phrases a tester can use/encourage is "can you show me?" I find that when working with developers and testers, many misconceptions and miscommunications can be avoided just by asking this question.  Using AB/Testing, feature flags, or toggles to turn on or off features allows us to do testing in production without it being a scary proposition. We also get to observe what our customers actually do and use and from that we can learn wi=hich features are actually used, or for that matter even wanted in the first place. We may also discover that features we develop to serve one purpose may actually be used in a different manner or for a different purpose than we intended. With that kind of discovery, we can learn how to better hit the mark or to provide features that we may not even be totally aware are needed.

The key to realize is there are testing initiatives that happen at every level of software development. It's important for us as organizations, not just us as testers, to learn how to leverage that testing focus at all levels and be able to learn, experiment, confirm or refute, and then experiment again. It will take time, it will take involvement, it will take investment, and it will take commitment. Still, the more that we are abe to leverage these testing areas, the better our overall quality approach will have the potential to be.

Managing The Test Data Nightmare with @AutomationPanda (@Xpanxion #QASummit 2021) : Live Blog

 Wooo hoooo! The second talk is done, I am officially free of obligations today :). However, conference still moves on and this session covers an area that I personally struggle with. My personal job does a lot of data transformation so I have a possibly endless range of test data that can be generated and transformed. 

Test data shows up everywhere. Not just the data needed to make the test work but your browser choice, your necessary created artifacts, before and after dependencies, etc.

Static Data is often created before testing. it's good fr slow or complicated data. It may make tests run faster but it may make tests brittle as data changes, and it may turn stale over time. Dynamic data gets created at run time for tests, it can avoid becoming brittle, it's exclusive tho that tests, creation and run time, it can slow down individual tests due to the creation dependencies, and it will need to be cleaned up after the test is finished.

The truth is, I probably initially do about 70% manually configured data and about 30% dynamic/automated data creation. The intermediate data that I create is always created dynamically (that's the nature of data transformation tests). Ultimately, my goal is to be able to take data from our database and dynamically generate flat files.

Additionally, there is a variety of test control inputs that we need to keep track of. Our browser. our destination URL, basically all information that can be entered and routed. There are also output references that we may know or we may not know anything about. 

The Quality Mind with Gwen Iarussi (@Xpanxion #QASummit 2021) : Live Blog

 Well, that was fun. I just delivered a talk about Self Healing Automation (spoiler: not really, much more opportunistic and using agents to help build a dynamic locator switch statement but that's nowhere near as cool sounding ;) ). Good group, great interactions, thanks to those who attended.

The next session is with Gwen Iarussi and we're talking about how to build a Quality mindset. Gwen focused on the past forty years and how we approached quality and software delivery, how the tools and processes we used developed and grew over those decades. The challenges we face today are similar but definitely "faster". We need to consider scalability as organizations often grow quickly and what worked one year will be wholly inadequate the next. More data, more interactions, more people, more, more, more! Always more!!! With this increase in infrastructure, knowledge of tools and tooling that we had last year is out of date already (not completely but there's a lot that happens in any given year).

"The quality of your thinking determines the quality of your life" -- A.R. Bernard

When we talk about replacing QA with automation, they are talking about replacing the human brain with machines. It's important to realize that machines are repetitive and exact but remarkably stupid without being told exactly what to do. Humans are slower, less prone to dealing with repetitive tasks but we can come up with so many interesting avenues. Our brains are pattern-recognition machines, in that we are very quick to catch on to patterns and then be able to anticipate what comes next.  

We can learn and believe a lot of things and how we approach the way that we learn and focus on tasks gives us the impulse to succeed or to hold us back. Neuroscience has shown that we all bring unique perspectives to our interactions. Our experiences both inform how we look at our approach to quality and, likewise, how we approach testing.

First questions:

Who uses our product? What is most important to them?

What is critical for my company's survival?

What tech do we need to invest in or understand what we already have?

What tools and heuristics can help me make sense of what I am seeing?

Continuous Learning is critical to success. When we stop learning, we stop progressing. Areas like Systems Thinking, Design Theory, Human Psychology, Learning about the world, Cognition Games, and Testing Disciplines/Methods are all important areas to study and understand.

Knowing these things is not enough. We have to actually know it and use it. that means we need to be at the table, "the room where it happens". In short, we need to be engaged and involved. If we are not, it's our loss.

On The Road Again: Speaking Today at the @XPansion #QASummit (Live Blog)

 Hi all!

I confess I have been struggling to participate with this blog. I just haven't felt mentally in it. Additionally, it took me a little while to get things sorted out with my Twitter handle (in a neat twist of fate, the person who took the account decided to give it back to me, so I will be putting mkltesthead back into my bio again. It took me a while to make sure that the gift of my account back didn't come with some "extra stuff" that would have made my reality unpleasant but thankfully that was not the case).

A couple weeks back I was asked if I'd like to speak at the XPansion QA Summit being held in South Jordan, Utah, USA. Seeing as I had a number of friends participating in the program and I hadn't spoken in a live setting in nearly two years, I decided it was time to say "yes" and get back to live speaking. That is part of what I will be doing today. I will be giving two talks today (actually, I'll be giving the same talk twice) about "Sef Healing Automation" or more to the point "what self-healing automation actually is (in most cases) and how it's basically a switch statement that rebuilds itself.

The first talk is being given by Andrew Brown and the topic is "Why Do People Break Software Projects". Andrew predicts that software development in 2031 will have about 20% of projects fail. Many will be late or over budget. Some projects will take crazy risks. Many will work in silos. They will develop too much technical debt, they will add more processes that will have no effect on quality, and their regression tests will be filled with junk. Sounds like today, huh? Well, that's the point. We've had these same problems for fifty-plus years. What are we missing? First, there's a technical part and that changes all the time but there is also a people/human part and those problems don't really change. What's worse, we don't change them because we don't really understand those issues. The key to realize is the human brain was never really designed to develop software. The fact that we can do it is kind of remarkable. The human mind is amazingly adaptable but the technology we create quickly outstrips our effective understanding of it. Our thought processes have deep evolutionary roots and many of our thoughts are much more primitive, tribal, and segmented. We are focused on survival and reproduction, and those aspects we do quite well. Those are far and away removed from the thought processes that help us develop software. The technology far outstrips our actual understanding. 

There is a lot of historical fears and issues, some might call this the lizard brain. Those fears and issues are the ones that get to the heart of being human and why we struggle with getting things done effectively. Often, we are overconfident. We see things the way we are, not the way they should be seen. 

Overall, this has been a neat discussion and some interesting ideas shared. I see, and agree, that the areas we need to spend more time on are not the technological issues but the human issues.