PNSQC 2020 Live Blog: “Rethinking Test Automation” with Paul Gerrard

I just realized that the last time I saw Paul in person was back in 2014 at Eurostar in Dublin, Ireland. I was really looking forward to seeing him again after so long but alas, I guess this will have to do.

It's interesting to see that the notion of being surprised about the troubles related to testing automation has been with us since the nineties at least (and some could argue even longer as I remember having issues and dealing with oddities back in the early 90s as I was first learning about Tcl/Tk and Expect. We still struggle with defining what test automation can do for us. Sure, it can automate our tests but what does that really mean?

Tools are certainly evolving and look nothing like the tools we were using 30 years ago. Still, we are dealing with many of the same principles. The scientific method has not changed. I share Paul's criticism that we are still debating what test automation does and what testers do. the issue isn't whether or not our tests work, it's whether or not the tests we perform are actually gathering data that can be agreed to or refuted for hypotheses. As a tester, we want to either confirm or refute the hypothesis. At the end of the day, that is what every relevant test needs to do. We can gather data but can that data that we gather give us meaningful information to actually tell us if the software is working as expected? One could argue that assertions being true are passes... but are they? They are proving we are seeing something that we expect to see but is it actually proving a hypothesis, or merely a small part of it? In short, we need people to look over the tests and the output to see if it is really doing what it should be.   

Paul suggests that we need to move away from scripted tests to more modeled based tests. OK, but what does that actually mean and how do we actually do that? Paul makes the assertion that tools don't think, they support our thinking. What if we removed all of the logistics around testing? If stripped of our usual talismans, what would we do to actually test? rather than stumble through my verbiage, I'm stealing Paul's slide and posting it here:

The key here is that test automation misleads us, in that we think that tools are actually testing and they are not. What they are doing is mapping out the steps we walk through and capturing/applying the sets of data and results that we get based on the data and actions we provide. The left is the exploration, the right is the evaluation, the middle is the testing or the setting up so that we can test. Automation won't work if we don't have a clear understanding of what the system should do. Paul is emphasizing that the problem and the area we need to improve is not the execution of tests (our tools can do that quite adequately) but in test design and test planning. In short, we need better and more robust models. 

The old notion of the human brain is that it is brilliant random, unpredictable but slow and lazy. Machines are literal, unimaginative, but blindingly fast and able to do the same things over and over again. Combined, we have a formidable combination. 

So what do we want to see the future be for our tools? First of all, regression testing needs to look at impact-analysis. How can we determine what our proposed changes might do? How can we stop being overly reliant on testing as an anti-regression measure? How can we meaningfully prove functionality? Also, how do we determine the optimal set of tests without guessing?

Paul makes the case we need to understand the history of failures in our tests. Where can we identify patterns of changes? What are the best paths and data to help us locate failure-prone features? Manual testing will not be able to do this. Machine learning and AI will certainly get us closer to this goal.

In short, we need to move away from passive to active collaboration. We need to stop being the people at the end and work towards active collaboration. We need to be able and willing to provoke the requirements. we also need to create better mental models so that we can better understand how to guide our efforts.

PNSQC 2020 Live Blog: Full Stack Testing Is A Culture with Christina Thalayasingam

Yay! another new speaker. Christina has said that this is her first time speaking at PNSQC. To which I say, welcome, and thank you for joining us.

Full-stack is a term that I have heard used mostly with developers and over the years I have joked about the development of "full-stack testers" (I recall Mark Tomlinson having a spirited debate on that term some years back ;) ). Still, that is the core point of Christina's talk so what does it mean?



My first question is "do we want this to be in one person, or are we talking about the full stack being in the team as a whole?" Is it realistic to have a full stack tester? Can any tester know everything from top to bottom? Or are we looking at full-stack testing meaning any tester or the team of testers can attack any given testing challenge? 

As Christina points out in her abstract, there are several areas and types when it comes to Quality Assurance

Testing
Test Automation
Performance
CI/CD/CT

Usability
Accessibility

to be frank this is an interesting and exciting prospect. It reminds me that I should probably review Alaister Scott's "Pride and Paradev" as that was one of his goals, to help encourage testers to branch out and become broader jacks of all trades. Basically, it comes down to any tester being able to tackle any testing challenge anywhere up and down the development stack. It is unlikely one tester is going to be able to completely cover everything out of the gate. However, it's a good bet if you have a team of three to six testers, you may well have one full-stack tester combined. From there, make a commitment to developing the team to spread that knowledge so that everyone can be competitive. Of course, the odds of getting all six team members to be equal across the board for all skills is probably not reasonable but it's a great goal to reach for. My guess is, even if you don't completely make it, there's a great chance the team will be far more formidable than they were previously :).

PNSQC 2020 Live Blog: Communication Is Key: Lessons Learned From Testing In Healthcare Technology with Rachael Lovallo

As we consider communication challenges, what would we do if those challenges didn't just end up being an issue of finding or missing bugs but potentially being an issue that could spell the difference between life or death? Did that get your attention? Yeah, it got mine, too.

Rachael Lovallo focuses her quality initiatives on the emergency healthcare industry. As she clearly stated at the beginning "we can’t afford bugs, as they could endanger real humans". 

Make bugs as bullet-proof and reproducible as possible. 

This may seem obvious but I would guess that missing anything or being vague would be something that would cause anxiety in this particular environment. By making sure that we identify all of the aspects that point to the bug, a reliable reproduction strategy, and an expected behavior or output to compare with is critical.

Talk to people writing the code you test

Again, seems evident but how many of us really communicate about the code that we are testing? How much of it do we really know inside and out?  How confident would we be to be able to go through and read the code and know exactly what it is doing? I know for myself that I wouldn't be able to do that all on my own,

Ask for a second set of eyes on your work

Oh, I can so relate to this! As someone who basically deals with data transformations at the moment and the methods involved in that covers so many tools and points of contact that it just isn't possible for me to know where everything is and what I should be looking at. Fortunately, the development team I work with are great about showing me the areas I need to hit and to also recognize where I find areas of friction or that are less than optimal for being able to complete tasks. By getting them to walk the workflows and steps with me I am able to uncover with them ways to make the overall flow work better.

PNSQC 2020 Live Blog: Of Machines and Men with Iryna Suprun

As is often the case at PNSQC, several of the talks are from people I have not seen speak before. Iryna Surpum is focusing her talk on areas of AI and Machine Learning. As we start her talk, we look at the fact that there are few tools available where AI and Machine Learning are prominent and prevalent for individual users. Some hallmarks of AI-based tools and what is being marketed are Codeless script generation, the ability to self heal with changes to the environment, meaning the script can collect data about elements of the application itself, and the ability of Natural Language Processing to be able to convert the documentation to actual tests (this is a new one to me, so hey, I'm intrigued). 

Comparison of Visual Output and the expected design is becoming more sophisticated. More tools are supporting these features and additional levels of comparison are being applied (not just pixel to pixel comparison these days). 

So while we have these changes coming (or already here, how can we leverage these tools or learn how to use them in the first place?

Example tools to try out for these comparisons were, Testim, Mabl, and TestCraft. What did they provide? All three allowed for a Quick Start so that they could learn and be able to automate the same basic initial test case. All of the tools had recording implemented, which allows for initial set cases to be created (testCraft had a few extra steps to be set up and utilized so not quite as easily started as the other two). Modifying and inserting/deleting steps was relatively fast.

So what challenges were discovered/associated with these tools? as could be expected, the Codeless Script Generation (Recording) is good to get started but its usefulness diminishes the more complex the test cases become. This is to be expected, IMO, as this has been the same issue with most automation tools that promise an easy entry. It's a place to start but getting further will require proficiency and experience beyond what the recorder can provide. Self-healing is a useful feature but we are still at a point where we have to be somewhat explicit as to what is actually being healed. thus calling it self-healing may still be a misnomer, though that is the goal. So how about self-generated tests? what data is actually being used to create these self-generated cases? This didn't seem to be very self-evident (again, this is me listening, so I may be misinterpreting this). An example is to check to see that links work and are pointing to literally legitimate end links. that tests to see that a link exists and can be followed but that doesn't automatically mean that the link is useful to the workflow or that it will validate that the link is relevant. People still need to make sure that the links go somewhere that makes sense. 

So even though we keep hearing that AI And Machine learning are on the horizon and are even here changing the landscape, there's still a lot of underlying knowledge needed to make these tools work effectively. There's definitely a lot of promise here and there's an interesting future to look forward to but we do not have anything close toa magic want to wave yet. In other words, the idea that AI is going to replace human testers might be a possibility at some point but that promise/scare is not quite ready for prime time yet. Don't be complacent, take the time to learn about how these tools can help us and how we can then leverage out brains for more interesting testing work.

 

PNSQC 2020 Live Blog: Capacity To Execute with John Cvetko

The first talk of the afternoon is being given by John Cvetko. The main point of John's talk is to look at the desire of an organization to change from a Waterfall development methodology to an Agile development methodology. for many of us, it may seem slightly odd that there are still Waterfall SDLC's out there but rest assured there certainly are, lots more than we might like to think. John focused on the considerations needed to make the changes necessary to make the move from Waterfall to agile.

The example organization had multiple release trains (36 release trains in total) and part of the initial process was to create a hybrid release train. They didn't go all in or replace every single release train. Rather, they took on the Agile transformation within a few release trains. They took an isolated and low risk approach so that no more than 20% of the projects would be affected. the Agile transition even with this focus and need was still going to be a challenge and that the implementation of Agile was going to require an entirely different way of thinking.   




This gets us to the title of the talk. The “capacity to execute” is the most important aspect, in short,"Plan your work and work your plan". More to the point, there's more need for planning to make a transition effect. Whether the approach is to make a huge sea change, or smaller waves or bubble up changes, there has to be the ability to plan and execute effectively. 

There are a variety of metrics that are looked at (an example is ISO 25010 Software Quality Standard) and which quality characteristics were chosen to determine the level of focus of the transition. 

By focusing on Scope, Schedule, Budget, and Quality, we as an organization can make a real determination as to what the transition will be able to focus on and to what extent. Just because we want to make the change doesn't mean we will be able to automatically get there just because we want to. Often the business context, the company culture, and the revenue and cash on hand will help influence those paths. 

There is often a Culture Clash that may occur when each team works to try to make these transitions. thus the intentions and plans to execute are doubly important at this time. 

Make no mistake, these are tricky and challenging transitions. It may very well prove to be worth it but it just as likely will not be easy.

 

PNSQC 2020 Live Blog: Software Performance And Load Testing Utilizing JMeter with Anna Sharpe

 

this is number two of the fast-paced talks this hour. Performance testing is often seen as a black art and an area that requires specialized skills. To be sure, doing performance testing at a high level on a regular basis will require special skills and often use tools that are specific and need to be set up in a way that everyone knows about.

However, that may be overkill for what an immediate team needs as well as to run smaller levels of stress tests that are not as all-encompassing. Another challenge is that, if you don't have performance testing skills, you may find it a challenge to getting those skills or getting people to give you the time to train on those skills. The good news is that you can start doing performance testing with a small amount of data and practice getting a handle on the skills that you want to learn. JMeter is a great place to start with this. It's free, it has a small footprint (relatively speaking) and it is friendly to running on smaller, more isolated environments.

JMeter is a well used and well-known tool that has a broad user community and active developer support. JMeter can also work with tools like Fiddler and Charles Proxy to help construct tests and control the responses. 

If at all possible, run your performance and load tests during off-peak hours if possible and especially to make sure not to run these tools on the actual production environment. If you have to use the production environment, do so at a time that coincides with a maintenance window or otherwise runs at a time when customers are not actively using the system. If you have a confirmed level of response that the server is meant to provide (say, 30 active users) a good rule of thumb is to try to run a load test based on the expected load of the system, at least at first. If the system doesn't fall over or requests don't fail, you may want to bump p the number of concurrent users./threads and see how performance changes.

This is a good reminder that testing with JMeter need not be a big involved process and can actually be fun. 

 

PNSQC 2020 Live Blog: Human Centric User Acceptance Testing with Rebecca Long @Amaya30

The next couple of talk are set as quick briefs, so there are two speakers in  an hour-long block Rebecca Long is the first speaker and focusing on  User Acceptance Testing (UAT). What is the primary focus of UAT and more to the point, what does Human-Centric UAT look like? 

For starters, there are ways to be inclusive in the way that we look at our users. Our users are all individuals that have specific needs and ways to be addressed and validated. In addition to focusing on the product being a tool for users to access and interact with, we need to consider what ways we might be inadvertently making those users uncomfortable or put off. In today's world of immediate downloads and use, if we alienate our customers, unless they are locked into/forced to use an application, they will just as likely remove it and never let you know why.

As I tend to focus on Accessibility issues, this is certainly an area where we want to consider this level of focus in our UAT efforts. Above and beyond this are also ways that people wish to refer to themselves and identify. Being inclusive allows for a minimum of friction or complete lack of friction if possible. By taking the time to look at these areas we will help to encourage as many people as possible to use our products and actively engage with us.

Additionally, it's important to realize that these efforts are not one size fits all nor are they one and done efforts. People are complicated and they can be difficult to manage and interact with. Making the effort to include as many people as possible will help make a baseline that includes as many as possible and thus will be usable by as many as possible

PNSQC 2020 Live Blog: Iron Chef Cucumber: Cooking Up Software Requirements That Get Great Results with Chris Cowell

This is our first track talk for the day. For those who are familiar with cucumber, it is both a cool way to do requirements outlining and a top-heavy tool to write tests with an additional plain language wrapper. Note that both of those statements are accurate and are not meant to be said as a positive or a negative. I've ween Cucumber used well and I've also seen people use it in sloppy careless ways. Chris Cowell seems to agree with me. In Chris' view, a bad Cucumber implementation is worse than no Cucumber implementation.

Cucumber is a free open source tool where you focus on behavior-driven development. Basically, how does the user interact with the program and how can we determine that proper behavior is seen.

Cucumber is based on scenarios and each scenario contains steps. Steps are typically written in a format that emphasized statements set up in a "Given, When, Then" series. Think of something like:

Given I want to log into the system

When I enter my username and password

Then I am on the welcome page

Of course, this is just a series of statements. they don't mean anything unless they have actions behind them. Cucumber has a number of harnesses and frameworks that it can interact with. Ruby and Java are perhaps the most well known but there are implementations for numerous other languages.

Rather than focus on the underlying code, let's consider Cucumber requirements specifically (and its underlying Gherkin language). there are a variety of ways to make scenarios and those scenarios can be made to be complex. Ideally, we would try to avoid doing that. The simpler the requirement and the more readable the requirement, the more likely that scenario will not be misunderstood. Adding too much logic or too many steps will make it difficult to determine if the requirement is actually being met.   

Cucumber is also not meant to be an exhaustive system or set of testing options. Plain English sentences are the goal. Look above where I created a simple login example. I could format this so that it looks at entering in multiple usernames and passwords. Is that a good series of tests? Possibly. Is that a good way to test behavior-driven requirements? Probably not. It's also better to focus on what is being done, rather than the exact steps or movements necessary to do every permutation. 

Personas are useful and can help make the descriptions more meaningful to all stakeholders. By giving names and personalities we can see how different users may require or expect different behaviors. 

Another anti-pattern is to avoid copying and pasting between scenarios. Granted, if done correctly, specific statements should be reusable in as many scenarios as possible. The key is to make sure that the statements made are as generic as possible and if generic doesn't cut it (as in there are terms that are specific to scenarios) then make the general statement as useful as possible without having to make multiple changes.

Scenarios can also have general statements that occur frequently. If you have scenario steps that are identical in different scenarios, it makes sense to extract them to a single location and use a keyword called "background". this background means any and all scenarios using that background will call that statement first and will call it with all of the scenarios listed beneath the background.

This all might seem a bit top-heavy and it's possible for some environments and well-established development environments, it may not make sense to wire up a full Cucumber test framework. Still, even if you don't actually run these Gherkin statements as literally automated tests, it is still useful to think of the phrasing and actions associated with Cucumber statements. these phrases and syntax may prove helpful when writing your requirements. Just using the base Cucumber syntax for writing out requirements may prove helpful. If you want to go the rest of the way, you could think of doing the rest and actually writing tests that use each requirement statement.

PNSQC 2020 Live Blog: TestOps with Jason Arbon

 First, this event is unique this go-around. Joe Colantonio is helping to host this event. Some of you may be familiar with Joe via Test Guild. Since the conference is virtual this time, having Joe help with running the event is a definite plus.

One of the changes this year is that they are using Slido to ask questions during the sessions. This will allow the presenters to answer the questions on the screen. There will also be virtual events happening during lunch. I intend to take part in one of those so expect me to see me doing so then :). Tomorrow, we will be recording an episode of "The Testing Show" from the "show floor". 

Philip Lew is starting the event. He is reminding us of some of the questions that came up back in January. What would this conference look like? Suffice it to say, nothing like what it looks like today. Still, we are making the best of the situation. One of the benefits of the conference being virtual is that speakers can take part from anywhere. One of the keynote speakers is presenting from Australia. We do not have the poster papers this time but there are many technical briefs that are being presented. 

Jason Arbon is the first keynote speaker. He is speaking about TestOps. The idea behind TestOps is to shake up the way that testing is being performed. It also allows for a way to have testing be present. Jason focuses on testing on a large scale and is currently with test.ai.

TestOps is the idea of operationalizing testing. the goal is to automate as many things, make as reliable as possible, and scale as much as possible. Testing is a unique area in that automation is often difficult. Tests can be flaky. Often, decisions focus on shipping even if the testing is not complete. Unit testing can't answer everything. Many areas are resistant to running end to end tests. Jason used the example of a car purchasing application that was frustrating to use. Its functionality was not working well. Yet, this application shipped as is and considered usable. It's likely that this application passed all internal unit tests. Still, a product can pass all tests and still be unusable. 

 Tests should be short-lived. They should focus on the simple rather than the complex. They should be capable of running fast. Consider we have ten tests that we have to perform and one of those tests is signing in. What if signing in is actually difficult to do because of security implementations? How do you test in parallel with one sign in? The answer is you drop sign-in as a necessary test and focus on other areas that don't need sign-in. Heresy, you say?  There may be other ways to do the same thing. There may be direct calls that can run via API. There may be individual authentical calls. Again, think simple where possible.

Maintaining attributes and states can be difficult. Avoid framework tweaks that may seem to be effective in the short term but become unreliable later on. Additionally, try to avoid the need to create an elaborate dashboard to display results. It seems counterintuitive but instead focuses on sending notifications if a specific problem is found. alert the right people at the right time in real-time.

 

Maintaining attributes and states can be difficult. Avoid framework tweaks. They may seem to be effective in the short term but become unreliable later on. Try to avoid the need to create an elaborate dashboard to display results. Focus on sending notifications for a specific problem. Alert the right people at the right time.

The more frequent the updates, the less likely complete testing will happen. It's tempting to think that eliminating testers is an answer. Instead, there needs to be a focus on good, robust end-to-end tests. Also, those tests need review and consideration by real testers. It's not enough for developers alone to do the testing. That's not to say that they can't do it but they need to focus on that testing as its own discipline.

Jason is a proponent of AI, which comes as no surprise considering test.ai and their focus. Frameworks and test applications have limitations that force hard-coded tests and methods. In a TestOps world, if the tests are not reliable, they will not execute. Many testers get thrown into automation without previous development experience. Putting a new developer into the space of automated testing can be a recipe for frustration. Not to say that new developers can't write tests. They can, and should, but it shouldn't be their first programming assignment. These tests need to be robust, they need to have the ability to be abstract, and they need to be adaptable. That will take some experience that the average tester does not yet have.

TestOps is happening, it's real, and it's not just an abstract goal. Test what you can, as cheaply and reliably as you can. This isn't abstract, it's a reality here and now and we will benefit if we adopt it.

Live Blogging for #PNSQC2020: Back in the Saddle Again

It has been far too long since I've had a chance to get in and do a live blogging session of an event. Usually, that is one of the ways that I take notes and focus on the conferences that I attend. This year has been different. I haven't attended any conferences or online events beyond hour-long meetups. I've also felt less motivated this go around because 2020 has not been a very inspiring year. Not in ways that I have felt positive about. I realize there is no learning without getting out there. There is no growth or development without discomfort, determined focus. and discipline. Thus, I'm climbing back into a favored role.

This year, the Pacific Northwest Software Quality Conference is a virtual event. I'm in my home office waiting for a Zoom meeting to begin. I've helped develop the conference content as a reviewer of papers and presentations. This is a role I have participated in for many of the past ten years (my first time attending PNSQC was in 2010). Even when I couldn't attend the conference, I've still assisted in the paper review process. I have had more direct involvement with this conference than any other over the past decade. Much of the fun is direct interaction with other conference attendees and volunteers. There are many people I look forward to seeing at this conference. To not have that level of interaction is frustrating. I am certain I'm not the only one to feel that way.


Still, I plan to make the best of it and use this as a means to break the funk I've been feeling about blogging. If nothing else, this gives me a great reason to reflect. It also allows me to think, contemplate ideas in real-time, and capture them to look back on later.


If you've enjoyed these missives in previous years, I hope you will join me over the next couple of days.