Surviving The AI Testing Apocalypse - a #PNSQC2019 Live Blog

Here's an interesting thought.

Let's say we are building a road. We come up to an anthill. What do we do? the answer is pretty simple. We run through or remove the anthill.

Now let's rephrase that... what if we develop AI that goes to a point of surpassing human civilizations barriers. What if we set AI on a goal that we get in the way of. Will AI look at us as little more than an anthill?

Unsettling? Maybe. Accurate? Remain to be seen, but again, maybe.

Now do understand, I have not cut my throat yet or anything. I have children and I want to see them grow up and thrive so I have a vested interest in a future for humanity. Yet I would be lying if I didn't say that I look at machine learning and AI development with both a sense of trepidation but also a sense of excitement and wonder.

Now let's step back a bit and let's talk about AI as it relates to software testing. Will AI and ML devastate software testing? It's possible but at what level? Do we believe that it can replace human creativity? Could a computer AI figure out how to write a coherent symphony? It can make some straightforward musical progressions now. Are they terribly exciting or unique? No, but they do follow the fundamental rules of music theory and to many ears, there's not really a difference. The AI mak serviceable and competent music. I'll reserve my comments about how creative or unique the music created is. Then again, I could say the same about most of the music humans are creating at the moment, including me.

We can certainly see AI/ML interacting with video games and improving/learning how to play a game. It's interesting to view in real-time.

This is all a premise and y thoughts on Dionny Santiago's talk about "Surviving the AI Testing Apocalypse". Bold statements but let's look at a few words.

Apocalypse is a much-maligned word. It sounds automatically like it is a calamity. However, Apocalypse comes from the Greek word "apokalupsis", which means "to uncover". Thus, do I think that we face an uncovering of AI and interacting with it? Yes, I absolutely do. BTW, it also means "to reveal" which is where the word "Revelation" comes from. I feel bad for Greek, it's getting a bad rap. Why does Apocalypse get the bad press while Revelation is so much more hopeful (Eschatological meanings not being included here).

AI is here. It's real. It's happening now. It will keep happening. It's important and we are benefitting from it. However, I do not believe we are facing anything like The Matrix or The Singularity. I likewise also believe that human invention and creativity will win out in the end. That doesn't necessarily mean that *I* will win out. I may may well be a casualty. Nevertheless, what a time to be alive :).

The Release Day Crime Scene - a #PNSQC2019 Live Blog

Bill Opsal and I have been in talks over the past year or so about putting together a simulation game that we jokingly called "CSI: Release Day". Think Clue but for software teams.

The premise is as follows.

There are six roles: Release Manager, Dev Lead, QA Manager, DBA, Project Manager, and Customer.

Each role has three key pieces of "mandated transparency", meaning if you have a role, depending on the scenario being run, you can say anything about what your role is but if you are asked a question where the answer is your piece of transparency, you have to say it. If your piece of transparency also indicates that you are the guilty party, you likewise have to say it.

The goal is to have the Crime Scene Investigation team question the people with their roles and see if they can ferret out the problem and the reason why it caused the problem.

All in the name of truth, justice, and the releasable way.

Bill and I enjoyed talking about this over the past few months leading up to this so it was fun to finally, FINALLY get to present it. On the whole, it seems like a lot of people enjoyed it and it also seems like the game has legs, so it's entirely possible we may expand on this and lmae ita little more concrete for a future conference :).

Test Scenario Design Models - a #PNSQC2019 Live Blog

This year I decided to emphasize the Test Engineering track, and I'm not saying that just because my talk was part of that track. It is actually one of the key areas I would like to see some personal improvements in what both I and my company do. With that in mind, I decided I want to check out "Test Scenario Design Models: What Are They and Why Are They Your Key to Agile Quality Success?"

Some interesting perspectives:

87% of respondents say management is on board with automated testing
72-76% of respondents say they are doing test automation and scripting
Most organizations are between 0-30% automated.
Of those, about 40% of their tests are completely redundant, meaning they are really not worth anything.

There seems to be a disconnect here. I do not doubt it in the slightest.

Systems are more complex, the proliferation of environments is continuing, manual testing will never be able to cover it all, and the automation we are doing is not even helping us tread water effectively.

Can I get a "HALLELUJIAH!", people?!!

What could happen if we actually said, each sprint, "we are going to send a couple of hours to actually get our testing strategy aligned and effective"? Robert Gormley encourages us to say "oh yeah, we dare!" :)


This is where the idea behind Test Scenario Design Models come in.

The goals are:
- user behavior is king
- test cases are short, concise, and business language-driven.

We should not care if we have 4500 total test cases. What we should care about is that we have 300 really useful high-quality tests. The numbers aren't really relevant the point is to have tests that are effective and to stop chasing quantity as to any kind of a meaningful metric.

So how do we get to that magical-unicorn-filled land of supremely valuable tests?

First, we want to get to a point where we can be specific with the tests that we need to run but no more specific. Extensive test scenarios are not necessary and should not be encouraged. Additionally, we need to be emphasizing that we are looking at testing relevant to the User Acceptance Testing level. That's where we find the bugs, so if we can push that discovery further back, we can work on more important things.

Why We Need New Software Testing Technologies - a #PNSQC2019 Live Blog

Dr. Carol Oliver makes the case that we are now seeing a proliferation of environments where software can run. Once upon a time, programming was a bespoke profession. Computers were standalone and in many cases not interoperable. That has certainly not been the case in the past few decades. Mobile and IoT software have the added challenge of even more potential places and environments for software to run. The testing paradigms that we use, however, have not kept pace with the proliferation of platforms. This should probably not be a big surprise but it is a real issue.


Dr. Oliver has her paper to describe a Release-Readiness Levels Framework with vocabulary and structure so as to discuss what software testers would like to be able to test and what the existing tools and technologies enable them to test.

Mobile and IoT show how many potential devices exist and how very few of them we actually test with. We are especially not looking at all of the possible variations of hardware and software. We literally can't. Modeling software also uses such a large level of abstraction that it is very narrow the range we are actually testing vs. what is out there.

Areas that can be addressed and looked at for pros and cons are:

• Physical Devices (the actual proliferation of machines)
• Virtual Devices (abstractions to represent real machines)
• Simulation Environments (even greater levels of abstraction, one size fits most, theoretically)
• Visual GUI Tree (what prt of the GUI can you interact with? How much can you interact with it?
• Image Comparison (How do we determine we have the appropriate technology to display the images correctly and appropriately.
• Code Manipulation
• System Monitoring

One of the things you will notice is that most of these areas are not immediate or easy to test. Additionally, many of these areas require a technical bent to the testers in question. We are not even close to having general-purpose tools but we do have some ideas as to what those general tools would entail. Needless to say, I am intrigued and will read the relative paper more closely :).

How to Start a Test Automation Framework and Not Die Trying - a #PNSQC2019 Live Blog

Hands down, the best title of the conference :).

As someone who is knee-deep in the process of revamping an automation framework at my company, this is very timely and understandable. Too often, we throw testers into the role of being an automation engineer with perhaps little to know real programming experience. Suffice it to say that having software testers' first programming project be producing a testing framework does not necessarily guarantee failure but it is certainly going to be a frustrating endeavor, no matter how we go about it.

Juan Delgado and Isaac Mende have written a paper that goes into depth about creating a formal and mature process for developing a framework. They lead off with the comment that failed automation efforts are too common and in many cases, a lack of guidance and understanding of principles is to blame.


Their examples use what is called the “UP Automation Framework”. This set of libraries has been developed by students of Artificial Intelligence engineering at “The University Panamericana campus Aguascalientes” (I think this may be the Aguascalientes my great-grandmother's family is from; its proximity to Mazatlan is the reason I think that to be the case. Not apropos of anything, it just brings a smile to my face :) ).

Juan and Isaac make the point that an automation framework is just as deserving of a development methodology as any project. This gets repeated often but I think it deserves repeating. For an automation project to be successful, it needs to be given a similar level of focus and attention as the min product software development. It can't be seen as an afterthought or as a "nice to have" project.

Is This Testable? - my talk at #PNSQC2019

All right, one down! My talk, I think, went well.

My huge thanks to Jennie Bramble (@jenniedoesthings) for live-tweeting basically my entire talk. Thus, I'm going to capture for posterity her stream of consciousness and again offer my enthusiastic gratitude :).
-----

Jenny Bramble @ #PNSQC2019

@jennydoesthings

 

 Profile

an hour ago, 27 tweets, 23 min read

 

It's time for Is This Testable? A Personal Journey to Learn How to Ask Better Questions From My Applications and Engineering Team with @mkltesthead at @PNSQC #PNSQC2019 

Also, I'm taking selfies with his phone. :D @mkltesthead #PNSQC2019 

mentions I like playing dress up in many forms. @mkltesthead #PNSQC2019 

mentions It's a voyage of discovery! @mkltesthead #PNSQC2019 

mentions The @ministryoftest 30 Days of Testability just got a shout out by @mkltesthead #PNSQC2019 

mentions Testability involves a few things: communication, involvement, and a willingness to look at a product objectively. @mkltesthead #PNSQC2019 

mentions A hypothesis is testable if there is some real possibility of deciding wether it is true or false based on real experience. @mkltesthead #PNSQC2019 

mentions What is light? What is dark? We don't really know, so we can't really test it. But if we use actual numbers for light and dark, then we can test if they are too close. @mkltesthead #PNSQC2019 

mentions "Generally speaking, math is pretty consistent. We'll leave string theory out of this." @mkltesthead #PNSQC2019 

mentions You can't automate a subjective feeling. You can automate objective things. @mkltesthead #PNSQC2019 

mentions Get familiar with your log files! Welcome to your first flash of the blindingly obvious. @mkltesthead #PNSQC2019 

mentions We can all agree this is generally how cars work. Tesla owners, don't get at me. @mkltesthead #PNSQC2019 

mentions We have to look at the things that plug into our application--each component has log files that I need to keep track of. @mkltesthead #PNSQC2019 

mentions The takeaway I want to encourage is to champion unified log files wherever you can. @mkltesthead #PNSQC2019 

mentions How many of you have a good relationship with your sales engineers or customer support engineers? Only a few? Okay, homework: go home, find one, buy them a drink, and learn about their pain points. @mkltesthead #PNSQC2019 

mentions If you can collaborate with someone like a sales engineer for CSR, they can help you have the best test environment available.@mkltesthead #PNSQC2019 

mentions Being able to plug in feature analytics will let you know how people are adopting your features. What features are they actually using? @mkltesthead #PNSQC2019 

mentions We can guess, but real data is much better. @mkltesthead #PNSQC2019 

mentions Automat-able does not necessarily mean testable. @mkltesthead #PNSQC2019 

mentions Second piece of homework: everyone go look up @eviltester and follow him. @mkltesthead #PNSQC2019 

mentions "How am I going to test this?" If you don't ask this for every feature, you will regret it. @mkltesthead #PNSQC2019 

mentions "What are situations I can throw at this problem that go beyond the status quo?" @mkltesthead #PNSQC2019 

mentions I'm a reenactor and a cosplay nerd--I love persona testing! @mkltesthead #PNSQC2019 

mentions Conduct regular show and tell--show people what you've been working on! @mkltesthead #PNSQC2019 

mentions Pair with your developers. Work on tests together! @mkltesthead #PNSQC2019 

mentions Identify your dependencies. Your application is not just your application--it is only as testable as he most remote dependency you have to interact with. @mkltesthead #PNSQC2019 

mentions Heat seeking: check your source code and find where you're making frequent changes. @mkltesthead #PNSQC2019  

Timeless Skills for Modern Testers - a #PNSQC2019 Live Blog

This is going to be fun! I've shared a podcast or a few with Gerie Owen. She's been a frequent guest and panelist on "The Testing Show" over the past few years and I realized today that this was the first time I'd actually met Gerie in person :).

Gerie's talking about developing and enhancing a new set of skills for "Modern Testers", the skills that are timeless and would/should carry us through our entire careers. Selenium is a transient skill. It can be important but it's likely to be superseded by something else. Communication, however, is a skill that transcends tools and will ultimately reward people who emphasize and put the time and attention into mastering that skill.

The irony with this talk as I'm liveblogging it is that we are talking about devices, and the fact that to really understand what is going on, we should put down the devices and actually listen.  Ummm, I plead the fifth ;).

Seriously, though, there is an ever-present reality that we don't really listen to "listen", we listen to reply. I'm guilty of that, I confess. It takes a real commitment to step back and say "I'm going to really listen here. In fact, I'm going to completely de-emphasize my "need to reply". Often I find that that can be surprisingly effective. Frustrating, yes, but effective.

Collaboration is an often-used word and we like to believe that we are working towards collaborating with others on our teams. Again, it's important to understand hat the purpose of collaborating actually is. Why are we doing it? What do we hope to gain from it? Yes, we can be working on a common goal, or we can be trying to accomplish a particular task. Sometimes one of the best ways to collaborate is to steer a little clear of the people who always talk or dominate the conversations. I have found value in starting with the quietest person in the room. Not to the point of having that person dread the idea (I will change up who I ask questions or who I encourage to talk first) but I strive to hold off on the main drivers going first out the gate.

It's not always possible to have complete and open communication all the time but we should all be aware of the silos that can develop. I've had the experience of both being the person with the knowledge, or trying to work with another person with the important knowledge and having to deal with the issues of efficiency vs. effectiveness. I've struggled with this much of my career where I've just said "oh heck with this, I know what to do, I'll just do it myself." Is that efficient? It certainly can be. Is it effective? Maybe not. What makes me think that my knowledge and skill are the best ideas for that moment? The fact is I don't know and I won't know if I always do what I've always done.

Creativity is a dark art. It's an aspiration and so often it's something we feel we are lacking in. The fact is, we are not creative, we create. Let that sink in for a second. Creativity isn't an ephemeral state, it's a verb, an action. We do stuff and we get better at doing stuff if we keep doing it. This is a tangent, but I hope it will make sense. My kids and I are actively involved in re-enactment of different eras. To that end, I work on clothes and garb for many periods. On the surface, you might think that would mean I'm an amazing bespoke tailor. Well, no, not really, but through practice and effort I can do some decent stuff, or at least make items that will pass the "five-foot test". It may fail at super close scrutiny but for the primary purposes, it works well enough. Also, over time, as I get better and more practiced, I get to a point where my work can be looked at more closely and feel good about the efforts I've put in.

One thing we need to be careful of is the biases that can filter into our efforts and activities. As testers, we are often taught to identify biases and fallacies. However, it's one thing to identify fallacies in others. It's another to identify them in ourselves and then counteract them.

Most of the things that Gerie has talked about are all within the realm of each of us to accomplish but they will require effort and practice for us to get good at them. None of us will become experts overnight. It will take time and experience to get good at these things :).

Testing the Untestable - a #PNSQC2019 Live Blog

Happy Monday everyone!

It's that time again. Welcome to the madness that will be the TESTHEAD Live Blog experience. I'm going to be a bit harried today seeing as I have not one, but two presentations to do today. First will be my talk, and then the game that Bill Opsal and I will be presenting today during lunch. However, you all don't really care about my trials and tribulations, you want to hear what the speakers have to say and my "hot take" on what they have to say. At least I'm guessing that's why you are reading his. It's possible you just want to see what dang fool thing I say next. I guess that's OK, too ;).

First out the gate is Angie Jones, and seriously, I don't think I have to tell anyone who will be reading this who she is. If you don't know, just follow her on Twitter and prepare to be amazed :).

Angie has worked a broad number of places and she's sharing stories about a few of those places. She has asked we not share company names since she doesn't like to be seen talking out of turn but she's describing how a number of places she has worked where testers were either non-existent or very rare. One of the areas that Angie has focused on was advertising. We can safely say that ads are somewhat important to companies. Get them right, and you make money. Get them wrong and you can lose money. In some cases a lot of money. When we talk about testing advertising, how do you know that you got it right? Is it possible to test and not actually know if your testing did what you set out to do?

Angie used an example of a campaign focused on women who are into "Space and Technology" and men who are into Art.

Wait, what?

Does something sound odd there? Yeah, that's the point. Imagine having to test that campaign. How do you target those two groups? Is there an overlap? Are you actually getting the message to the people you intend to? How do you know you are targeting the right people? Are you even clear on who the right people are?!

What we are looking at here is an example of what Angie is calling "Testing the Untestable". This resonates with me because my own talk is based around testability (more on that later, I promise ;) ) but here would be a great example of where I might very actively be asking, "How in the world am I supposed to test this?" If it's advertising, odds are I'm not going to be able to test this in some quasi-staging environment. Advertising needs real eyeballs and real interactions. Is it possible to target literal individuals? Are there limitations to that approach?

Angie shared a few different examples of how real-world issues get in the way of actual testing. Minimum users may be required. Users that are too new may not be selectable. How do we get around these areas? More to the point, do we even understand how the application works so that we can confirm something is behaving as expected? Oh, does this resonate with me (LOL!). There are so many instances I can recall where I was sure that I understood what was going on and then realized that I actually had no idea what was going on. Again, this is where asking all of those wonderfully annoying questions can at least get us on a similar page.

So often the greatest enemy to testing isn't desire, or understanding, or technology. So often, it comes down to time. We can look at so many instances where we wanted to do good testing but time just wasn't on our side. Again, I'm so happy to be hearing Angie talking about this because it makes me feel like my talk is maybe as coherent as I believe it to be (jury's out on that, I have a couple of hours to find that out for sure). I will say that this is an example of wanting to make sure that we understand the problem and the testability issues as early as possible, even if it's just to get everyone on the same page of "oh yeah, this is totally testable" or "oh wow, this is going to be kind of ugly!"

Ultimately, we have to be able to get granular enough and get down to hypotheses that can be tested. It's not enough to say "my part works" or "that's outside of the components I need to care about". The entire team needs to have a philosophy of testability and capability of verifying hypotheses. Simple? Yes. Easy? Hardly. Ultimately, the team(s) need to work together and have a shared focus to be successful. It won't necessarily end untestable situations but it should help to minimize them.

En Route to Portland - a #PNSQC2019 "Live Blog"

It's been a while since I've done this.

As I sit in San Francisco International Airport, I am laughing at the change this year has brought about. One o the more interesting ones is just how many times I actually come and go from this airport every day, albeit for different reasons than most. My daughter started working or SFO during the summer. Should you fly on Scandinavian Air or  Air New Zealand, you might get to interact with her :). However, it's because I'm flying out today that I'm seating and gearing up the little blog engine that hopes to.

Yes, it's time for the Pacific Northwest Software Quality Conference, one of my officially most attended and participated in conferences. I will be giving a talk on Monday morning called "Is This Testable? A Personal Journey to Learn How to Ask Better Questions From My Applications and Engineering Team". It's all about my adventures in Testability Land. What, you expected an Accessibility talk ;)? I can't say I blame you as that's been my go-to advocacy for the past five years but this year I decided to try something a little different. This time I'm going to share the details of my experiences with looking at Testability and what I learned from that process. If you are attending PNSQC and want to check it out, here's the details and a link to my paper:


Also, tomorrow at Lunch, Bill Opsal and I will be playing a little game with those who want to attend called "The Release Day Crime Scene":

Join Michael Larsen and Bill Opsal for a game of sleuthing. It is release day and a crime against code and customers has been committed. Can we help identify who done it? Take part in this interactive panel and discuss ways to identify and examine clues, build a case with evidence and make the streets safe for functions and methods.

All in the name of truth, justice, and the releasable way.

So, yeah, if you are at the conference, I look forward to seeing and talking with you. With that, it looks like it's time to check-in and get ready to board. Next stop, Portland :).

My Other Passion Project - ENSIGN RED

This is about as indulgent as they come but I don't care. While this blog is mostly devoted to software testing, it also is the repository for many of my thoughts, ideas and half-baked schemes that I want to record and tell the world about in some way.

There's a running joke in the software testing world about the "rock star tester" and the fact that, at one point in my life, the first two words better described me than the third word did. In truth, I was never comfortable with the term "rock star", mainly because even at the most celebrated our material ever became, any "stardom" was a very local and niche kind of thing. I've actually been OK with that. I got to do all of the things I set out to do as a musician (well, most of them, in any event) without having to experience too many of the downsides. At the age of twenty-five, I chose to "retire" and live a more normal kind of life, one which involved a more standard kind of career and a family. Still, the desire to create, to write, to express my self in what is basically bad poetry, and to sing on a stage has never been far from my thoughts. Now, I'm actively enjoying that process once again and I'd like to tell you all about it.

I am the lead singer of a hard rock/heavy metal band called Ensign Red!

First, a disclaimer. I have no idea what any one person's definition of hard rock or heavy metal is. If you hear us and consider us metal, great. If you don't, that's fine too :).


Second, we are probably never going to win this battle of pronunciation but I'm going to state it anyway. Our name is "En-SIGN Red"! Think of a flag. An Ensign. Specifically, we are named after the red flag pirates would raise that meant "no quarter given" (more appropriately referred to as "The Red Ensign". We switched the words because we felt it sounded more interesting. We've also heard people refer to us as "EN-sin Red", as though it were a formal name of a person holding the military rank. The fact is, many people find it easier to say the latter so we're cool with either. It's all good :).

Third, this is a project that takes up a fairly large amount of my time and attention and I want a place to talk about it. I want to talk about the creative process of songwriting, of oddball source materials, of inspirations that make their way into our songs. Some things will be kept secret (so to speak) but otherwise, any thoughts or questions about being a later in life band member trying to exercise a bit of creativity are totally open.

I may even drop in some software testing relevant content with these posts from time to time. Who knows. In any event, here's to future days :).