PNSQC 2026: Inside Tesla: The Birth of the Software-Defined Vehicle and Quality in the Age of Autonomy

 Well now... that was quick!

We are already at the end of the 2-day conference. Quite a whirlwind experience, and it's neat to see that the last keynote was the first of the PNSQChronicles we did this year.

See the teaser PNSQChronicles video with Florian Rohde

Florian is talking about the idea of the software-defined car, a topic that we have come to see with both interest and annoyance, but that effectively came to be a thing with the advent of Tesla. So what is a software-defined vehicle? It's the idea that, instead of having a vehicle that is limited mechanically, the vehicle is effectively created one time, one version, and the owner/user can choose what capabilities the vehicle provides with how much they have paid and what features they have paid for. In other words, everyone gets a fully loaded car, but how much of the fully loaded car you actually access is up to you and your payment level.

There are a bunch of software components and microcontroller elements inside any Tesla, so what you choose to get and how your vehicle communicates with the broader world can be effectively controlled and optimized for a variety of users. One platform, all sorts of potential configurations and capabilities.

I remember talking about this with Florian in the interview we did and suggested that there was potentially a bit of an annoyance in having to buy/rent features of a vehicle I already own. Florian points out that, in his estimation, this is actually a beneficial arrangement. I remember buying back when I was a starving musician back in 1990 a Ford Escort Pony. This was the most bare-bones entry-level vehicle imaginable. I bought it with no AC, no power steering, no power brakes, manual transmission, crank windows; it didn't even come with a radio (I put that in myself shortly thereafter). I bought this vehicle because it was literally all I could afford at the time (we're talking a vehicle that cost ~$7000 brand new). It was a surprisingly durable vehicle. One of the neat things about an all-manual vehicle is that there's so little on it that could break. I put 168,000 miles on this vehicle in about 6 years, and then I gave it to my brother, who put another 100,000 miles on it. My point is, this was an insanely good value for a vehicle, but I was locked into the choices I could have afforded in 1990. I didn't really mind that, but there were indeed times when I wish it could have done/been more. So what if I could buy that same $7,000 car, but later on, as I got further along in my career, I had an easy way to enable things like power steering, power brakes, turn on the AC, etc. How would I have reacted? Probably favorably, as long as I felt it was a reasonable trade-off for the functionality offered. 

In any event, I want to thqank FLorian for describing this approach that they put together. I'm not a Tesla owner, but as we discussed the details of this talk, I moved over into the "adopt as you need" category. How cool would it be to be able to buy a vehicle that I may only need 2 WD for, but later on I might decide, "Hey, I think I'd want to have 4WD on this vehicle." A very expensive and time-consuming process with any other vehicle, but how cool would it be to be able to just "turn it on?" This is a fast overview of the details of Florian's talk, and I have to say I found it interesting and refreshing to hear about an industry I have zero career involvement in. Am I entirely on board with the "pay-as-you-go" approach of an already loaded vehicle? Not sure, but it makes a lot more sense now :).

PNSQC 2026: "Testing Without Boundaries in the AI Era" w/ Tariq King

Yay! A Tariq talk.

For those not familiar with Tariq King, he's a great speaker, a great innovator, and a fun person to hear speak. The last time I saw Tariq on stage was back in 2024, when he was talking about the capability of generative AI and what it could do at the time. HE was discussing some of the Turing test details regarding human faces and artwork that were generated by AI and what we could tell vs. what we couldn't.

It's been a couple of years since then, and we have now seen some of the outcomes from these Generative AI tools, as well as the blowback and retrenchment of people's opinions on these technologies. It's made an interesting conundrum for me. How do I reconcile the work that I do that may require AI to even be viable vs. the creative industries I am in where Generative AI is absolutely seen as a bridge too far and that, by using Generative AI in the creative spaces of art and in music, you are seen as a "clown show." We weren't there two years ago, and now we are having different conversations than we were then.

This opens up an interesting element of the Generative AI puzzle, even beyond what we are doing, and now discussing when we should or should not use it. We can see that on Spotify, music that has been generated by AI tools is being delisted and forced takedowns applied. I know that's not something that Tariq is necessarily going to address here, but it may be interesting to discuss. If anyone is interested in my take on that, maybe that's a series I can do here.

In my workday world, using AI or being considered for employment requiring my use of AI is not a matter of nuance or delicacy. If I can't demonstrate I know how to use it in both semantic ways as well as autonomous agents working for me, I'm not going to be employed. So in some cases, I may not be able to take some stance and say "AI for thee but not for me". More to the point, AI has given me a desire to be more focused on testing it and trying to figure out specifically what AI is doing, so I can be part of making sure that AI doesn't do the wrong stuff.  

I've been having this conversation with a number of people that I am deeply distrustful of any application I can't reasonably test. In short, I'm not "all in" on AI. I had a funny conversation where I compared AI to the early days of MIDI and how people were hugely dismissive and antagonistic towards using MIDI to make music, especially when it came to sequencing in real time in performances. We don't have those conversations any longer because those MIDI pioneers not only showed how challenging it was but also demonstrated through their creative output that they could actually write music and be inventive in their creations. Will Generative AI get to that point in the music space? On one hand, the music that is being created is polished; it's entertaining, and it's well produced. What it isn't is unique or interesting.  It's a sum total of everything that has come before. Same progressions, same melodies, same structures. And of course it would be. It's been trained on the world of music that's out there. So it can recreate all of that. But what it's not good at doing is breaking the paradigms. It's not good at creating something from the heart. Something human. Something organic. Granted, some people may not care a whole lot about any of that. However, for many of us, to quote the late great Chuck Mosley... "We Care a Lot!"

Back to Tariq's direct points, what do we need to be doing to both test these technologies and determine the future we want to make? One of the fascinating things I came to the conclusion of many years ago was: if I found I could live the rest of my life without working, what would I do? I'll not kid anyone; I'd devote myself and my time to Ensign Red, my band (which, I will be fully transparent and honest, has been how I've been spending the majority of my time since being laid off from my last job ;) ). I also said I'd love to get into snowboard R&D, as at the time I last had that thought experiment, I was a competitive snowboarder (retired since 2005, but it was a great ten years :) ). The third thing I'd want to do is advocate for software quality in any way I could, even if I never got paid another penny to do it. That may be a weirdo thing about me, but it is interesting to think about. So far, I have spent a tremendous amount of time still focusing on software quality even when I'm not actively being paid to do it. The fact is, I love testing, and a part of me has always hoped I'd be able to test forever, or until I shuffled off this mortal coil. Well, I have zero control over who will choose to hire me, but I can advocate anyway, and advocate I will. We are dealing with models and tools that make mistakes, and often pretty fundamental mistakes. 

I like Tariq's thoughts that software testers haven't been taken out of the equation, but that our roles and responsibilities are shifting. We are less needed for our actual physical testing acumen. We are more needed for our judgment and clarity of purpose. To borrow from my comments earlier, MIDI itself doesn't write anything that is new or innovative. That's coming from the mind of the musician. I may have tools that can structure chords and chord progressions, but it's still up to me to know my modes, where I can use them, and how to know when they actually sound good. 

I'm reminded of a comment a dear friend of mine taught me back in the early 1990s, at the start of my testing career (and Chuck, if you by chance are reading this, "hi" and "thank you for everything")... "Every tester, every programmer, every person who works in tech needs to know when they can afford their attitude." That is a statement that has stuck with me over the years. Sometimes we are very valuable, and sometimes the things we think are valuable can be devalued overnight. If we cling too tightly to the technical skills we have,  those can be rendered obsolete quickly. The desire to learn, the desire to explore, and the ability to use those skills to pivot to new realities- that's the valuable skill, and that is the honest, real area where we can see if we can really afford our attitude. 

PNSQC 2026: "Communication Challenges in the Intersection of Diversity, Agility, and Leadership"

I had a lot of fun talking with Dr. Gwen Avington and Diana Larwsen for our PNSQChronicles interview for this talk, so while I can't be in person to experience it, I can at least appreciate a lot of what they are talking about.

PNSQChronicles interview with Dr. Gwen Avington and Diana Larsen

In the talk today, we were presented with an image for a brief period of time and then asked to say what we saw in the picture. As can be guessed, the image was not clear, and there was a lot of interpretation as to what was in the image (as a remote viewer, I had an even fuzzier view of the image, so it was interesting to hear what people in the room saw.

Quality is one of those areas that we talk a mean game about, and we know we should care about it. However, oftentimes our first quality hits (as in the bad kind, not the sensation kind) tend to come about because of poor communication. We don't have a clear understanding of what we should be doing. It doesn't matter who the person in the room is, or what the responsibility is. If we don't have a clear idea of what the problem is that needs to be solved or addressed, we are already at a disadvantage. Communication can absolutely help to mitigate risks and help avoid costs later on because we didn't discuss or understand the issues.

If we have conversations about quality or lack thereof later in the process, we will have to find ways to shoehorn those capabilities or quality measures in at a point when, had we done it earlier, we could have avoided so many issues. This is definitely an issue for newer people. We might feel like we don't have the status or the understanding necessary to speak up in early stages. We often mistake longevity for expertise and knowledge. We may have questions, but we fear being seen as dumb or lacking. Perhaps it behooves us to say, "Hey, I may be new and lacking information, but have we considered 'X' here?" I've had those experiences, and I've learned that it is valuable to have that attitude. Just because someone has experience or longevity doesn't mean they necessarily know if they have considered areas. Even if they have, they may not know that they are not communicating it well.

So how does diversity work into this? Sadly, a lot of the time, there's a set idea of who has expertise and understanding. We make mistakes all the time in this capacity. Everyone brings something of value to the table, and sometimes the best insights may come from people who may not be as obvious. Sexism, racism, classism, and other "isms" can get in the way of who we think may be able to offer insights of value.

We always get down to the question of what we perceive quality to be. One person will have some different ideas compared to another person. We are all used to the idea that "Quality is something of value to some person(s) who matter". That's a part that we often struggle with. We think that we have an idea of what quality is, but we might not be thinking of the areas that are actually important to the people who are most concerned with it. We also need to know when to reinforce and encourage people to show that their input and observations are both important and valued. No one wants to speak up if they think they will be shot down. It takes courage to open up and communicate. Additionally, help foster an environment where they can step up to speak and know that we have the space to speak in the first place.

 

PNSQC 2026: "With Great Judgment Comes Great Great Responsibility" with Paul Grizzaffi

Good morning again. This is one of the reasons I am sad to not be in person at PNSQC. Paul and I go way back, and I always have fun when I hear him speak. We frequently bond over our love of music, especially heavy metal, so Paul, just know that I'd be bending your ear if I were there ;).

How many of us know what a Pyrrhic Victory is? Pyrrhus of Epirus is famous for winning battles with absolutely dreadful losses, so much so that the term "Pyrrhic victory" means a win that comes at such a great cost it's arguable if it was actually worth it.

PNSQChronicles Podcast previewing the details of Paul's talk

AS testers, we are often asked to "do the right thing" (a Spike Lee joint ;) ). For the few of you out there who chuckled, you're my people... I love you :). Automation is something that has a specific purpose. We can argue all day long if that purpose is actually needed or necessary, but for many of us, our jobs would be rendered very difficult without it. We all do some form of automation if we have done something enough times. That's often the first wave of automation, the "Oh my stars! If I have to do this process one more time by hand, I'm going to lose my mind." I'd argue that the people who are least fond of these steps are probably the ones who will embrace and get good at automation the quickest.

In this current age of AI, we have a fresh new danger. AI can write all of the tests and even the test code for us. In truth, it can. However, if we think that we are going to get "one and done" tests that will be resilient, durable, and effective over time, there may be some surprises in your future. In my last gig, I had the experience of doing both roll-your-own automation and working with AI to write the test code and run it. As I was working through those processes, I came to a simple conclusion. The time to have a vetted and confirmed version of automated tests that AI generated vs. tests I created myself... was roughly equal. That's not to say that I'm some amazing automator (I mean, I know some stuff ;) ) and no question, AI can crank out code so much faster than I can. Still, when I question the code and ask why it's doing what it does, the rounds of revisions to get to a truly working and robust version of tests, at the end of the day, were roughly even.

For those who know how I do these, I'm writing this as I'm listening to Paul, but I'm not going to regurgitate his talk. I am going to consider what he is saying and apply it to my own experiences and see what rings true and what I might learn something fresh from. There are absolutely costs associated with using AI, even if it feels like it is cranking stuff out that feels fast, fresh, and amazing. One of the projects I worked on was to take our Documentation as published and make an analysis gap of what tests we had and what options our documentation was saying we did. While I will not claim that's the be-all and end-all of what should be automated, if you want to do an interesting exercise, I recommend doing that process. You can do it all on your own, or you can run it through AI, but either way, it gives you a simple place to start. Do we stop there? Of course not, but at the minimum, you can say that you are making tests that at least cover the areas that you would want to encourage your customers to be doing. Start with the happy paths and then look for the areas that could trip you up. Look for the error code that exists, and then try to create tests that surface all of that error code effectively. Again, whether you do that as a roll-your-own project or you turn over the considerations to Claude Code, OpenAI Codex, or some other AI methodology, ultimately, we want to know what's happening and why.

As a musician, I have many tools that are at my disposal. I use Digital Audio Workstations to do actual recording; I use multiple amp sims and effects that are built into plugins, and I have lots of virtual instruments (even singing voices) so that I can experiment with various things. Some people might remember the early 80s and the advent of MIDI and sequencing software. I remember well the antipathy people had toward MIDI back then, and the feeling that "people were pretending to be musicians" by using MIDI and sequencers. I'd argue that any performer who chose to bring MIDI onstage in those days and orchestrate all of that in a live performance was a braver soul than I. It was easier, in my mind, to just perform than try to wrangle all of that stuff in real time. I remember the snide remarks coming from the metal community especially. The irony is that, a couple of decades later, you would be hard-pressed to hear a metal album produced today that isn't using all of the software recording features that exist today. The beat alignment to the grid and the level of precision that is orchestrated at the point of recording is absurd now. Is it easier than performing it? It might well be. Is there still a performance to be heard there? That's another question entirely. If we are going through and we are massaging every guitar part with quantizing, aligning every beat to the grid, and using Melodyne to clean up every vocal note's tuning, what we end up with may be a pristine recording, an elaborate artifact, but we can't call it a "performance".

To borrow from Jurassic Park, sometimes we get so caught up with what we can do that we might lose sight of what we should do. Knowing when those two options align, and when they don't, is the big thing. I wish I could be there in person to riff with Paul about this, but I hope he enjoys this rambling missive in the meantime :).

PNSQC 2026: The AI Assurance Imperative: Quality Engineering for the Agentic Era w/ Jonathon Wright

Hello everyone, and for those at PNSQC this year, I hope you are having a wonderful time. I, sadly, am not in Portland for the first time in many years, and I am genuinely sad about that. There are reasons I can't be there this year, and I hope that I can participate next time, but for now I am happy and grateful that virtual PNSQC is an option.

Many of you may have seen my PNSQChronicles interviews over the past few months where I've had a chance to talk wth many of the presenters, and Jonathon was a fun interview for me, so I think it's a good bet that you all will enjoy his talk, too.

Jonathon just made a point about the latest series about AI that Netflix has been airing, and pointed out that Sam Altman has said that the most important thing that these companies could be doing is testing. 

At the same time, we are dealing with AI agents that operate independently, generate code, run tests, make choices, and develop systems without human involvement. So how do we actually test stuff like this?

Self driving card, next-gen technology, and everything we are looking at today is aiming for stuff that is wildly different than what we are used to. We are talking about new sciences, ways of using technology that we have not even considered, and many of us are still testing things for the paradigms that currently exist. Nothing wrong with that, but the most value we can add is to actually consider how to test for what comes next.

The Human in the Loop is a common phrase, and something we should all make sure that we advocate for. We cannot let these systems run without some level of human judgment making sense of what's coming out of it. We have the phrase "Agentic" now that we need to embrace. It's a fad; it's an experiment. We've all heard that, right? That's likely not going to be true. We have reached a point where so much stuff is coming out now that there is simply no way that we can keep up with it, much less test it effectively.

So while we have Sam Altman saying the most important thing is testing, why does it feel like actual testing is not being included in this conversation, or if it is, it feels very peripheral? We are currently looking at a situation where seasoned engineers are struggling to find work (raises hand: I'm one of those people at present). Don't get me wrong, I haven't given up hope here, but I am certainly curious as to how we are going to address these situations if literally seasoned programmers and engineers are being pushed out at this stage. So if testing is going to be needed in this Agentic AI age, who's going to do it if we are actively removing people? This seems to be an interesting conundrum. Either AI systems are going to have to learn how to effectively test on their own, or people are going to have to help bridge that gap. Again, it's an interesting question: how much are we willing to give of ourselves to help make AI better able to test itself, and perhaps risk losing our place in being able to be part of that conversation in the first place?

The intriguing part of all of this is wrapping our heads around ways that we as people could be effective in this new world. What methods will we use? What approaches are we going to pioneer that we have not yet had the chance to get a feel for? How much of what we are going to need to test do people actually even understand at this point? I don't say that lightly. To borrow from a favorite quote: "The significant problems of today will not be fixed by the same logic and reasoning that created them." (I'll have to go back and give that proper attribution and wording, but that's how I remember it right now). Jonathon is talking about the idea that we can literally load multiple testers' experiences and approaches into a database and effectively have those testers working for him (including me, if they decided my corpus of test writing is worth anything ;) ). That's an intriguing option to think I could have Angie Jones, Randy Rice, or Matt Heusser at my immediate beck and call because I could create agents that would test based on their writing and principles that they have published to apply to my own tasks. 

This is a wild new world we are playing in now. I don't have a crystal ball, and I can't say what will be happening next year, much less five years. The key Jonathon is trying to point out is that we don't have five years to absorb this. This is a reality we are going to have to embrace really quickly, and we ignore it at our peril. It's proving to be an expensive option for many, but don't assume that means that it's going to just fall apart. AI is here, enmeshed, and not going away. Testing it is going to be critical to it not running off the rails. I'm glad to see some people are actually considering the testing needs and realizing the elephant in the room. How we go about doing it is anyone's guess.

Hello From Cocoa Beach: CAST 2026 is Underway: "How Avoiding Conflict Is Breaking Your Team"

 This is going to be an interesting and different CAST this go around. While I have live blogged conferences for years, I will not be doing much of that this go-around. That is because I am going to be doing live webcasting and video casting. We are launching a Webcast called "Responsive Ilities", we being me and Justin Bench. I will be posting the links to those as they go live. They will be fast clips, fun interactions, and we hope a good representation of the fun we are having here.

There will, of course, be some sessions we will not be vidcasting, such as our morning keynote right now. We are starting the conference with Dr. Jen Fry and her talk, "The Bugs Aren’t the Problem: How Avoiding Conflict Is Breaking Your Team"

The key message here is, "Software teams don’t struggle because they lack tools, frameworks, or technical skill. They struggle because the conversations that should happen… don’t."

I know this to be true, in some capacity, for every job I have ever had, both in my testing life and in the other things I do. I'm the enthusiastic, bubbly dude who likes doing, likes getting cool things done, and if there are conflicts or issues, we just let them roll off, and we figure we will just muscle through with our enthusiasm, and if we just kick enough butt, those issues will just disappear.

"Stop me, oh oh oh stop me.... stop me if you think that you've heard this one before!"

As you all know, I tend to riff on the ideas I'm hearing. I don't summarize the talk; I think through their points, and I take what I'm hearing and think through it with my own experiences.

I am intimately familiar with seeing something that I think might be wrong, but I'm unsure. Or I might want to make sure I can articulate everything before I vocalize my concerns.  Where does that come from? It comes from voicing concerns too early in the past, or thinking something is wrong but not having a full picture. What's the root problem here? I have an anxiety of looking like I don't know what I'm doing. Ever heard the phrase (paraphrased), "If you sit silently, you may be seen as a fool, but you can say what you are thinking and remove all doubt!" Yes, that's real. We all do that to a certain degree. We fear being seen as foolish or unwise, so the easiest tactic is to hold our thoughts until we are really sure. Maybe, possibly, we will gather what we need to make sure we have a thorough understanding and can present everything brilliantly. More often than not, though, life comes at you fast, and you are onto something else. Then when the problem rears its head, and others see it, you sit there thinking, "oooh, I knew that was an issue, but I didn't say anything because I didn't want to look stupid." Well, now the problem is out in the open, and I look more stupid because I held my peace until I was sure. 

Okay, so how can we counteract this very real issue people face? Part of it is practicing these words (again, this is how I handle this, or at least try to).

"I'm not fully sure I have a handle on this, but I'm noticing something that concerns me about [key area]."

"I may be missing something, but are you seeing an issue with [component]?"

"Are you familiar with this [area of misunderstanding] because I think I may have an issue... but again, I'm not sure."

Am I risking being seen as a fool right now? Absolutely. I'm sharing the fact that I'm "ignorant". Understand, that's a loaded word for many, and we hate hearing it about ourselves. Truth is, everyone is ignorant about a lot of things. Ignorance comes from Latin "ignorantia" and "ignorare", meaning "not to know" or "to be unacquainted with". I try to remind myself that "ignorance" is easily solved. I can learn, I can ask, and I can be taught. 

From a different place, I am often seen as the "psychic glue" in a group. That's a phrase a guitar player in one of my previous bands coined for me. He said that I had a tendency, due to my enthusiasm and general overall good nature (I think the phrase now would be termed 'Golden Retriever Energy' (GRE), a phrase that I actually love :) ), I make it possible for people who would struggle to interact a buffer, so that I cam the person in between who, because of my GRE, I can be that central glue point so we can all work together. Conflict doesn't stick to me or affect me when it's other people's conflict. If other people have issues with each other, I can be the go-between and easily smooth that challenge. The problem with GRE is that when the conflict comes to us, we can internalize it and think we are the problem. So we strive to make others happy and be cheerful, to hopefully cause the conflict to dissipate. I'm good at soothing issues. I'm not so good at actually resolving the conflict. The funny thing is that sometimes, my enthusiasm can have me jumping the gun at certain things, and then I get yelled at... but if people are yelling at me, they aren't yelling at each other. Since I have GRE, I can take having people be annoyed with me, and I can laugh or shrug it off easily. Net result: I glue everyone else together, and we function well,,, but you see the problem, right? Being psychic glue can be a good thing when it's applied correctly, but glue can also be messy if it's not applied correctly. Thus, I need to be willing to step back and speak more openly about the conflicts and be willing to not be seen as the happy-go-lucky golden retriever. I may not be able to be that glue at the moment, but I may be able to be the glue that is needed later on.

That's a great start. It's good to be back. Did you miss me :)?

Breaking the Cycle: Leading Organizations That Thrive Amid Constant Change with Allan Rennebo Jepsen (a PNSQC Live BLog)

Well, that was fast! We have reached the end of the conference with our final keynote. There has been a lot to absorb and learn over the past few days, and I've enjoyed my time here. With that, here's the closing talk for PNSQC 2025.

Allan Rennebo Jepsen is a leadership expert and author based in Copenhagen, Denmark, who has spent over two decades helping organizations adapt to change rather than fighting it. Allan opened up with an example of an online banking initiative that went horribly wrong, where everything that could go wrong did, to the point where the CEO at a corporate meeting said that a large majority of customers of the bank would not be around a year from now. Meaning their business model was cratering and they risked going out of business. Change is needed, but how in the world are they going to do it? Large banks are optimized for compliance, and because of that, they are literally designed to preserve the status quo. 

PNSQChronicles: Leading Organizations That Thrive Amid Constant Change with Allan Rennebo Jepsen

So what ends up happening? “We are trapped in a loop of meetings, updates, and alignment sessions, with initiatives and plans made, but somehow, progress keeps slipping away.” 

This ongoing “cycle of coordination” eats up energy and time while creating an illusion of progress. Wouldn't it be nice to be able to break the cycle and replace it with focused, adaptive, and empowered teams with goals and efforts that can actually move the organization forward?

As Allen points out, the one true constant is change, and change is not slowing down. New technologies, shifting customer expectations, and global competition mean that businesses must evolve faster than ever before. “Success today requires a new paradigm, one that values bold leadership over small tweaks.”

Many organizations instead fall into reactive patterns and mindsets. They organize based on what the business thinks is important, rather than looking at what the actual customers need. Every new change triggers a cascade of alignment meetings, but before that alignment solidifies, another change hits. The result is paralysis through coordination.

What will it take to break out of this vicious cycle? Genuine adaptability, that's what. Organizations that show that they can learn, adjust, and act decisively in uncertain conditions are the ones that will ultimately thrive.

What do we need to do to actually make change happen?

Strategic Prioritization

We need to focus our energy where it matters most. Not every initiative deserves attention. Leaders need to be ruthless in prioritizing efforts that drive real impact and cut through the noise of constant change.

Winning Together

Too many departments chase local wins at the expense of the organization’s overall success. Instead, focus on organizational collaboration and allow for genuine long-term thinking. Optimizing one team while degrading the system as a whole is counterproductive.

Thriving Amid Change

When teams are empowered to make decisions and take action without waiting for endless approvals, they can develop true adaptability. With those experiences, the team can get better and address challenges related to change. Success likelihood grows when teams have both the authority and the accountability to act quickly. Resilience and adaptability are learned competencies, not traits reserved for a few innovative companies. If you want to be resilient and adaptive, you need to practice and work towards being resilient and adaptive.

Allan encourages creating a “low-risk, high-impact” framework for building organizations that are genuinely adaptable. Continuous learning loops, clear ownership, and a culture of shared goals help with and reinforce the opportunities to effectively adapt to change.

As Allen points out, “These organizations didn’t get there by accident. They created the conditions where adaptability could flourish.”

Artificial intelligence, decentralized decision-making, and global competition are reshaping how organizations operate. Instead of fearing disruption, leaders should view it as a constant condition to master. Focus on adaptability as the ultimate organizational competency.

“You can’t predict the next disruption, but you can prepare your organization to handle whatever comes next.”

Key Takeaways

- Prioritize what truly matters. Avoid being spread thin across too many initiatives.

- Empower your teams. Give them the clarity and authority to act.

- Collaborate across silos. Success comes from shared goals, not isolated wins.

- Embrace continuous learning. Every challenge is a chance to adapt and improve.

If there's any one message or idea to bring home today, it is that change doesn’t have to be chaotic. Leaders who build adaptable, empowered teams will be the ones who turn constant change into a competitive advantage.

Want to know more? Allan’s book “Impactful Organizations – And How to Become One!" expands on many of the ideas shared today. 

Monsters & Magicians: Testing the Illusions of Generative AI with Ben Simo (a PNSQC Live Blog)

Day 2 of the main program is underway (Day 3 includes the workshop presentations). It's crazy to see/feel how quickly this event goes by. As always, I've had a great time at this event and enjoyed my interactions with everyone. It's especially neat to realize how many people I know in this space, and when keynote speakers are literal friends, such as today with Ben Simo. 

Machines to do things beyond our physical or mental powers have existed for thousands of years. We can go back to the Antikytheria Mechanism for what may quite possibly be the world's first "artificial intelligence," depending on how you want to interpret that term. Over time, as we have come to grips with and developed an understanding of the rules, laws, and repeatability of activities, what was magical once upon a time has become commonplace in our everyday use. 

PNSQChronicles: Brief Interview of "Monsters and Magicians" with Ben Simo on YouTube

We now see Large Language Models and predictive text generation as the current amazeballs part of our reality. Many people are excited about these technologies, but at the same time, there are many risks surfacing, with reports of organizations suffering actual harm or damage because of using AI tools. We have heard of apps that have jacked up rates arbitrarily, published legal documents with no basis in law, fact, or reality, and taken models of "virtual people" who learn from interactions and the biases and inputs trained these models to be incredibly racist and hateful. 

These situations point to an interesting set of questions: how specifically can we as testers benefit from this wild new world of seemingly random query and response systems? How do we test software that produces inexplicable fuzzy outputs? At the end of the day, software deals with patterns and algorithms. We have technologies such as machine learning, clustering, and ways that data can be grouped and sorted. If we give LLM's a closed data set of information and ask it to work with just that information, it does a remarkably good job of transforming or "creating" work and assets. The key here is that we have given it a known and bounded set of information. Because it is bounded, it is working with a known set of information and can be guided specifically as to what to do with it. As we open it to the outer world and give it fewer controls or restrictions, we open the model up to having to look at vague clusters of data with potentially dubious provenience. A great example of this that I saw in practice during one of the workshops was with the idea of creating spec documents in markdown that would reside at the base of your document tree. By making the spec document the oracle of choice, and giving the instructions that the spec document was the arbiter of what the model was to do, we limit the chance of hallucinations and odd reactions considerably. Not completely, but we make it much easier to track what the model is doing.

An example I had fun with recently was when I heard through a podcast a story of the son of Hephaistos who went to Olympus during the waning days of the Greek pantheon's influence and took a remnant of the fire from Olympus (in an homage to the myth of Prometheus). However, something about this seemed "off", as in it was being presented as an ancient myth, but it clearly wasn't. Could we identify where the original story came from? Through various prompts, reviewing the text transcript of the story, and other clues, the LLM determined that, indeed, it was a modern story being told in the manner of an ancient Greek myth, and even noted that the delivery of the story in meter and timing mimicked closely the delivery of Hesiod. I was not able to determine who wrote the story or where to find it on the internet, but the details it did provide were interesting. Many of them felt fanciful, but all of them felt plausible. That's the danger with LLM output. Unless we are diligent, the very plausibility of the output could be accepted easily as though it were fact. Closer inspection found numerous areas that were not accurate (referencing older myths that I was aware of and had history with, but attributing individuals and characters that didn't belong there). People who may not have this knowledge or familiarity might accept what's being presented as fact because it flows so naturally and just "feels right and authoritative". 

In these cases, the Boolean PASS vs FAIL rationale doesn't work. It's not that the tests pass or fail, but that large elements do pass, but there are outputs that are "off". It's not a total fail, but it's also "corrupted" in a way that we cannot simply rely on the output. Additionally, we can run tests multiple times and get slightly different outputs with the exact same information being presented. In my own world of testing AI, we use a variety of benchmarks and monitors that help us determine if the models are behaving in ways that we expect them to. We have a variety of tests and models that allow us to determine things like Performance drift, comprehensive analysis, bias drift and disparity, the currency of the data, and comparing for homoscedasticity (a $10 word that means looking at the variance of the errors in a model and determining if it is constant/consistent across all of our observations). 

A neat tool that we have is the "AI Risk Repository" which helps us identify risks and domains of use of models where the various risks can be found. By looking at the areas that are potential risks, we can better be informed or consider aspects we can apply to our testing efforts.

Ultimately, one of the key takeaways from this talk is the idea that we are at peril of being beguiled by the magic surrounding us. We want to be responsible with our use of AI, and thus, we need to test and consider how best to apply what we learn and spread that knowledge amongst our colleagues. Magic is often sleight of hand, and it's important that we understand and can understand how that sleight of hand is performed.

Dance Bot Showcase at PNSQC: Where Robots Move to the Beat of Quality (a PNSQC Live Blog)

 This is an interesting and unusual activity... unless you have been coming to PNSQC for at least the past 15 years. The various LEGO League entries for talks and demonstrations are nothing new. A robot dance competition? Okay, that's different ;).

This year, PNSQC is hosting the first-ever Dance Bot Showcase—a celebration of creativity, code, and community!

We are starting out with a Q&A with the various team captains and learning about the challenges they faced putting on this event, and determining the criteria necessary to make this event happen.

PNSQC 2025 Robot Dance Off

Teams featured are Gear In Motion, The Beaver Bots, and Mechanical Mages (placing 3rd, 2nd, and 1st, respectively).

The various robotics teams described a number of challenges and issues they faced while preparing for this event or for other events that they have participated in. This particular demonstration is a literal "robot dance off", meaning they need to program and choreograph robots to actually dance in sync with music. These teams demonstrate technical skill as well as showcasing creativity and artistic flair. It’s more than just a performance—it's a hands-on demonstration of how software quality, testing, and intelligent systems intersect in the real world.

Each of the youth teams engages with key software concepts like adaptability, precision, and resilience through robotics. With a mix of live performances, a student-led fireside chat on software and testing challenges related to robot programming and choreography, the Showcase connects generations through shared curiosity and learning. Oh, but that's not all. Judges will develop end-user quality criteria to determine how to evaluate and judge the participants and their robot entries. Just as we would evaluate web application software quality from the end-user perspective, we will see a variety of aspects within the dance bot competition to highlight various quality characteristics.

In addition to the three local teams, we have three international teams (from the Dominican Republic, the Democratic Republic of Congo, and Mauritius) that have also contributed presentations of their dancing robots (also included in the video footage).

Have I piqued your interest? I hope so, because as soon as I can get the footage up, we will showcase the dancing robots on our YouTube channel as well as here.

Being a Change Agent with April K Mills (Welcome to PNSQC Live Blog Stream)

Hello all, and welcome to PNSQC, coming at you from Portland State University Place Hotel and Conference Center. Some of you may be hearing concerning things coming from Portland and having been here for two days now in real time, it is absolutely quiet and normal here (though I must confess that the media I have seen of protests with people in frog and chicken suits, I love how absolutely unserious Portland is. We literally started the conference with "The Beat Goes On" marching band (I will upload and link media when I get the chance).

Everyone is a Change Agent

Our first talk/keynote today is coming from April K. Mills, and her talk is "Everyone is a Change Agent". I had a chance to discuss this wth April in our PNSQChronicles vidcast (will link here later), but the key point here is that change agents don't have to be high-power, in-charge people. Every one of us has the capability, and I'd argue the responsibility, to be the change agents our organizations need. To borrow from an idea I agree with a great deal, "everyone needs to stand for something, even if doing so means we have to stand alone".

One of the ideas that April is recommending we start right now to do and be a change agent is to stand against and fight the spread of "AI Slop." How many of us have found ourselves literally struggling with and drowning in AI-generated content in our organizations that seems to say a bunch but doesn't really mean anything? I have run into this a fair bit in social media, but it is absolutely creeping into publications and corporate marketing missives. I saw a person online coin a phrase I have now started using: "AI;DR".

Note: I use this when I am seeing obvious AI output that has been generated, reads as completely soulless, and you can see that much of it is not factual or is genuinely wrong in various areas. In these cases, I do believe that announcing "AI;DR" is an excellent response.

YouTube: PNSQChronicles episode with April K. Mills

One of the things I have always appreciated (and I'm quoting Matt Heusser is the wording here) when I come to conferences like PNSQC is that I am not looking to upend the entire world with everything I learned. First of all, it's not practical, and second, systems are developed over time, and there may well be entrenched interests to overcome. Thus, it's not practical to come back from a conference with everything under the sun to overturn everything we do. Instead, we need to find a few areas that we can implement and do so without needing permission to do those things. All of us can do that. We find a way to add a new approach, process, or method based on what we learned, and then, in a few weeks or months, when people are curious as to how we have made key improvements, we can then say/demonstrate what we have been doing. There may be varying barriers to exactly how much we can do in certain areas, but we can all do something in a different way. 

When we can implement an under-the-radar change, it's often a good idea to try it out quietly and see if the change is worth pursuing. Often, we can determine pretty quickly if our brilliant idea may not be so brilliant (or at least we discover the key contexts that help explain why we haven't tried doing this before). Make no mistake, that in and of itself is often a valuable exercise. 

Sometimes the best question we can ask is "Why?" There is an idea called the "Power Paradox" that comes from the idea that people can only do so much because they don't have the power to make changes. We hear that in politics and senior management all the time. If only we could get the people at the top to agree with us and make the changes necessary. Truth be told, we don't always need to do that, and sometimes, the best way to break down that false assumption of power is to ask why we do something (strategically, don't become the kid in the back seat asking, "Are we there yet?" every five minutes). By asking "Why?" at strategic times, we may not sway the people at the top, but we may get more of us at lower levels to also ask why, and if more of us start asking why, that often triggers those higher up to realize that this is an issue that is not serving the people it intends to. Sometimes it takes more than just asking why, but strategic action often starts from just that question.

Key Takeaway: Being a change agent doesn't have to be a major initiative, and we don't have to reinvent the wheel to be a change agent. We don't need to have the right boss, with the right budget, with the right revenue, at the right time to make changes. Often, we can make changes just by trying to do some different things, and often, we can do them without having to ask permission to do so. Start there first and work your way up :).

Building Your Tester Survival Guide with Dawn Haynes: A CAST Live Blog

For the past couple of days as we have been getting CAST ready to go, I've gone and done a number of item runs, food stops, and logistical troubleshootings with Dawn Haynes, which is a common occurrence over my years with CAST. Dawn and I have frequently been elbows deep in dealing with the realities of these conferences. One funny thing that we quipped about was the fact that any time we appear at conferences together as speakers, somehow we are always scheduled at the same time (or at least a lot of the time). I thought that was going to be the case this time as well but NO, the schedule has allowed us to not overlap... for ONCE :)!!!

I first learned about Dawn through her training initiatives long before I was actually a conference attendee or speaker. She appeared as a training course provider in "Software Test and Performance" magazine back in the mid 2000s. Point being, Dawn has been an expert in our field for quite some time, and thus ,if Dawn is presenting on a topic, it's a pretty good bet it's worth your time to sit and listen. Daw is the CEO and resident Testing Yogini at PerfTestPlus, so if you want to get a first hand experience with her, I suggest doing it if you can. For now, you get me... try to contain your excitement ;).

Onekey area tht Dawn and I are both aligned on and wholeheartedly agree with is that we are individually as testers, quality professionals, whatever we call ourselves, we are responsible for crating our own careers and if you have been in testing for an extended period, you have probably already had to reinvent yourself at least once or twice. Dawn wants to encourage all testers and quality professionals to actively develop their survival instincts. Does that sound dire. It should... and it shouldn't. Dawn's point is that testing is a flexible field and what is required one day may be old hat and not needed the next. As testers, we are often required to take on different roles and aspects. During my career, I have actually transitioned a few times into doing technical support over active day to day testing.  That's a key part of my active career curation. I've actually been hired as a tech support engineer only for them to realize that I have had a long career in software testing and the next thing I know, I'm back and actively doing software testing full time. In some cases, I have done both simultaneously and that has kept me very busy. My point is, those are examples of ways that testing skills can be applied in many different ways and with many different jobs. 

Automating stuff, doing DevOps, running performance or security audits, or looking at areas your organization may not be actively working towards and playing around with those areas. As you learn more and bring more to the table, don't be surprised that you may be asked to do more of it or leverage those skills to learn about other areas.

Some areas are just not going to be a lot of fun all of the time. Sometimes you will take a while to get the skills you need. You may or may not get the time to do and learn these things but even if you can just spend 20 minutes a day, those efforts add up. Yes, you will be slow, unsure, and wary at first. You may completely suck at the thing that you want to/need to learn. You may have deficiencies in the areas that you need to skill up on. The good news is tat's normal. Everyone goes through this. Even seasoned developers don't know every language or every aspect of the languages they work with. If you are not learning regularly, you will lose ground. I like Dawn's suggestion of a 33/33/33 aproach. Learn something for work, reach out to people, train and take care of yourself. By leveraging these three areas, we can be effective over time and have the healeth and stamina to actually leverage what we are learning. We run the risk of burning ourselves out if we put too much emphasis on one area, so take the time to balance those areas and also, allow yourself to absorb your learning. It may take significant time to get good at something but if you allow yourself the time (not to excess) to absorb what you are learning, odds are you will be better positioned to maintain and even grow those skills.

One of the best skills to develop is to be collaborative whenever possible. Being a tester is great but being able to help get the work done in whatever capacity we can is usually appreciated. A favorite phrase on my end is, "There seems to be a problem here... how can I help?" Honestly, I've never to date been turned down when I've aproached my teams with that attitude.

Glad to have the chance to hear Dawn for a change. Well done. I'm next :).   

Empathy is a Technical Skill With Andrea Goulet (PNSQC)

 

Today has been a whirlwind. I was up this morning before 5:00 a.m. to teach the penultimate class of my contract (sorry, I just love working that word into things ;) ) but suffice it to say the class is still happening while PNSQC is happening. That has made me a little tired and thus a little less blogging today. Add to that the fact I was called in to do a substitute talk in the afternoon (glad to do it but that was not on my dance card today) and I'm really wondering how we are at the last talk of the day and formal conference. regardless, we are here and I'm excited to hear our last speaker.

I appreciate Andrea talking about this topic, especially because I feel that there has been a lot of impersonal and disinterested work from many over the past several years. I was curious as to what this talk would be about. How can we look at Empathy as a technical skill? She walked us through an example with her husband where he was digging into a thorny technical problem that was interrupted by Andrea asking him for a moment. His reaction was... not pleasant. As Andrea explained, she realized that he was deeply focused on something so all-consuming that it was going to be a big deal to get his attention for needful things. Instead of it being an ugly altercation, they worked out a phrase (in this case, "Inception") to help see when a person is on a deep dive and needs to be in their focused state, at least for a little while longer. While I don't quite know that level of a dive, I have times in my own life when I get caught up in my own thoughts and I bristle when someone interrupts/intrudes. By realizing these things, we can not just recognize when we ourselves are focusing on deep dives, but we can also recognize when others are as well. This is a development of our own empathy to aid us in the process of understanding when people are dealing with things.

Okay, that's all cool, but why is this being called a technical thing? Because we are free and loose with the use of the word "technical". Technical comes from the Greek word "Techne", and techne means "skill". That means any skill is technical when we get down to it. It also means it's a skill that can be learned. Yes, we can learn to be empathetic. It's not something we are born with, it's something we develop and practice. Motivation likewise drives empathy. In many ways, empathy can be a little mercenary. That's why we get it wrong a lot of the time. We often want to reach out and help in ways that we would want to be helped, and thus our empathy is highly subjective and highly variable. Additionally, empathy grades on a curve. There are numerous ways in which we express and experience empathy. it's not a monoculture, it is expressed in numerous ways and under different circumstances and conditions. There are a variety of components, mechanisms, and processes that go into our understanding and expressions of empathy. It's how we collaborate and solve complex problems. In short, it's a core part of our desire and ability to work together.

Andrea showed us a diagram with a number of elements. We have a variety of inputs (compassion, communication) that drive the various mechanisms that end up with a set of outputs. Those outputs come down to:

• Developing an understanding of others 
• Creation of Trust
• A Feeling of Mutual Support
• An overall synergy of efforts   

 Empathy requires critical thinking. It's not all feelings. We have to have a clear understanding and rational vision of what people want, and not specifically what we want. 

On the whole, this is intriguing and not what I was expecting to hear. Regardless, I'm excited to see if I can approach this as a developed skill.

Automation, You're Doing It Wrong With Melissa Tondi (PNSQC)

This may feel a bit like deja -vu because Melissa has given a similar talk in other venues. The cool thing is I know each time she delivers the talk, it has some new avenues and ideas. So what will today have in store? Let's find out :).

What I like about Melissa's take is that she emphasizes what automation is NOT over what it is.

I like her opening phrase, "Test automation makes humans more efficient, not less essential" and I really appreciate that. Granted, I know a lot of people feel that test automation and its implementation is a less than enjoyable experience. Too often I feel we end up having to play a game of metrics over making any meaningful testing progress. I've also been part of what I call the "script factory" role where you learn how to write one test and then 95 out of 100 tests you write are going to be small variations on the theme of that test (login, navigate, find the element, confirm it exists, print out the message, tick the pass number, repeat). Could there be lots more than that and lots more creativity? Sure. Do we see that? Not often.

Is that automation's fault? No. Is it an issue with management and their desire to post favorable numbers? Oh yeah, definitely. In short, we are setting up a perverse expectation and reward system. When you gauge success in numbers, people will figure out the ways to meet that. Does it add any real value? Sadly, much of the time it does not.   

Another killer that I had the opportunity to work on and see change was the serial and monolithic suite of tests that take a lot of time to run. I saw this happen at Socialtext and one of the first big initiatives when I arrived there was to see the implementation of a docker suite that would break out our tests into groupings split into fours. Every test was randomized and shuffled to run on the four server gateways. We would bring up as many nodes as necessary to run the batches of tests. By doing this, we were able to cut our linear test runs down from 24 hours to just one. That was a huge win but it also helped us determine where we had tests that were not truly self-contained. It was interesting to see how tests were set up and how many tests were made larger specifically to allow us to do examinations but also to allow us to divvy up more tests than we would have been able to otherwise. 

Melissa brought up the great specter of "automate everything". While granted, this is impossible, it is still seen forlornly as "The Impossible Dream". More times than not, it's the process of taking all of the manual tests and putting them to code. Many of those tests will make sense, sure, but many of them will not. The amount of energy and effort necessary to cover all of the variations of certain tests will just become mind-numbing and, often, not tell us anything interesting. Additionally, many of our tests that are created in this legacy manner are there to test legacy code. Often, that code doesn't have hooks that will help us with testing, so we have to do end runs to make things work. Often, the code is just resistant to testing or requiring esoteric identification methods (the more esoteric, the more likely it will fail on you someday). Additionally, I've seen a lot of organizations that are looking for automated tests when they haven't done unit or integration tests at lower levels. This is something I've realized having recently taught a student group to learn C#. We went through the language basics and then later started talking about unit testing and frameworks. After I had gone through this, I determined if I were to do this again, I would do my best to teach unit testing, even if at fundamental levels, as soon as participants were creating classes that processed actions or returned a value beyond a print statement. Think about where we could be if every software developer was taught about and encouraged to use unit tests at the training wheels level!

Another suggestion that I find interesting and helpful is that a test that always passes is probably useless. Not because the test is necessarily working correctly and the code is genuinely good but because we got lucky and/or we don't have anything challenging enough in our test to actually run the risk of failing. If it's the latter, then yes, the test is relatively worthless. How to remedy that? I encourage creating two tests wherever possible, one positive and one negative. Both should pass if coded accurately but both approach the problem from opposite levels. If you want to be more aggressive, make some more negative tests to really push and see if we are doing the right things. This is especially valuable if you have put time into error-handling code. The more error-handling code we have, the more negative tests we need to create to make sure our ducks are in a row.

A final item Melissa mentions is the fact that we often rely on the experts too much. We should be looking at the option that the expert may not be there (And at some point if they genuinely leave, they WON'T be there to take care of it. Code gets stale rapidly if knowledgeable people are lost. Take the time to include as many people as possible in the chain (within reason) so that everyone who wants to and can is able to check out builds, run them, test them, and deploy them.

Common Pitfalls/Cognitive Biases In Modern QA with Leandro Melendez (PNSQC)

Ah yes, another date with the legendary Señor Performo :). 

Leandro is always fun to hear present and I particularly liked the premise of his talk as I frequently find myself dealing with cognitive biases, both in the way of locating them when others use them but also to admonish myself when I do (and yes, I do fall prey to them from time to time). 



I've been in the process of teaching a class for the past few months related to software test automation, specifically learning about how to use a tool like Playwright with an automated testing framework. To that end, we have a capstone project that runs for three weeks. As anyone involved in software development knows, three weeks is both a lot of time and no time at all. This is by design, as there is no way to do everything that's needed, and because of that, there are things that we need to focus on that will force us to make decisions that will not be optimal. This fits into the conversation that Leandro is having today. How do you improve and get better when you have so many pressures and so little time to do it all? 

Note: I am not trying to throw shade at my students. I think they are doing a great job, especially in the limited time frame that they have (again, by design) and seeing what choices they make (as I'm literally a "disinterested shareholder" in this project, meaning I care about the end product but I'm trying my level best to not get involved or direct them as to what to do. In part, it's not the instructor's role to do that but also, I'm curious to see the what and the why concerning the choices that are made).

We often act irrationally under pressure and with time limitations. Often we are willing to settle for what works versus what is most important or helpful. I'm certainly guilty of that from time to time. An interesting aspect of this, and one I have seen, is the "man with a hammer" syndrome, where once we have something we feel works well, we start duplicating and working with it because we know we can have great wins with that. That's all well and good but at times, we can go overboard. Imagine that you have an application with navigation components. You may find that many of those components use similar elements, and with that, you can create a solution that will cover most of your navigation challenges. The good thing? We have comprehensive navigation coverage. The disadvantage? all of that work on Navigation, while important and necessary, has limited the work on other functionalities with the unit under test. Thus, it may be a better use of time to do some of the navigation aspects and get some coverage on other aspects of the application rather than have a comprehensive testing solution that covers every navigation parameter and little else to show for it. 

Another example that Leandro gives is "Living Among Wolves" or we can consider this an example of "conformance bias" meaning that when we do certain things or we are part of a particular environment, we take on the thinking of those people to fit in with the group. Sometimes this is explicit, sometimes it is implicit, and sometimes we are as surprised as anyone else that we are doing something we are not even aware of. 

The "sunk cost" appears in a lot of places. Often we will be so enamored with the fact that we have something working that we will keep running and working with that example as long as we can. We've already invested in it. We've put time into it, so it must be important. Is it? Or are we giving it an outsized importance merely because we've invested a lot of time into it? 

One of the lessons I learned some time back is that any test that never fails is probably not very well designed or it offers little value in the long run. It's often a good idea to approach tests both from a positive and a negative perspective. It's one thing to get lucky and get something that works in a positive/happy path test (or not necessarily lucky but limited in what's being done. Now, does your logic hold up when you inver the testing idea? Meaning can you create a negative test or multiple negative tests that will "fail" based on changing or adding bogus data or multiple bogus data examples. Better yet, are you doing effective error handling with your bogus data? The point is, that so many of our tests are balanced to only happy path, limited depth tests. If you have a lot of positive tests and you don't have many tests that handle negative aspects (so that the incorrect outcome is expected... and therefore makes a test "pass" instead of fail), can you really say you have tested the environment effectively?   

Ending with a Shameless plug for Leandro. Leandro is now an author, having written "The Hitchhikers Guide To Load Testing Projects", a fun walkthrough that will guide you through the phases or levels of an IT load testing project. https://amzn.to/37wqpyx

Amplifying Agile Quality Practices with Selena Delesie (PNSQC)

I had the opportunity and privilege to introduce Selena Delesie on the program today. It was fun to reminisce a bit because Selena and I were both in the same Foundations class for Black Box Software Testing all the way back in 2010. We also both served on the Board of Directors for AST, so we had a lot of memories and fun/challenging things to deal with over the years. Thus, it was a pleasure to introduce Selena as our first Keynote speaker. Also, part of her talk was discussed on a recent The Testing Show podcast, so if you want a sample, you can listen to that :).

The tool that Selena and Janet Gregory put together is called the Quality Practices Assessment Model (QPAM). The idea behind this is that there are ways to identify potential breakdowns in the quality of our work. Areas we should consider are:

• Feedback Loops
• Culture
• Learning and Improvement
• Development Approach
• Quality and Test Ownership
• Testing Breadth
• Code Quality and Technical Debt
• Test Automation and Tools
• Deployment Pipeline
• Defect Management

The fascinating thing is the order and how these are identified and examined. Selena makes the case that the order in which these are presented and examined is important and by examining them in this order or weighting, the best chances for overall and lasting improvement are possible. Yes, Defect management is important but it will be less effective if more weight is not given to the previously mentioned items.

A key aspect to this is that quality is not just a technical issue, it's also a social issue and it should not be dealt with in isolation. Selena introduces us to a group code-named "Team Venus" and identifies many of the issues they are facing and where those issues fall on the ten quality aspects. The key element is that each area is looked at holistically and in conjunction with the other areas, not in isolation. As anyone familiar with GeneralSystems Thinking can tell you, there is no such thing as a standalone and isolated change, any modifications made will have ripple effect. It's also critical to realize that a process alone is meaningless if the overall values are not solid or agreed upon. 

In the ten quality aspects that Selena referenced, there are four quadrants/dimensions to consider:

• Beginning
• Unifying
• Practicing
• Innovating

What I like about considering these as quadrants is the fact that these areas are not separate from each other but they are dependent on each other. Some areas of the ten practices will be closer aligned with a particular dimension. Additionally, it's common for teams to spend more time in a given quadrant/dimension for the ten areas than others. I like the diagram Selena uses that looks like a spider web. The center of the web means that that is an informational or foundational level, and the farther out from the center, the greater the expertise and experience. Ideally, of course, all of the aspects should be on the outer rim of the spider web but in practice, there will be color splotches in all four of the dimensions. That is normal and should not be discouraged, especially since each new team member will typically need to start from zero.

For those interested, the book and full model example for Team Venus is available in "Assessing Agile Quality Practices with QPAM" so if you want to learn more, go there.

Amping It Up!: Back at the Pacific Northwest Software Quality Conference

 

It's that time of year once again. I'm excited to be at PNSQC and in a new location. We are at the University Place Hotel and Conference Center, which is part of Portland State University. This is the first time PNSQC has been here but not the first time I've been here. A few years back, in 2018, they had run out of rooms at the primary hotel, so I had to find someplace else to stay. 

I happened upon University Place, and while it was a walk from the previous venue at the Portland World Trade Center, it was a comfortable hotel, and I liked my stay. As we were looking for different places to hold the conference this year I mentioned my experience and thought it would make for an excellent possible venue. And thus, here we are. It wasn't solely up to me but I definitely put in a good word ;).

This year's conference is a strange feeling for me, as it is one in which I am feeling unsure and unsteady after many years. I am at the tail end of a contract I've been working for several months. In a few weeks, barring any changes or new contracts, I will be out of work again. Thus I am attending this conference from a different headspace than usual. Previously, I was looking for small tips I could bring back to do my current job. This year, part of me feels the need for a literal reinvention. I'm having that uneasy feeling that I have too many potential options and not enough time to consider them all, so this year's talks are probably going to be focused on my current mental state. If you see me attending talks that may seem different or out of character, that's why.

Additionally, this year had an additional challenge and excitement in that I was the Marketing Chair for PNSQC this year. If you felt that there was either too much or not enough of a marketing presence for the conference, I get both the praise and/or the blame. Either way, for those who are here and for those following along, I'm happy you are here.

   

Low-level Approaches for Testing AI/ML: an #InflectraCON2013 Live Blog

 One of the great parts of conferences like this is that I meet people I have interacted with for years. Jeroen is one of those people. We worked together on the book "How to Reduce the Cost of Software Testing" back in 2010 but we have never met in person before this week. We've had some great conversations here and now I finally see him present.

Jeroen Rosink

Sr. Test consultant, Squerist

I think it's safe to say everyone has been hit with some form of AI and ML in some capacity. If you need an explanation of AI and Machine learning, I'll let Chat GPT tell you ;).

AI, or Artificial Intelligence, refers to the development of computer systems that can perform tasks that would typically require human intelligence. These tasks might include things like recognizing speech or images, understanding natural language, making decisions, and solving problems. AI can be classified into various categories such as supervised learning, unsupervised learning, reinforcement learning, and deep learning.

Machine learning is a subset of AI that focuses on teaching computers how to learn from data without being explicitly programmed. In other words, it's a method of training algorithms to make predictions or decisions based on patterns in data. Machine learning algorithms can be trained on a variety of data types, including structured data (like spreadsheets) and unstructured data (like text or images). The most commonly used machine learning algorithms are supervised and unsupervised learning algorithms.

I mean, that's not bad, I'll take it. So I used AI to explain AI. What Inception level is this ;).

AI is always learning and it has been trained on large data sets. I often look at AI as a good research assistant. It can do some pretty good first-level drafting but it may miss out on some of the nuances and it may also not be completely up to date with the information it provides. Also, Machine Learning really comes down to ranking agents and probability. The more successes it establishes, the higher it ranks certain responses. To be clear, even with how rad AI and ML seem to be, we are still in the early days of it. We can have all sorts of debates as to how much AI will take over our work lives and make us obsolete. Personally, I don't think we are anywhere near that level but I'd be a fool to not pay attention to its advances. Therefore, we need to consider not just how we are going to deal with these things but how we are going to test them going forward.

 Jeroen talks about the confusion matrix and how that is used to test ML.

The confusion matrix is used to evaluate machine learning models, particularly in classification tasks. Think of it as a table with a number of correct and incorrect predictions made by a model for each class in a set of data.

The four possible outcomes are:
- true positives (TP)
- false positives (FP)
- true negatives (TN)
- false negatives (FN).

A true positive occurs when the model correctly predicts a positive instance.
A false positive occurs when the model incorrectly predicts a positive instance.
A true negative occurs when the model correctly predicts a negative instance.
A false negative occurs when the model incorrectly predicts a negative instance.

Jeroen has two approaches that he is recommending:

The Auditor's Approach

First, we perform a walkthrough so that we can see if the data is reliable and useful. From there, we do a Management Test to use data in enough volume to see if the data as presented works with small and larger numbers. If we can see that the data is relevant with one, and with 25, then we can see if it's relevant with 50 or 100, or 1000 and so on. We can't predict the output but we can have some suppositions as to what they might do.

The Blackhole Approach

This is an interesting approach in which we don't necessarily know what the data is or what we would actually have as data. We can't describe what is actually inside the black hole but we can describe what surrounds or is visible around the black hole. In this capacity, we look for patterns and anomalies that don't correspond with our expectations. If we see a pattern that doesn't match what we expect, we may have an issue or something that we should investigate but we are not 100% sure of that fact. Jeroen explained that there's a technique that can be used in the classic illustrations for "Where's Waldo?" The idea is that with a pen and making some marks on the page, we can figure out where Waldo is in about ten passes. To be clear, the system doesn't know where Waldo is, but it examines patterns in the image and breaks down the patterns to figure out where the item it is looking for might be.

These are neat ideas and frankly, I would not have considered these prior to today but be sure I'm going to think a lot more about these going forward :).

Technical Debt Is Not Free: an #InflectraCON2013 Live Blog

 

Chad Green

Director of Architecture, Glennis Solutions

When you hear the term "Technical Debt", what does that mean to you? Often the term "a quick and dirty fix" is used with the idea that it will be taken care of later. In short, anything you have to revisit later because of the limitations of today is specifically technical debt. 

The fact is, many of us have probably participated in the process of developing technical debt, whether we intended it or not. Even mature teams find themselves in technical debt, sometimes by active means, and sometimes by inaction. I remember working with an organization that had a great automation framework and it was very robust, with a lot of code libraries to support it. It was a great system, as long as the developers that created it were there to maintain and support it. However, we came to a point where our developers that did support this were no longer there and the team members that were there did not have the level of expertise necessary to maintain it. What was once a vital linchpin of our efforts became outdated and in some ways dangerous to do any work on. We went from having a reliable system to a need to modernize. In short, we woke up with a technical debt through no intentional effort on our part but we had to address it. We ultimately did but it took time and effort and a lot of iteration. We learned from that experience that we needed to make sure that what we developed had a broad base of support so that any of us could work on and maintain it.

Going into debt is not a crime or even a bad situation by itself. It can certainly become a bad situation if it's not addressed or worse, ignored. In many ways, we have to be more careful and make sure we are doing things in ways that are maintainable and understood. I went through this recently with some changes I proposed to a system that had a different way of handling API data (this came from suggestions of one of our devs). When I submitted it, the comments back were, "This is interesting and a way that we hadn't considered. We're not saying "no" but we may want to make sure we understand why we'd want to do it this way. Can we revisit this in the next sprint?" That's a perfectly reasonable request. Let's understand what making this change might be and how it might modify our testing methodology.

Technical debt sounds scary but there are ways to limit it or mitigate it. Take the time to do actual code reviews and make sure that the changes being made are understood. Talk about these things in Stand Up if necessary but surface these issues, as well as discuss these in your sprint reviews. Much of the time technical debt will be hiding in plain sight. The best way to deal with technical debt is to look for it and identify it as early as possible.