PNSQC 2026: Inside Tesla: The Birth of the Software-Defined Vehicle and Quality in the Age of Autonomy

 Well now... that was quick!

We are already at the end of the 2-day conference. Quite a whirlwind experience, and it's neat to see that the last keynote was the first of the PNSQChronicles we did this year.

See the teaser PNSQChronicles video with Florian Rohde

Florian is talking about the idea of the software-defined car, a topic that we have come to see with both interest and annoyance, but that effectively came to be a thing with the advent of Tesla. So what is a software-defined vehicle? It's the idea that, instead of having a vehicle that is limited mechanically, the vehicle is effectively created one time, one version, and the owner/user can choose what capabilities the vehicle provides with how much they have paid and what features they have paid for. In other words, everyone gets a fully loaded car, but how much of the fully loaded car you actually access is up to you and your payment level.

There are a bunch of software components and microcontroller elements inside any Tesla, so what you choose to get and how your vehicle communicates with the broader world can be effectively controlled and optimized for a variety of users. One platform, all sorts of potential configurations and capabilities.

I remember talking about this with Florian in the interview we did and suggested that there was potentially a bit of an annoyance in having to buy/rent features of a vehicle I already own. Florian points out that, in his estimation, this is actually a beneficial arrangement. I remember buying back when I was a starving musician back in 1990 a Ford Escort Pony. This was the most bare-bones entry-level vehicle imaginable. I bought it with no AC, no power steering, no power brakes, manual transmission, crank windows; it didn't even come with a radio (I put that in myself shortly thereafter). I bought this vehicle because it was literally all I could afford at the time (we're talking a vehicle that cost ~$7000 brand new). It was a surprisingly durable vehicle. One of the neat things about an all-manual vehicle is that there's so little on it that could break. I put 168,000 miles on this vehicle in about 6 years, and then I gave it to my brother, who put another 100,000 miles on it. My point is, this was an insanely good value for a vehicle, but I was locked into the choices I could have afforded in 1990. I didn't really mind that, but there were indeed times when I wish it could have done/been more. So what if I could buy that same $7,000 car, but later on, as I got further along in my career, I had an easy way to enable things like power steering, power brakes, turn on the AC, etc. How would I have reacted? Probably favorably, as long as I felt it was a reasonable trade-off for the functionality offered. 

In any event, I want to thqank FLorian for describing this approach that they put together. I'm not a Tesla owner, but as we discussed the details of this talk, I moved over into the "adopt as you need" category. How cool would it be to be able to buy a vehicle that I may only need 2 WD for, but later on I might decide, "Hey, I think I'd want to have 4WD on this vehicle." A very expensive and time-consuming process with any other vehicle, but how cool would it be to be able to just "turn it on?" This is a fast overview of the details of Florian's talk, and I have to say I found it interesting and refreshing to hear about an industry I have zero career involvement in. Am I entirely on board with the "pay-as-you-go" approach of an already loaded vehicle? Not sure, but it makes a lot more sense now :).

PNSQC 2026: "Testing Without Boundaries in the AI Era" w/ Tariq King

Yay! A Tariq talk.

For those not familiar with Tariq King, he's a great speaker, a great innovator, and a fun person to hear speak. The last time I saw Tariq on stage was back in 2024, when he was talking about the capability of generative AI and what it could do at the time. HE was discussing some of the Turing test details regarding human faces and artwork that were generated by AI and what we could tell vs. what we couldn't.

It's been a couple of years since then, and we have now seen some of the outcomes from these Generative AI tools, as well as the blowback and retrenchment of people's opinions on these technologies. It's made an interesting conundrum for me. How do I reconcile the work that I do that may require AI to even be viable vs. the creative industries I am in where Generative AI is absolutely seen as a bridge too far and that, by using Generative AI in the creative spaces of art and in music, you are seen as a "clown show." We weren't there two years ago, and now we are having different conversations than we were then.

This opens up an interesting element of the Generative AI puzzle, even beyond what we are doing, and now discussing when we should or should not use it. We can see that on Spotify, music that has been generated by AI tools is being delisted and forced takedowns applied. I know that's not something that Tariq is necessarily going to address here, but it may be interesting to discuss. If anyone is interested in my take on that, maybe that's a series I can do here.

In my workday world, using AI or being considered for employment requiring my use of AI is not a matter of nuance or delicacy. If I can't demonstrate I know how to use it in both semantic ways as well as autonomous agents working for me, I'm not going to be employed. So in some cases, I may not be able to take some stance and say "AI for thee but not for me". More to the point, AI has given me a desire to be more focused on testing it and trying to figure out specifically what AI is doing, so I can be part of making sure that AI doesn't do the wrong stuff.  

I've been having this conversation with a number of people that I am deeply distrustful of any application I can't reasonably test. In short, I'm not "all in" on AI. I had a funny conversation where I compared AI to the early days of MIDI and how people were hugely dismissive and antagonistic towards using MIDI to make music, especially when it came to sequencing in real time in performances. We don't have those conversations any longer because those MIDI pioneers not only showed how challenging it was but also demonstrated through their creative output that they could actually write music and be inventive in their creations. Will Generative AI get to that point in the music space? On one hand, the music that is being created is polished; it's entertaining, and it's well produced. What it isn't is unique or interesting.  It's a sum total of everything that has come before. Same progressions, same melodies, same structures. And of course it would be. It's been trained on the world of music that's out there. So it can recreate all of that. But what it's not good at doing is breaking the paradigms. It's not good at creating something from the heart. Something human. Something organic. Granted, some people may not care a whole lot about any of that. However, for many of us, to quote the late great Chuck Mosley... "We Care a Lot!"

Back to Tariq's direct points, what do we need to be doing to both test these technologies and determine the future we want to make? One of the fascinating things I came to the conclusion of many years ago was: if I found I could live the rest of my life without working, what would I do? I'll not kid anyone; I'd devote myself and my time to Ensign Red, my band (which, I will be fully transparent and honest, has been how I've been spending the majority of my time since being laid off from my last job ;) ). I also said I'd love to get into snowboard R&D, as at the time I last had that thought experiment, I was a competitive snowboarder (retired since 2005, but it was a great ten years :) ). The third thing I'd want to do is advocate for software quality in any way I could, even if I never got paid another penny to do it. That may be a weirdo thing about me, but it is interesting to think about. So far, I have spent a tremendous amount of time still focusing on software quality even when I'm not actively being paid to do it. The fact is, I love testing, and a part of me has always hoped I'd be able to test forever, or until I shuffled off this mortal coil. Well, I have zero control over who will choose to hire me, but I can advocate anyway, and advocate I will. We are dealing with models and tools that make mistakes, and often pretty fundamental mistakes. 

I like Tariq's thoughts that software testers haven't been taken out of the equation, but that our roles and responsibilities are shifting. We are less needed for our actual physical testing acumen. We are more needed for our judgment and clarity of purpose. To borrow from my comments earlier, MIDI itself doesn't write anything that is new or innovative. That's coming from the mind of the musician. I may have tools that can structure chords and chord progressions, but it's still up to me to know my modes, where I can use them, and how to know when they actually sound good. 

I'm reminded of a comment a dear friend of mine taught me back in the early 1990s, at the start of my testing career (and Chuck, if you by chance are reading this, "hi" and "thank you for everything")... "Every tester, every programmer, every person who works in tech needs to know when they can afford their attitude." That is a statement that has stuck with me over the years. Sometimes we are very valuable, and sometimes the things we think are valuable can be devalued overnight. If we cling too tightly to the technical skills we have,  those can be rendered obsolete quickly. The desire to learn, the desire to explore, and the ability to use those skills to pivot to new realities- that's the valuable skill, and that is the honest, real area where we can see if we can really afford our attitude. 

PNSQC 2026: "Communication Challenges in the Intersection of Diversity, Agility, and Leadership"

I had a lot of fun talking with Dr. Gwen Avington and Diana Larwsen for our PNSQChronicles interview for this talk, so while I can't be in person to experience it, I can at least appreciate a lot of what they are talking about.

PNSQChronicles interview with Dr. Gwen Avington and Diana Larsen

In the talk today, we were presented with an image for a brief period of time and then asked to say what we saw in the picture. As can be guessed, the image was not clear, and there was a lot of interpretation as to what was in the image (as a remote viewer, I had an even fuzzier view of the image, so it was interesting to hear what people in the room saw.

Quality is one of those areas that we talk a mean game about, and we know we should care about it. However, oftentimes our first quality hits (as in the bad kind, not the sensation kind) tend to come about because of poor communication. We don't have a clear understanding of what we should be doing. It doesn't matter who the person in the room is, or what the responsibility is. If we don't have a clear idea of what the problem is that needs to be solved or addressed, we are already at a disadvantage. Communication can absolutely help to mitigate risks and help avoid costs later on because we didn't discuss or understand the issues.

If we have conversations about quality or lack thereof later in the process, we will have to find ways to shoehorn those capabilities or quality measures in at a point when, had we done it earlier, we could have avoided so many issues. This is definitely an issue for newer people. We might feel like we don't have the status or the understanding necessary to speak up in early stages. We often mistake longevity for expertise and knowledge. We may have questions, but we fear being seen as dumb or lacking. Perhaps it behooves us to say, "Hey, I may be new and lacking information, but have we considered 'X' here?" I've had those experiences, and I've learned that it is valuable to have that attitude. Just because someone has experience or longevity doesn't mean they necessarily know if they have considered areas. Even if they have, they may not know that they are not communicating it well.

So how does diversity work into this? Sadly, a lot of the time, there's a set idea of who has expertise and understanding. We make mistakes all the time in this capacity. Everyone brings something of value to the table, and sometimes the best insights may come from people who may not be as obvious. Sexism, racism, classism, and other "isms" can get in the way of who we think may be able to offer insights of value.

We always get down to the question of what we perceive quality to be. One person will have some different ideas compared to another person. We are all used to the idea that "Quality is something of value to some person(s) who matter". That's a part that we often struggle with. We think that we have an idea of what quality is, but we might not be thinking of the areas that are actually important to the people who are most concerned with it. We also need to know when to reinforce and encourage people to show that their input and observations are both important and valued. No one wants to speak up if they think they will be shot down. It takes courage to open up and communicate. Additionally, help foster an environment where they can step up to speak and know that we have the space to speak in the first place.

 

PNSQC 2026: "With Great Judgment Comes Great Great Responsibility" with Paul Grizzaffi

Good morning again. This is one of the reasons I am sad to not be in person at PNSQC. Paul and I go way back, and I always have fun when I hear him speak. We frequently bond over our love of music, especially heavy metal, so Paul, just know that I'd be bending your ear if I were there ;).

How many of us know what a Pyrrhic Victory is? Pyrrhus of Epirus is famous for winning battles with absolutely dreadful losses, so much so that the term "Pyrrhic victory" means a win that comes at such a great cost it's arguable if it was actually worth it.

PNSQChronicles Podcast previewing the details of Paul's talk

AS testers, we are often asked to "do the right thing" (a Spike Lee joint ;) ). For the few of you out there who chuckled, you're my people... I love you :). Automation is something that has a specific purpose. We can argue all day long if that purpose is actually needed or necessary, but for many of us, our jobs would be rendered very difficult without it. We all do some form of automation if we have done something enough times. That's often the first wave of automation, the "Oh my stars! If I have to do this process one more time by hand, I'm going to lose my mind." I'd argue that the people who are least fond of these steps are probably the ones who will embrace and get good at automation the quickest.

In this current age of AI, we have a fresh new danger. AI can write all of the tests and even the test code for us. In truth, it can. However, if we think that we are going to get "one and done" tests that will be resilient, durable, and effective over time, there may be some surprises in your future. In my last gig, I had the experience of doing both roll-your-own automation and working with AI to write the test code and run it. As I was working through those processes, I came to a simple conclusion. The time to have a vetted and confirmed version of automated tests that AI generated vs. tests I created myself... was roughly equal. That's not to say that I'm some amazing automator (I mean, I know some stuff ;) ) and no question, AI can crank out code so much faster than I can. Still, when I question the code and ask why it's doing what it does, the rounds of revisions to get to a truly working and robust version of tests, at the end of the day, were roughly even.

For those who know how I do these, I'm writing this as I'm listening to Paul, but I'm not going to regurgitate his talk. I am going to consider what he is saying and apply it to my own experiences and see what rings true and what I might learn something fresh from. There are absolutely costs associated with using AI, even if it feels like it is cranking stuff out that feels fast, fresh, and amazing. One of the projects I worked on was to take our Documentation as published and make an analysis gap of what tests we had and what options our documentation was saying we did. While I will not claim that's the be-all and end-all of what should be automated, if you want to do an interesting exercise, I recommend doing that process. You can do it all on your own, or you can run it through AI, but either way, it gives you a simple place to start. Do we stop there? Of course not, but at the minimum, you can say that you are making tests that at least cover the areas that you would want to encourage your customers to be doing. Start with the happy paths and then look for the areas that could trip you up. Look for the error code that exists, and then try to create tests that surface all of that error code effectively. Again, whether you do that as a roll-your-own project or you turn over the considerations to Claude Code, OpenAI Codex, or some other AI methodology, ultimately, we want to know what's happening and why.

As a musician, I have many tools that are at my disposal. I use Digital Audio Workstations to do actual recording; I use multiple amp sims and effects that are built into plugins, and I have lots of virtual instruments (even singing voices) so that I can experiment with various things. Some people might remember the early 80s and the advent of MIDI and sequencing software. I remember well the antipathy people had toward MIDI back then, and the feeling that "people were pretending to be musicians" by using MIDI and sequencers. I'd argue that any performer who chose to bring MIDI onstage in those days and orchestrate all of that in a live performance was a braver soul than I. It was easier, in my mind, to just perform than try to wrangle all of that stuff in real time. I remember the snide remarks coming from the metal community especially. The irony is that, a couple of decades later, you would be hard-pressed to hear a metal album produced today that isn't using all of the software recording features that exist today. The beat alignment to the grid and the level of precision that is orchestrated at the point of recording is absurd now. Is it easier than performing it? It might well be. Is there still a performance to be heard there? That's another question entirely. If we are going through and we are massaging every guitar part with quantizing, aligning every beat to the grid, and using Melodyne to clean up every vocal note's tuning, what we end up with may be a pristine recording, an elaborate artifact, but we can't call it a "performance".

To borrow from Jurassic Park, sometimes we get so caught up with what we can do that we might lose sight of what we should do. Knowing when those two options align, and when they don't, is the big thing. I wish I could be there in person to riff with Paul about this, but I hope he enjoys this rambling missive in the meantime :).

PNSQC 2026: The AI Assurance Imperative: Quality Engineering for the Agentic Era w/ Jonathon Wright

Hello everyone, and for those at PNSQC this year, I hope you are having a wonderful time. I, sadly, am not in Portland for the first time in many years, and I am genuinely sad about that. There are reasons I can't be there this year, and I hope that I can participate next time, but for now I am happy and grateful that virtual PNSQC is an option.

Many of you may have seen my PNSQChronicles interviews over the past few months where I've had a chance to talk wth many of the presenters, and Jonathon was a fun interview for me, so I think it's a good bet that you all will enjoy his talk, too.

Jonathon just made a point about the latest series about AI that Netflix has been airing, and pointed out that Sam Altman has said that the most important thing that these companies could be doing is testing. 

At the same time, we are dealing with AI agents that operate independently, generate code, run tests, make choices, and develop systems without human involvement. So how do we actually test stuff like this?

Self driving card, next-gen technology, and everything we are looking at today is aiming for stuff that is wildly different than what we are used to. We are talking about new sciences, ways of using technology that we have not even considered, and many of us are still testing things for the paradigms that currently exist. Nothing wrong with that, but the most value we can add is to actually consider how to test for what comes next.

The Human in the Loop is a common phrase, and something we should all make sure that we advocate for. We cannot let these systems run without some level of human judgment making sense of what's coming out of it. We have the phrase "Agentic" now that we need to embrace. It's a fad; it's an experiment. We've all heard that, right? That's likely not going to be true. We have reached a point where so much stuff is coming out now that there is simply no way that we can keep up with it, much less test it effectively.

So while we have Sam Altman saying the most important thing is testing, why does it feel like actual testing is not being included in this conversation, or if it is, it feels very peripheral? We are currently looking at a situation where seasoned engineers are struggling to find work (raises hand: I'm one of those people at present). Don't get me wrong, I haven't given up hope here, but I am certainly curious as to how we are going to address these situations if literally seasoned programmers and engineers are being pushed out at this stage. So if testing is going to be needed in this Agentic AI age, who's going to do it if we are actively removing people? This seems to be an interesting conundrum. Either AI systems are going to have to learn how to effectively test on their own, or people are going to have to help bridge that gap. Again, it's an interesting question: how much are we willing to give of ourselves to help make AI better able to test itself, and perhaps risk losing our place in being able to be part of that conversation in the first place?

The intriguing part of all of this is wrapping our heads around ways that we as people could be effective in this new world. What methods will we use? What approaches are we going to pioneer that we have not yet had the chance to get a feel for? How much of what we are going to need to test do people actually even understand at this point? I don't say that lightly. To borrow from a favorite quote: "The significant problems of today will not be fixed by the same logic and reasoning that created them." (I'll have to go back and give that proper attribution and wording, but that's how I remember it right now). Jonathon is talking about the idea that we can literally load multiple testers' experiences and approaches into a database and effectively have those testers working for him (including me, if they decided my corpus of test writing is worth anything ;) ). That's an intriguing option to think I could have Angie Jones, Randy Rice, or Matt Heusser at my immediate beck and call because I could create agents that would test based on their writing and principles that they have published to apply to my own tasks. 

This is a wild new world we are playing in now. I don't have a crystal ball, and I can't say what will be happening next year, much less five years. The key Jonathon is trying to point out is that we don't have five years to absorb this. This is a reality we are going to have to embrace really quickly, and we ignore it at our peril. It's proving to be an expensive option for many, but don't assume that means that it's going to just fall apart. AI is here, enmeshed, and not going away. Testing it is going to be critical to it not running off the rails. I'm glad to see some people are actually considering the testing needs and realizing the elephant in the room. How we go about doing it is anyone's guess.

Breaking the Cycle: Leading Organizations That Thrive Amid Constant Change with Allan Rennebo Jepsen (a PNSQC Live BLog)

Well, that was fast! We have reached the end of the conference with our final keynote. There has been a lot to absorb and learn over the past few days, and I've enjoyed my time here. With that, here's the closing talk for PNSQC 2025.

Allan Rennebo Jepsen is a leadership expert and author based in Copenhagen, Denmark, who has spent over two decades helping organizations adapt to change rather than fighting it. Allan opened up with an example of an online banking initiative that went horribly wrong, where everything that could go wrong did, to the point where the CEO at a corporate meeting said that a large majority of customers of the bank would not be around a year from now. Meaning their business model was cratering and they risked going out of business. Change is needed, but how in the world are they going to do it? Large banks are optimized for compliance, and because of that, they are literally designed to preserve the status quo. 

PNSQChronicles: Leading Organizations That Thrive Amid Constant Change with Allan Rennebo Jepsen

So what ends up happening? “We are trapped in a loop of meetings, updates, and alignment sessions, with initiatives and plans made, but somehow, progress keeps slipping away.” 

This ongoing “cycle of coordination” eats up energy and time while creating an illusion of progress. Wouldn't it be nice to be able to break the cycle and replace it with focused, adaptive, and empowered teams with goals and efforts that can actually move the organization forward?

As Allen points out, the one true constant is change, and change is not slowing down. New technologies, shifting customer expectations, and global competition mean that businesses must evolve faster than ever before. “Success today requires a new paradigm, one that values bold leadership over small tweaks.”

Many organizations instead fall into reactive patterns and mindsets. They organize based on what the business thinks is important, rather than looking at what the actual customers need. Every new change triggers a cascade of alignment meetings, but before that alignment solidifies, another change hits. The result is paralysis through coordination.

What will it take to break out of this vicious cycle? Genuine adaptability, that's what. Organizations that show that they can learn, adjust, and act decisively in uncertain conditions are the ones that will ultimately thrive.

What do we need to do to actually make change happen?

Strategic Prioritization

We need to focus our energy where it matters most. Not every initiative deserves attention. Leaders need to be ruthless in prioritizing efforts that drive real impact and cut through the noise of constant change.

Winning Together

Too many departments chase local wins at the expense of the organization’s overall success. Instead, focus on organizational collaboration and allow for genuine long-term thinking. Optimizing one team while degrading the system as a whole is counterproductive.

Thriving Amid Change

When teams are empowered to make decisions and take action without waiting for endless approvals, they can develop true adaptability. With those experiences, the team can get better and address challenges related to change. Success likelihood grows when teams have both the authority and the accountability to act quickly. Resilience and adaptability are learned competencies, not traits reserved for a few innovative companies. If you want to be resilient and adaptive, you need to practice and work towards being resilient and adaptive.

Allan encourages creating a “low-risk, high-impact” framework for building organizations that are genuinely adaptable. Continuous learning loops, clear ownership, and a culture of shared goals help with and reinforce the opportunities to effectively adapt to change.

As Allen points out, “These organizations didn’t get there by accident. They created the conditions where adaptability could flourish.”

Artificial intelligence, decentralized decision-making, and global competition are reshaping how organizations operate. Instead of fearing disruption, leaders should view it as a constant condition to master. Focus on adaptability as the ultimate organizational competency.

“You can’t predict the next disruption, but you can prepare your organization to handle whatever comes next.”

Key Takeaways

- Prioritize what truly matters. Avoid being spread thin across too many initiatives.

- Empower your teams. Give them the clarity and authority to act.

- Collaborate across silos. Success comes from shared goals, not isolated wins.

- Embrace continuous learning. Every challenge is a chance to adapt and improve.

If there's any one message or idea to bring home today, it is that change doesn’t have to be chaotic. Leaders who build adaptable, empowered teams will be the ones who turn constant change into a competitive advantage.

Want to know more? Allan’s book “Impactful Organizations – And How to Become One!" expands on many of the ideas shared today. 

The Ethics of Quality with Gwen Iarussi (a PNSQC Live Blog)

I'm back. I spent the morning moderating the Management, Leadership & People track, so I wasn't able to do that and live blog like I normaly do but let's suffice it to say that Sophia McKeever (Quality With Hearts Aligned - Bolstering Your Quality with Emotional Intelligence), Heather Wilcox (Is the 'Iron Triangle' Dead?) and Kristine O'Connor (Quality Intelligence: Embedding Customer Voice to Drive Agile Excellence) all did great jobs and if you get a chance to see their talks, do so :).

For the first session this afternoon, I get to see long time friend (and actual hirer of my consulting services during my stint in the wilderness) Gwen Iarussi discuss The Ethics of Quality

PNSQChronicles: The Ethics of Quality & Quality as a Living System with Gwen Iarussi

A great quote to start out this conversation is, "Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should." (Dr. Ian Malcolm, Jurassic Park).  You may well remember this line and it is just as valid a warning about unchecked innovation. It's not enough to ensure a product "works". Additionally, we have a continued need to advocate for not just that something works, but that it works appropriately. We don't have the luxury of testing blindly and ignoring the possible impacts of our work and efforts. 

I remember back in the days of my Cisco Systems work life that the NetFlow protocol tracking software was developed. I was on the team that was ultimately codifying the "call record" or any and all internet data. I remember part of me was a little leery about what we were doing but I recall being told by one of the developers that, "if we don't develop this with our approach and methodology and ethics, then our competition will create it, and we will have no control over the ethics that they choose (or do not choose) to use."We can argue whether or not that was a good idea (we do of course have end to end tracking of all network data but we also have the ability to encrypt that data as well, so yin/yang, I guess ;) ).

My point with that last example may be seen as trivial today but part of me genuinely wondered if our efforts were being done in the appropriate place. At that time, I was young and unsure of my voice, so I just rolled with it. Today me would be less likely to do that, I'd want to ask a number of questions. Granted, I may not have been able to have modified or altered any of the outcomes, but I certainly would have been more direct and focused with my interactions and questions. I would have advocated for positions I felt we should represent.

Our tech environment today is getting even more fragmented and over the top. Speed of execution is taking precedence over the quality and ethics of what we are creating. We have all sorts of areas where the haves are taking advantage of the have nots through these systems, and it's very much by design. Let's not forget the fact that this age of AI and fast, ubiquitous tech is consuming land, fuel and water resources at an alarming rate. Much of the tech advances and buildout to host and serve it is not sustainable in the long term. At least not if we want to make sure that people have access to water, fuel, and food a few years from now.

Much of this comes down to accountability. How do we develop accountability and stand for what we want to see happen in our environments? How do we make our ethics more than just platitudes? By us building a culture of accountability, we are able to foster a culture of ethics and making considerations that go beyond merely speed and efficiency.

Ultimately, this all comes down to a quote that is attributed to Sitting Bull, "Let us put our minds together and see what life we can make for our children". Ultimately, every good or bad decision we make may have an effect on us, but they will surely and certainly effect future generations. With this isn mind, I would absolutely say it would be in our best interests, including business interests, to be able to leave to our children a world and systems that have effective safeguards to prevent our actively destroying our environment, whatever and wherever those may be.

Monsters & Magicians: Testing the Illusions of Generative AI with Ben Simo (a PNSQC Live Blog)

Day 2 of the main program is underway (Day 3 includes the workshop presentations). It's crazy to see/feel how quickly this event goes by. As always, I've had a great time at this event and enjoyed my interactions with everyone. It's especially neat to realize how many people I know in this space, and when keynote speakers are literal friends, such as today with Ben Simo. 

Machines to do things beyond our physical or mental powers have existed for thousands of years. We can go back to the Antikytheria Mechanism for what may quite possibly be the world's first "artificial intelligence," depending on how you want to interpret that term. Over time, as we have come to grips with and developed an understanding of the rules, laws, and repeatability of activities, what was magical once upon a time has become commonplace in our everyday use. 

PNSQChronicles: Brief Interview of "Monsters and Magicians" with Ben Simo on YouTube

We now see Large Language Models and predictive text generation as the current amazeballs part of our reality. Many people are excited about these technologies, but at the same time, there are many risks surfacing, with reports of organizations suffering actual harm or damage because of using AI tools. We have heard of apps that have jacked up rates arbitrarily, published legal documents with no basis in law, fact, or reality, and taken models of "virtual people" who learn from interactions and the biases and inputs trained these models to be incredibly racist and hateful. 

These situations point to an interesting set of questions: how specifically can we as testers benefit from this wild new world of seemingly random query and response systems? How do we test software that produces inexplicable fuzzy outputs? At the end of the day, software deals with patterns and algorithms. We have technologies such as machine learning, clustering, and ways that data can be grouped and sorted. If we give LLM's a closed data set of information and ask it to work with just that information, it does a remarkably good job of transforming or "creating" work and assets. The key here is that we have given it a known and bounded set of information. Because it is bounded, it is working with a known set of information and can be guided specifically as to what to do with it. As we open it to the outer world and give it fewer controls or restrictions, we open the model up to having to look at vague clusters of data with potentially dubious provenience. A great example of this that I saw in practice during one of the workshops was with the idea of creating spec documents in markdown that would reside at the base of your document tree. By making the spec document the oracle of choice, and giving the instructions that the spec document was the arbiter of what the model was to do, we limit the chance of hallucinations and odd reactions considerably. Not completely, but we make it much easier to track what the model is doing.

An example I had fun with recently was when I heard through a podcast a story of the son of Hephaistos who went to Olympus during the waning days of the Greek pantheon's influence and took a remnant of the fire from Olympus (in an homage to the myth of Prometheus). However, something about this seemed "off", as in it was being presented as an ancient myth, but it clearly wasn't. Could we identify where the original story came from? Through various prompts, reviewing the text transcript of the story, and other clues, the LLM determined that, indeed, it was a modern story being told in the manner of an ancient Greek myth, and even noted that the delivery of the story in meter and timing mimicked closely the delivery of Hesiod. I was not able to determine who wrote the story or where to find it on the internet, but the details it did provide were interesting. Many of them felt fanciful, but all of them felt plausible. That's the danger with LLM output. Unless we are diligent, the very plausibility of the output could be accepted easily as though it were fact. Closer inspection found numerous areas that were not accurate (referencing older myths that I was aware of and had history with, but attributing individuals and characters that didn't belong there). People who may not have this knowledge or familiarity might accept what's being presented as fact because it flows so naturally and just "feels right and authoritative". 

In these cases, the Boolean PASS vs FAIL rationale doesn't work. It's not that the tests pass or fail, but that large elements do pass, but there are outputs that are "off". It's not a total fail, but it's also "corrupted" in a way that we cannot simply rely on the output. Additionally, we can run tests multiple times and get slightly different outputs with the exact same information being presented. In my own world of testing AI, we use a variety of benchmarks and monitors that help us determine if the models are behaving in ways that we expect them to. We have a variety of tests and models that allow us to determine things like Performance drift, comprehensive analysis, bias drift and disparity, the currency of the data, and comparing for homoscedasticity (a $10 word that means looking at the variance of the errors in a model and determining if it is constant/consistent across all of our observations). 

A neat tool that we have is the "AI Risk Repository" which helps us identify risks and domains of use of models where the various risks can be found. By looking at the areas that are potential risks, we can better be informed or consider aspects we can apply to our testing efforts.

Ultimately, one of the key takeaways from this talk is the idea that we are at peril of being beguiled by the magic surrounding us. We want to be responsible with our use of AI, and thus, we need to test and consider how best to apply what we learn and spread that knowledge amongst our colleagues. Magic is often sleight of hand, and it's important that we understand and can understand how that sleight of hand is performed.

Dance Bot Showcase at PNSQC: Where Robots Move to the Beat of Quality (a PNSQC Live Blog)

 This is an interesting and unusual activity... unless you have been coming to PNSQC for at least the past 15 years. The various LEGO League entries for talks and demonstrations are nothing new. A robot dance competition? Okay, that's different ;).

This year, PNSQC is hosting the first-ever Dance Bot Showcase—a celebration of creativity, code, and community!

We are starting out with a Q&A with the various team captains and learning about the challenges they faced putting on this event, and determining the criteria necessary to make this event happen.

PNSQC 2025 Robot Dance Off

Teams featured are Gear In Motion, The Beaver Bots, and Mechanical Mages (placing 3rd, 2nd, and 1st, respectively).

The various robotics teams described a number of challenges and issues they faced while preparing for this event or for other events that they have participated in. This particular demonstration is a literal "robot dance off", meaning they need to program and choreograph robots to actually dance in sync with music. These teams demonstrate technical skill as well as showcasing creativity and artistic flair. It’s more than just a performance—it's a hands-on demonstration of how software quality, testing, and intelligent systems intersect in the real world.

Each of the youth teams engages with key software concepts like adaptability, precision, and resilience through robotics. With a mix of live performances, a student-led fireside chat on software and testing challenges related to robot programming and choreography, the Showcase connects generations through shared curiosity and learning. Oh, but that's not all. Judges will develop end-user quality criteria to determine how to evaluate and judge the participants and their robot entries. Just as we would evaluate web application software quality from the end-user perspective, we will see a variety of aspects within the dance bot competition to highlight various quality characteristics.

In addition to the three local teams, we have three international teams (from the Dominican Republic, the Democratic Republic of Congo, and Mauritius) that have also contributed presentations of their dancing robots (also included in the video footage).

Have I piqued your interest? I hope so, because as soon as I can get the footage up, we will showcase the dancing robots on our YouTube channel as well as here.

Being a Change Agent with April K Mills (Welcome to PNSQC Live Blog Stream)

Hello all, and welcome to PNSQC, coming at you from Portland State University Place Hotel and Conference Center. Some of you may be hearing concerning things coming from Portland and having been here for two days now in real time, it is absolutely quiet and normal here (though I must confess that the media I have seen of protests with people in frog and chicken suits, I love how absolutely unserious Portland is. We literally started the conference with "The Beat Goes On" marching band (I will upload and link media when I get the chance).

Everyone is a Change Agent

Our first talk/keynote today is coming from April K. Mills, and her talk is "Everyone is a Change Agent". I had a chance to discuss this wth April in our PNSQChronicles vidcast (will link here later), but the key point here is that change agents don't have to be high-power, in-charge people. Every one of us has the capability, and I'd argue the responsibility, to be the change agents our organizations need. To borrow from an idea I agree with a great deal, "everyone needs to stand for something, even if doing so means we have to stand alone".

One of the ideas that April is recommending we start right now to do and be a change agent is to stand against and fight the spread of "AI Slop." How many of us have found ourselves literally struggling with and drowning in AI-generated content in our organizations that seems to say a bunch but doesn't really mean anything? I have run into this a fair bit in social media, but it is absolutely creeping into publications and corporate marketing missives. I saw a person online coin a phrase I have now started using: "AI;DR".

Note: I use this when I am seeing obvious AI output that has been generated, reads as completely soulless, and you can see that much of it is not factual or is genuinely wrong in various areas. In these cases, I do believe that announcing "AI;DR" is an excellent response.

YouTube: PNSQChronicles episode with April K. Mills

One of the things I have always appreciated (and I'm quoting Matt Heusser is the wording here) when I come to conferences like PNSQC is that I am not looking to upend the entire world with everything I learned. First of all, it's not practical, and second, systems are developed over time, and there may well be entrenched interests to overcome. Thus, it's not practical to come back from a conference with everything under the sun to overturn everything we do. Instead, we need to find a few areas that we can implement and do so without needing permission to do those things. All of us can do that. We find a way to add a new approach, process, or method based on what we learned, and then, in a few weeks or months, when people are curious as to how we have made key improvements, we can then say/demonstrate what we have been doing. There may be varying barriers to exactly how much we can do in certain areas, but we can all do something in a different way. 

When we can implement an under-the-radar change, it's often a good idea to try it out quietly and see if the change is worth pursuing. Often, we can determine pretty quickly if our brilliant idea may not be so brilliant (or at least we discover the key contexts that help explain why we haven't tried doing this before). Make no mistake, that in and of itself is often a valuable exercise. 

Sometimes the best question we can ask is "Why?" There is an idea called the "Power Paradox" that comes from the idea that people can only do so much because they don't have the power to make changes. We hear that in politics and senior management all the time. If only we could get the people at the top to agree with us and make the changes necessary. Truth be told, we don't always need to do that, and sometimes, the best way to break down that false assumption of power is to ask why we do something (strategically, don't become the kid in the back seat asking, "Are we there yet?" every five minutes). By asking "Why?" at strategic times, we may not sway the people at the top, but we may get more of us at lower levels to also ask why, and if more of us start asking why, that often triggers those higher up to realize that this is an issue that is not serving the people it intends to. Sometimes it takes more than just asking why, but strategic action often starts from just that question.

Key Takeaway: Being a change agent doesn't have to be a major initiative, and we don't have to reinvent the wheel to be a change agent. We don't need to have the right boss, with the right budget, with the right revenue, at the right time to make changes. Often, we can make changes just by trying to do some different things, and often, we can do them without having to ask permission to do so. Start there first and work your way up :).

What Are We Thinking — in the Age of AI? with Michael Bolton (a PNSQC Live Blog)

In November 2022, the release of ChatGPT 3 brought almost overnight the world of the Large Language Model (LLM) to prominence. With its uncanny ability to generate human-like text, it quickly led to lofty promises and predictions. The capabilities of AI seemed limitless—at least according to the hype.

In May 2024, GPT-4o further fueled excitement and skepticism. Some hailed it as the next leap toward an AI-driven utopia. Others, particularly those in the research and software development communities, took a more skeptical approach. The gap between magical claims and the real-world limitations of AI was becoming clearer. 

In his keynote, "What Are We Thinking — in the Age of AI?", Michael Bolton challenges us to reflect on the role of AI in our work, our businesses, and society at large. He invites us to critically assess not just the technology itself, but the hype surrounding it and the beliefs we hold about it.

From the moment ChatGPT 3 debuted, AI has seen a lot of immense fascination and speculation. On one hand, we’ve heard the promises of AI revolutionizing software development, streamlining workflows, and automating complex processes. On the other hand, there have been dire warnings about AI posing an existential threat to jobs, particularly in fields like software testing and development.

For those in the testing community, we may feel weirdly called out. AI tools that can generate code, write test cases, or even perform automated testing tasks raise a fundamental question: Will AI replace testers?

Michael’s being nuanced here. While AI is powerful, it is not infallible. Instead of replacing testers, AI presents an opportunity for testers to elevate their roles. AI may assist in certain tasks, but it cannot replace the critical thinking, problem-solving, and creativity that human testers bring to the table.

One of the most compelling points Bolton makes is that **testing isn’t just about tools and automation**—it’s about **mindset**. Those who fall prey to the hype of AI without thoroughly understanding its limitations risk being blindsided by its flaws. The early testing of models like GPT-3 and GPT-4o revealed significant issues, from **hallucinations** (where AI generates false information) to **biases** baked into the data the models were trained on.

Bolton highlights that while these problems were reported early on, they were often dismissed or ignored by the broader community in the rush to embrace AI’s potential. But as we’ve seen with the steady stream of problem reports that followed, these issues couldn’t be swept under the rug forever. The lesson? **Critical thinking and skepticism are essential in the age of AI**. Those who ask tough questions, test the claims, and remain grounded in reality will be far better equipped to navigate the future than those who blindly follow the hype.

We should consider our relationship with technology. As AI continues to advance, it’s easy to become seduced by the idea that technology can solve all of our problems. Michael instead encourages us to examine our beliefs about AI and technology in greater depth and breadth.

- Are we relying on AI to do work that should be done by humans?
- Are we putting too much trust in systems that are inherently flawed?
- Are we, in our rush to innovate, sacrificing quality and safety?

Critical thinking, and actually practicing/using it, is more relevant than ever. As we explore the possibilities AI offers, we must remain alert to the risks. This is not just about preventing bugs in software—it’s literally about safeguarding the future of technology and ensuring that we use AI in ways that are ethical, responsible, and aligned with human values. 

Ultimately, testers have a vital role in this new world of AI-driven development. Testers are not just there to check that software functions as expected, this is our time to step up and be the clarions we claim we are. We are the guardians of quality, the ones who ask “What if?”, and probe the system for hidden flaws. In the age of AI, we need to be and do this more than ever.

Michael posits that AI may assist with repetitive tasks, but it cannot match the *intuition, curiosity, and insight that human testers bring to the job. 

It’s still unclear what the AI future will hold. Will we find ourselves in an AI-enhanced world of efficiency and innovation? Will our optimism give way to a more cautious approach? We don't know, but to be sure, those who practice critical thinking, explore risks, and test systems rigorously will have a genuine advantage.

The Test Automation Blueprint: A Case Study for Transforming Software Quality with Jeff Van Fleet (a PNSQC Live Blog)

Today, delivering high-quality software at speed isn’t just a goal, it’s a necessity. Whether your organization has a small Agile team or a huge corporation, creating a streamlined, efficient testing process can dramatically reduce costs and accelerate time to market. But how do you actually achieve that transformation? Jeff Van Fleet, President and CEO of Lighthouse Technologies, goes into depth with some practical tips and proven principles to guide organizations toward effective test automation. 

One of the most important steps in transforming your organization’s approach to test automation is engaging your leadership team. Test automation initiatives often require significant investment in tools, training, and process changes—investments that can only happen with leadership support. Jeff highlights the importance of showing clear ROI by presenting leaders with real-time reporting dashboards that demonstrate how automation accelerates delivery and improves quality.

These dashboards provide visibility into the success of the test automation effort, making it easy for leadership to see the value in continuing to invest. Data-driven views and knowledge keep leadership engaged and committed to long-term quality improvement.

It's a big leap from manual testing to automation. I know, I've been there! Many manual testers may feel apprehensive about making that transition. However, Jeff emphasizes that with the right training and support, manual testers can successfully transition to automation and get fully involved in the new process. Lighthouse Technologies focuses on equipping testers with the tools, skills, and confidence to tackle automation.

We have to approach this training with empathy and patience. Many manual testers bring invaluable domain expertise, which, when combined with automation skills, can significantly enhance the quality of the testing process. Investing in your existing team, instead of sidelining them, can transform teams and build a strong, motivated automation workforce.

we've pushed the idea of shift-left testing for a while now.  Many organizations are eager to adopt it, but few know how to implement it effectively. Moving testing earlier in the development cycle helps catch bugs before they snowball into more complex, costly issues.

By collaborating closely with developers to improve unit testing, teams can identify and address defects at the code level, long before they reach production. 

One of the challenges teams face is trying to implement automation while managing in-flight releases. Jeff offers practical strategies for balancing catch-up automation (automating legacy systems or current processes) with ongoing development work. His advice: start small, automate critical paths first, and build incrementally. This allows teams to gradually integrate automation without derailing existing release schedules.

Engaging with developers is another critical component of successful test automation. Often, there’s a disconnect between QA and development teams, but Lighthouse Technologies’ approach bridges that gap by partnering closely with developers throughout the testing process. By working together, developers and testers can create more effective test cases, improve unit test coverage, and ensure that automated tests are integrated seamlessly into the CI/CD pipeline.

For organizations looking to embrace test automation, the key takeaway is that it’s not just about tools—it’s about people, processes, and leadership. By following these principles, teams can accelerate their test automation efforts and create a culture of quality that drives both speed and innovation.

When Humans Tested Software (AI First Testing) with Jason Arbon (a PNSQC Live Blog)

Are we at the edge of a new era in software development—an era driven by Generative AI? Will AI fundamentally change the way software is created? As GenAI begins to generate code autonomously, with no developers in the loop, how will we test all this code?

That's a lot of bold questions, and if I have learned anything about Jason Arbon over the years, bold is an excellent description of him. To that end, Jason suggests a landscape where AI is set to generate 10 times more code at 10 times the speed, with a 100-fold increase in the software that will need to be tested. The truth is, that our traditional human-based testing approaches simply won’t scale to meet this challenge.

Just like human-created code, AI-generated code is not immune to bugs. As GenAI continues to evolve, the sheer volume of code it produces will surpass anything we’ve seen before.  Think about it: if AI can generate 10 times more code, that’s not just a productivity boost—it’s a tidal wave of new code that will need to be tested for reliability, functionality, and security. This surge is not just a matter of speed; it’s a "complexity crisis". Modern software systems, like Amazon.com, are far too intricate to be tested by human hands alone. According to Jason, AI-generated code will require AI-based testing. Not just because it’s faster, but because it’s the only solution capable of scaling to match this growth.

The current approach to software testing struggles to keep pace with traditional development cycles. In the future, with the explosion of AI-generated code, human-based testing methods will fall short unless we somehow hire a tenfold increase in software testers (I'm skeptical of that happening). Manual testing will absolutely not be able to keep up, and automated testing as we know it today won’t be able to keep up with the increasing volume and complexity of AI-generated systems.

What’s more, while GenAI can generate unit tests, it can’t test larger, more complex systems. Sure, it can handle individual components, but it stumbles when it comes to testing entire systems, especially those with many interdependencies. Complex applications, like enterprise-level platforms or global e-commerce sites, don’t fit neatly into a context window for GenAI to analyze. This is where Jason says the need for AI-based testing becomes critical.

The future isn’t just about AI generating code—it’s about AI testing that AI-generated code. According to Jason,  AI-based testing is the key to addressing the 100X increase in software complexity and volume. Only AI has the ability to scale testing efforts to match the speed and output of Generative AI. 

AI-first testing systems should be designed to:

Automate complex testing scenarios that would be impossible for traditional methods to cover efficiently.

Understand and learn from system behaviors, analyzing patterns and predicting potential failures in ways that humans or current automated tools cannot.

Adapt and evolve, much like the AI that generates code, enabling continuous testing in real-time, as software systems grow and change.

As Jason points out, AI is not a fad or a trend, it’s the only way forward. As we move into an era where Generative AI produces vast amounts of code at breakneck speed, AI-based testing will be the way that we help ensure that the software we create tomorrow will be reliable, functional, and secure.

Humanizing AI with Tariq King (a PNSQC Live Blog)

I've always found Tariq's talks to be fascinating and profound and this time around we're going into some wild territory.

AI is evolving, and with each new development, it’s becoming more "human". It’s not just about executing tasks or analyzing data—it’s about how AI communicates, adapts, and even imitates.

So AI is becoming human... in how it communicates. That's a big statement but with that qualifier, it is more understandable. AI is no longer a cold, mechanical presence in our lives. Today’s AI can respond based on context, understanding the tone of requests and adjusting replies accordingly. It can mimic human conversation, match our language, and create interactions that feel amazingly real. Whether you’re chatting with a customer service bot or getting personalized recommendations, AI can engage with us in ways that were once the domain of humans alone.

Okay, so if we are willing to say that AI is "becoming human", how should we shape these interactions?What should the boundaries be for AI communication, and how do we ensure it serves us, rather than replaces us?

Beyond just communication, AI is showing remarkable creativity. AI can now write stories, compose music, and generate art, ranging from wild and weird to quite stunning (I've played around with these for several years, and I have personally seen the development of these capabilities and they have indeed become formidable and impressive). What once seemed like the exclusive realm of human creativity is now being shared with machines. AI is no longer just a tool—it’s being used as a collaborator that can generate solutions and creative works that blur the line between human and machine-generated content.

Tariq points out that this raises some significant and critical questions.  Who owns AI output? How do we credit or cite AI authorship? How do we confirm the originality of works? Perhaps more to the point, as AI generates content, what is the human role in the creative process? And how do we ensure that the human element remains at the forefront of innovation?

AI is getting better at how convincingly it can imitate humans. But there’s a caveat: AI is prone to hallucinations, meaning it can produce plausible and relatable material that feels right for the most part but may be wrong (and often is wrong). I have likened this in conversations to having what I call the "tin foil moment". If you have ever eaten a food truck burrito (or any burrito to go, really) you are familiar with the foil wrapping. That foil wrapping can sometimes get tucked into the folds and rolls of the burrito. Occasionally, we bite into that tin foil piece and once we do, oh do we recognize that we have done that (sometimes with great grimacing and displeasure). Thus, when I am reading AI-generated content, much of the time, I have that "tin foil" moment and that takes me out of believing it is human (and often stops me being willing to read what follows, sadly).

The challenge here is not just humanization. We need to have critical oversight over it so that we can have it do what we want it to do and not go off the rails. How do we prevent AI from spreading misinformation? And how can we design systems that help us discern fact from fiction in a world where AI-generated content is increasingly common?

Okay, so we are humanizing AI... this begs a question... "Is this something we will appreciate or is it something that we will fear?" I'm on the fence a bit. I find a lot of the technology fascinating but I am also aware of the fact that humanity is subject to avarice and mendacity. Do we want AI to be subject to it as well, or worse, actively practice it? What unintended consequences might we see or incur? 

For some of you out there, you may already be thinking of some abstract idea called "AI Governance", which is the act of putting guardrails and safety precautions around AI models so that they perform as we want them to. This means setting clear ethical guidelines, robust oversight mechanisms, and working to ensure that AI is used in ways that benefit society. More to the point, we need to continuously monitor and work with AI to help ensure that the data that it works with is clean, well-structured, and not poisoned. That is a never-ending process and one we have to be diligent and mindful of if we wish to be successful with it. 

Make no mistake,  AI will continue to evolve. To that end, we should approach it with both excitement and caution. AI’s ability to communicate, create, and imitate like humans presents incredible opportunities, but it also brings with it significant challenges. Whether AI becomes an ally or a threat depends on how we manage its "humanization".

AI-Augmented Testing: How Generative AI and Prompt Engineering Turn Testers into Superheroes, Not Replace Them with Jonathon Wright’s (a PNSQC Live Blog)

Sad that Jonathon couldn't be here this year as I had a great time talking with him last year but since he was presenting remotely, I could still hear him talking on what is honestly the most fun title of the entire event (well played, Jonathon, well played ;) ).

It would certainly be neat if AI was able to enhance our testing prowess, helping us find bugs in the most unexpected places, and create comprehensive test cases that could cover every conceivable scenario (editors note: you all know how I feel about test cases but be that as it may, many places value and mandate them, so I don't begrudge this attitude at all).

Jonathon is calling for us to recognize and use "AI-augmented testing" where AI doesn't replace testers but instead amplifies their capabilities and creativity. Prompt engineering can elevate the role of testers from routine task-doers to strategic innovators. Rather than simply executing tests, testers become problem solvers, equipped with "AI companions" that help them work smarter, faster, and more creatively (I'm sorry but I'm getting a "Chobits" flashback with that pronouncement. If you don't get that, no worries. If you do get that, you're welcome/I'm sorry ;) (LOL!) ).

The whole goal of AI-augmented testing is to elevate the role of testers. Testers are often tasked with running manual or automated tests, getting bogged down in repetitive tasks that demand "attention to detail" but do not allow much creativity or strategic thinking. The goal of AI is to "automate the routine stuff" so we can "allowing testers to focus on more complex challenges" ("Stop me! Oh! Oh! Oh! Stop me... Stop me if you think that you've heard this one before!") No disrespect to Jonathon. whatsoever, it's just that this has been the promise for 30+ years (and no, I'm not going to start singing When In Rome to you, but if that earworm is in your head now.... mwa ha ha ha ha ;) ).

AI-augmented testing is supposed to enable testers to become strategic partners within development teams, contributing, not merely bug detection but actual problem-solving and quality improvement. With AI handling repetitive tasks, testers can shift their attention to more creative aspects of testing, such as designing unique test scenarios, exploring edge cases, and ensuring comprehensive coverage across diverse environments. This shift is meant to enhance the value that testers bring to the table and make their roles more dynamic and fulfilling. Again, this has been a promise for many years, maybe there's some headway here.

The point is that testers who want to harness the power of AI will need a roadmap for mastering AI-driven technologies. there are many of them out there and there is a plethora of options in a variety of implementations from LLMs to dedicated testing tools. No tester will ever master them all but even if you only have access to a LLM system like Chat GPT, there is a lot that can be done with Prompt Engineering and harnessing the output of these LLM systems. They are of course not perfect but they are getting better and better all the time. AI can process vast amounts of data, analyze patterns, and predict potential points of failure, but it still requires humans to interpret results, make informed decisions, and steer the testing process in the right direction. Testers who embrace AI-augmented testing will find themselves better equipped to tackle the challenges of modern software development. In short, AI will not take your job... but a tester who is well-versed in AI just might.

This brings us to Prompt engineering. This is the process of precise, well-designed prompts that can guide generative AI TO perform specific testing tasks. Mastering prompt engineering will allow testers to customize AI outputs to their exact needs, unlocking new dimensions of creativity in testing.

Ss What Can we Do With Prompt Engineering? We can use it to...

-  instruct AI to generate test cases for edge conditions
- simulate rare user behaviors
- explore vulnerabilities in ways that would be difficult or time-consuming to code manually.
- validating AI outputs so that we ensure that generated tests align with real-world needs and requirements.

Okay, so AI can act as a trusted companion—an ally helping testers do their jobs more effectively, without replacing the uniquely human elements of critical thinking and problem-solving. Wright’s presentation provides testers with actionable strategies to bring AI-augmented testing to life, from learning the nuances of prompt engineering to embracing the new role of testers as strategic thinkers within development teams. We can transform workflows so they are more productive, efficient, and engaging. 

I'll be frank, this sounds rosy and optimistic but wow, wouldn't it be nice? The cynic in me is a tad bit skeptical but anyone who knows me knows I'm an optimistic cynic. Even if this promise turns out to be a magnitude of two less than what is promised here... that's still pretty rad :).

Vulnerabilities in Deep Learning Language Models (DLLMs) with Jon Cvetko (A PNSQC Live Blog)

Vulnerabilities in Deep Learning Language Models (DLLMs)

There's no question that AI has become a huge topic in the tech sphere in the past few years. It's prevalent in the talks that are being presented at PNSQC (it's even part of my talk tomorrow ;) ). The excitement is contagious, no doubt exciting but there's a bigger question we should be asking (and John Cvetko is addressing)... what vulnerabilities are we going to be dealing with, specifically in Deep Learning Language Model Platforms like ChatGPT?

TL;DR version: are there security risks? Yep! Specifically, we are looking at Generative Pre-trained Transformer (GPT) models. As these models evolve and expand their capabilities, they also widen the attack surface, creating new avenues for hackers and bad actors. It's one thing to know there are vulnerabilities, it's another to understand them and learn how to mitigate them.

Let's consider the overall life cycle of a DLLM. we start with our initial training phase, then move to deployment, and then monitor its ongoing use in production environments. DLLMs require vast amounts of data for training. What d we do when this data includes sensitive or proprietary information? If that data is compromised,  organizations can suffer significant privacy and security breaches.

John makes a point that federated training is growing when it comes to the development of deep learning models. Federated training means multiple entities will contribute data to train a single model. The benefit is that it can distribute learning and reduce the need for centralized data storage, it also introduces a new range of security challenges. Federated training increases the risk of data poisoning, where malicious actors intentionally introduce harmful data into the training set to manipulate the model’s generated content.

Federated training decentralizes the training process so that organizations can develop sophisticated AI models without sharing raw data. However, according to Cvetko, a decentralized approach also expands the attack surface. Distributed systems are nearly by design more vulnerable to tampering. Without proper controls, DLLMs can be compromised before they even reach production.

there is always a danger of adversarial attacks during training. Bad actors could introduce skewed or intentionally biased data to alter the behavior of the model. This can lead to unpredictable or dangerous outcomes when the model is deployed. These types of attacks can be difficult to detect because they occur early in the model’s life cycle, often before serious testing begins.

OK, so that's great... and unnerving. We can make problems for servers. So what can we do about it? 

Data Validation: Implement strict data validation processes to ensure that training data is clean, accurate, and free from malicious intent. By scrutinizing the data that enters the model, organizations can reduce the risk of data poisoning.

Model Auditing: Continuous monitoring and auditing of models during both training and deployment phases. This helps detect oddities in the model behavior early on, allowing for quicker fixes and updates.

Federated Learning Controls: Establish security controls around federated learning processes, such as encrypted communication between participants, strict access controls, and verification of data provenance.

Adversarial Testing: Conduct adversarial tests to identify how DLLMs respond to unexpected inputs or malicious data. These tests can help organizations understand the model’s weaknesses and prepare for potential exploitation.

There is a need today, for "Responsible AI development." DLLMs are immensely powerful and can carry significant risk potential if not properly secured. While this "new frontier" is fun and exciting, we have a bunch of new security challenges to deal with. AI innovation does not have to come at the expense of security. By understanding the life cycle of DLLMs and implementing the right countermeasures, we can leverage the power of AI while at the same time safeguarding our systems from evolving threats.

Mistakes I Made So You Don’t Have To: Lessons in Mentorship with Rachel Kibler (A PNSQC Live Blog)

I have known Rachel for several years so it was quite fun to sit in on this session and hear about struggles I recognized all to well. I have tried training testers over the years, some I've been successful with, others not so much. When a new tester comes along quickly, seems to get it, and digs testing, that's the ultimate feeling (well, *an* ultimate feeling). 

However, as Rachel points out, it’s also full of potential missteps, and as she said clearly at the beginning, "Believe me, I’ve made plenty!" This was a candid and honest reflection of what it takes to be a mentor and help others who are interested in becoming testers, as well as those who may not really want to become testers, but we mentor them anyway.

We can sum this whole session up really quickly with "Learning from our mistakes is what makes us better mentors—and better humans"... but what's the fun in that ;)?

Mistake 1: One-Size-Fits-All Training Doesn’t Work

There is no single, ideal method to teach testing that would work for everyone. Rachel had clear plans and expected to get consistent results. However, "people are not vending machines". You can’t just input the same words and expect identical outcomes. Each person learns differently, has different experiences, and responds to unique challenges.

Mistake 2: Setting the Wrong Challenges

It's possible to give team members tasks that are either too difficult or too easy, failing to gauge their current abilities. The result? Either they are overwhelmed and lost confidence, or they felt under-challenged and disengaged. Tailoring challenges to a trainee’s current skill level not only builds their confidence but also keeps them engaged and motivated. As mentors, our role is to provide enough support to help them succeed while still pushing them to grow.

Mistake 3: Don't Forget the Human Element

At the end of the day, we’re working with humans. Rachel’s talk highlights the importance of remembering that training isn’t just about passing on technical knowledge—it’s about building relationships.  Everyone has unique needs, emotions, and motivations. By focusing on the human element, we can create an environment where people feel supported and valued, making them more likely to succeed.

Mistake 4: Not Embracing Mistakes as Learning Opportunities

Mistakes are opportunities to learn. Mistakes aren’t failures—they’re stepping stones. Whether it’s a trainee misunderstanding a concept or a mentor misjudging a situation, these moments are chances to grow. They teach us humility, patience, and resilience.

Rachel’s talk is a reminder that no one is a perfect mentor right out of the gate. The process of becoming a great mentor is filled with trial and error, reflection, and growth. Also, Imposter Syndrome is very real and it can be a doozy to overcome.  Ultimately, the key takeaway is this: mentorship is a journey, not a destination. We will make mistakes along the way, but those mistakes will help shape us into more effective, empathetic, and responsive mentors.

Scaling Tech Work While Maintaining Quality: Why Community is the Key with Katherine Payson (a PNSQC Live Blog)

If someone had told me ten years ago I'd be an active member of the "gig economy" I would have thought they were crazy (and maybe looked at them quizzically because I wouldn't entirely understand what that actually meant. in 2024? Oh, I understand it, way more than I may have ever wanted to (LOL!). Rather than. looking at this as a bad thing, I'm going to "Shift Out" (as Jon Bach suggested in the last talk) and consider some aspects of the gig economy that are helping to build and scale work and, dare we say it, quality initiatives. 

Katherine Payson offers some interesting perspectives:

- The gig economy generates $204 billion globally
- many companies are leveraging and taking advantage of this, including international companies hiring all over the world for specific needs (I know, I did exactly this during 2024)
- In 2023, the anticipated growth rate for gig work was expected to be 17%
- By 2027 the United States is expected to have more gig workers than traditional full-time employees

This brings up an interesting question... with more people involved in gig work, and not necessarily tied to or beholden to a company for any meaningful reasons, how do these initiatives scale, and how do quality and integrity apply?

Strong Community is the approach that Katherine is using and experiencing over at Cobalt, a company that specializes in "pentesting-as-a-service". Cobalt has grown its pool of freelance tech workers to over 400 in three years. That's a lot of people in non-traditional employment roles. So what does that mean? How is trust maintained? How is quality maintained? Ultimately, as Katherine says, it comes down to effective "Community Building".

Today, many businesses are looking for specialized skills, frequently beyond what traditional full-time employees and employment can do. Yes, AI is part of this shift but there is still a significant need for human expertise. As Cobalt points out, cybersecurity, software development, and other technical fields definitely still require human employees with a very human element to them. What this means is that there is a large rise in freelance professionals actively offering niche talents on a flexible, on-demand basis (likely also on an as-needed basis both for the companies and the gig workers themselves). Again, the bigger question is "Why should a gig worker really care about what a company wants or needs?"

Community can be fostered directly when everyone is in the same town, working on the same street, going to the same office. When Cobalt first began scaling, they relied on a traditional trust model that worked well for a smaller, more centralized team. As the number of freelancers grew, however, this model began to show its limitations. Without a more robust system in place, it would be impossible to ensure consistent quality across a distributed workforce.

Tools can go a certain distance when it comes to helping manage quality and production integrity but more to the point, developing actual communities within organizations is another method for helping develop quality initiatives that resonate with people from all involvements in their organizations.

Cobalt prides itself as a company that is able to maintain quality at scale. It claims to create a culture where freelancers feel connected, supported, and motivated to deliver their best work. So how does Cobalt do that?

Collaboration and Communication: Freelancers can work independently, but they don't work in isolation. Cobalt believes in open communication, where freelancers can collaborate with one another, share knowledge, and learn from each other’s experiences.

Mentorship and Professional Development: Cobalt invests in the professional growth of freelancers. Mentorship opportunities, training programs, and access to industry resources help their freelance community continuously hone their skills.

Recognition and Incentives: High-performing freelancers are recognized and rewarded for their contributions. This helps retain top talent and encourages others to aim for top-quality work.

Feedback Loop: Freelancers receive regular feedback on their work, helping them improve and keep quality high across the board.

As the gig economy continues to grow, maintaining quality at scale will become increasingly important everywhere. Cobalt aims to embrace the strengths of their freelance workforce, not just as individual contributors but as part of a larger community. Scaling with freelancers is not just about hiring more people—it’s about building a culture of collaboration, growth, and trust. To ensure quality remains front and center, companies need to invest in their communities every bit as much as much as they do in their tools and processes.

Exploring Secure Software Development w/ Dr. Joye Purser and Walter Angerer (a live blog from PNSQC)

Okay, so let's get to why I am here. my goal is to focus on areas that I might know less about and can see actionable efforts in areas I can be effective (and again, look for things I can use that don't require permission or money from my company to put into play).

Dr. Joye Purser is the Global Lead for Field Cybersecurity at Veritas Technologies. Walter Angere is Senior Vice President for Engineering at Veritas and co-author of the paper. To be clear Dr. Purser is the one delivering the talk.

Creating secure software involves a lot of moving parts. So says someone who labels herself as "at the forefront of global data protection."

High-profile security incidents are increasing and the critical nature of secure software development is needed more than ever. Because of high-profile cases that end up in the news regularly, Dr. Purser shared her journey and experiences with Veritas, a well-established data protection company. She shared their journey of ensuring software security.

Veritas has a seven-step SecDevOps process, demonstrating how they aim to control and secure software at every stage.

1. Design and Planning: Building security in from the outset, not bolting it on as an afterthought.

2. Threat Modeling: Identifying potential threats and mitigating them before they can become problems.

3. Code Analysis: Veritas uses advanced code analysis tools to identify vulnerabilities early in the process.

4. Automated Testing: Leveraging automation to continuously test for weaknesses.

5. Chaos Engineering: Veritas has a system called REDLab, which simulates failures and tests the system’s robustness under stress.

6. Continuous Monitoring: Ensuring that the software remains secure throughout its lifecycle.

7. Incident Response: Being prepared to respond quickly and effectively when issues do arise.

A little more on chaos engineering. This technique actively injects failures and disruptions into the system to see how it responds, with the idea that systems are only as strong as their weakest points under pressure. Veritas' REDLab is central to this effort, putting systems under tremendous stress with controlled chaos experiments. The result is a more resilient product that can withstand real-world failures

Veritas also focuses on ways to validate and verify that code generation is done securely, along with a variety of ways to stress test software during multiple stages of the build process. The talk also touched on the importance of keeping technical teams motivated. Including examples of role-playing scenarios, movie stars, and innovative content ads a touch of fun and can help keep development teams engaged. 

As technologies evolve, so do the techniques required to keep software safe. Security is needed at every stage of the software development lifecycle. Using techniques like chaos engineering along with creative team engagement has helped Veritas stay at the front of secure software development.

Empathy is a Technical Skill With Andrea Goulet (PNSQC)

 

Today has been a whirlwind. I was up this morning before 5:00 a.m. to teach the penultimate class of my contract (sorry, I just love working that word into things ;) ) but suffice it to say the class is still happening while PNSQC is happening. That has made me a little tired and thus a little less blogging today. Add to that the fact I was called in to do a substitute talk in the afternoon (glad to do it but that was not on my dance card today) and I'm really wondering how we are at the last talk of the day and formal conference. regardless, we are here and I'm excited to hear our last speaker.

I appreciate Andrea talking about this topic, especially because I feel that there has been a lot of impersonal and disinterested work from many over the past several years. I was curious as to what this talk would be about. How can we look at Empathy as a technical skill? She walked us through an example with her husband where he was digging into a thorny technical problem that was interrupted by Andrea asking him for a moment. His reaction was... not pleasant. As Andrea explained, she realized that he was deeply focused on something so all-consuming that it was going to be a big deal to get his attention for needful things. Instead of it being an ugly altercation, they worked out a phrase (in this case, "Inception") to help see when a person is on a deep dive and needs to be in their focused state, at least for a little while longer. While I don't quite know that level of a dive, I have times in my own life when I get caught up in my own thoughts and I bristle when someone interrupts/intrudes. By realizing these things, we can not just recognize when we ourselves are focusing on deep dives, but we can also recognize when others are as well. This is a development of our own empathy to aid us in the process of understanding when people are dealing with things.

Okay, that's all cool, but why is this being called a technical thing? Because we are free and loose with the use of the word "technical". Technical comes from the Greek word "Techne", and techne means "skill". That means any skill is technical when we get down to it. It also means it's a skill that can be learned. Yes, we can learn to be empathetic. It's not something we are born with, it's something we develop and practice. Motivation likewise drives empathy. In many ways, empathy can be a little mercenary. That's why we get it wrong a lot of the time. We often want to reach out and help in ways that we would want to be helped, and thus our empathy is highly subjective and highly variable. Additionally, empathy grades on a curve. There are numerous ways in which we express and experience empathy. it's not a monoculture, it is expressed in numerous ways and under different circumstances and conditions. There are a variety of components, mechanisms, and processes that go into our understanding and expressions of empathy. It's how we collaborate and solve complex problems. In short, it's a core part of our desire and ability to work together.

Andrea showed us a diagram with a number of elements. We have a variety of inputs (compassion, communication) that drive the various mechanisms that end up with a set of outputs. Those outputs come down to:

• Developing an understanding of others 
• Creation of Trust
• A Feeling of Mutual Support
• An overall synergy of efforts   

 Empathy requires critical thinking. It's not all feelings. We have to have a clear understanding and rational vision of what people want, and not specifically what we want. 

On the whole, this is intriguing and not what I was expecting to hear. Regardless, I'm excited to see if I can approach this as a developed skill.