Continuous Testing Integration With CI/CD Pipeline (PNSQC)

Today, I'm taking a walk down memory lane. I'm listening to Junhe Liu describe integrating various automatic tests into the CI/CD pipeline.

It's interesting to think about where we are today compared to 30 years ago when I first came into the tech world. Waterfall development was all that I or anyone knew (we may not have wanted to call it that, or we'd dress it up. Realistically speaking, any given release was a linear process, and each sequence flowed into each other. While I had heard of Agile come the early 2000s, I didn't work on such a team (or one that presented itself as such) until 2011. 

Likewise, it was around the mid-200s that I started hearing the idea of Development and Operations being two great tastes that went better together being discussed ;). Again, it wouldn't be another decade until I saw it in practice but over time, I did indeed start to see this and I was able to participate in it. 

One of the interesting arrangements in the group I was working at (Socialtext), every member of the team had their turn at being the "Pump King". That's a piece of lore that I miss and it is a long story involving an old USB drive that was kept in a toy jack-o-lantern bucket, hence the person who took care of the protective pumpkin became known as the "Pump King" and after everything went online, the name stuck. The key point was that the Pump King was the person responsible for the Jenkins system and making sure that it was working, up to date, and patched when necessary, as well as running it to build and deploy our releases. Every few weeks, it would be my turn to do it as well. 

Thus it was that I was brought into the world of Continuous Delivery and Continuous Deployment, at least in a limited sense (most of the time this was related to staging releases). We actually had a three-tiered release approach. Each developer would deploy to demo machines to test out their code and make sure it worked in a more localized and limited capacity. Merging to the staging branch would trigger a staging build (or the pump king would call one up whenever they felt it warranted, typically at the start of each day. We'd run that and push changes and version numbering to our staging server, and then we'd run our general tests, as well as all the underlying automated tests with the Jenkins process, of which there were a *lot* of them. Finally, due to our service agreements, we would update our production server and then push uploads to customers who opted in to be updated at the same time. We never go to production daily pushes but weekly was more common towards the end of my time on that product.   

It was interesting to get into this mode and I was happy that we were all taught how to do it, not just one and when needed but that all of us were expected to be able to do it at any time. Thus all of us knew how to do it and all of us were expected to do it every time it was our turn to be Pump King. 

Bad Tests Running Wild - an #InflectraCON2023 Live Blog

 

Paul Grizzaffi

Senior QE Automation Architect, Vaco

Paul and I go way back. It's always fun to see my fellow in heavy metal arms at these events. We frequently talk music as much as we talk testing, so we are often in each other's sessions and today is no exception. Plus, Paul and I love making musical puns in our talk titles, and seeing Bad Tests Running Wild, I knew that was a reference to Scorpions' lead-off track from 1984's "Love at First Sting", aka "Bad Boys Running Wild"... yeah, this is going to be fun :).

The point here is that, especially with CI/CD pipelines, we need to have the tests pass to successfully complete and deploy an application. If a test fails, the whole process fails. By virtue of how tests run in a CI/CD pipeline, we need to make sure that any test that we have can run all the time, independent of any other test, and independent of any state of our product. This means a flakey test can really derail us. Note, this is not talking about a test legitimately failing or finding a fault. This is more the "random timeout because of a latency that occurs and that has nothing to do with our application".    

Let's think about how we create our calls and procedures. Do we have everything under our own umbrella? How much of our solution uses third-party code? Do we understand that third-party code? If we are using threaded processes for concurrency, are all of our components able to use those concurrent thread approaches?  

Let's think about configuration and how we set things up. Why do we want or need parallelization? Overall, it comes down to time and speed. I remember well our earlier setup with Jenkins from about a decade ago. It took us several hours to run everything in serial. Thus, we needed to set up the environment in such a  way that we could run four servers in parallel. At a point, we have to look at the costs of running our CI/CD pipeline vs. the time it takes to deploy. Our sweet spot was determined to be four servers running in parallel. Those four servers ran our tests in twenty minutes and then did our deployment if everything went smoothly. Going from several hours to twenty minutes was a big time saving but yes, it cost to set up robust enough servers to get those savings in time. After those four servers, we determined that adding more servers created a less favorable cost to time savings, as compared to running four servers. Still, it was critical to make sure that any tests we ran and any states that changed had to be all self-contained. No test was allowed to leave any residual footprints. Additionally, we had to ensure that our main server and out client machines were responding quickly enough to make sure that we didn't have potential latency with multiple machines (heck, spinning up a machine in a different server farm could mess everything up, so you needed to make sure that everything was proximate to each other.   

Also, we are only considering what happens when a test fails when we don't want it to or it's not supposed to fail. However, we also have to consider the flip side, which is what happens if a test passes that shouldn't? That's the flip side of a flaky test. What if we have made a change but our test is too generic to capture the specific error that we have introduced? That means we may well have introduced a bug that we didn't or wouldn't catch. 

Risks are always going to be present and our goal as testers and automation specialists is that we want to look at the potential risks that are present. What is the basic risk we need to mitigate? What happens when we deploy our systems? Do we have the ability to back out of a change? What do we need to do to redeploy if necessary? If we deploy do we have an easy way to monitor what has gone in? Paul makes the point that, if a change is potentially expensive, then you probably need human eyes to watch and monitor the situation. If there's little cost or risk of failures, then it could be handled without a person looking over it. Regardless, you will need to have the ability to monitor and to that end, you need logs that tell you meaningful information. More to the point, you need to know where the logs are and that they are actually accessible.

As always, exciting, interesting, and great fod for thought. Thanks, Paul. Rock on!!!

How Holistic Testing Affects Product Quality with @janetgregoryca (@Xpanxion #QASummit 2021) : Live Blog

 We're down to our final keynote and it's a pleasure to see Janet Gregory, if only virtually, this year. Since the border situation between USA and Canada is still in question (and considering the situation with outbreaks we are seeing, I don't blame it in the slightest), We're still getting to hear Janet talk about the value of DevOps and the fact that it genuinely works when the teams in question genuinely put in the time and energy to make sure that the teams can work. 

Quality is always a vague and odd thing to get one's head around. What makes something good to one person may not be so excellent to someone else. In some areas it is objective but much of the time it is subjective and not even related to the end product itself. Janet uses the example of a cup of coffee. For some, the best coffee is experienced black, so that every sense of the flavor of the beans can be examined. For others, the best crafted iced frappuccino with all of the extra flavors makes the experience a quality one. Does one approach replace the validity of the other? It really doesn't but it matters a lot to the person in question at that point in time. Quality is what matters to a person experiencing the item in question and in the way that they want to experience it.

So, how do you build quality into your product? In many cases, quality is not just one figure but many that come together. Some may argue that Lamborghini sports cars are of high quality. I may or may not agree but the cost for a Lamborghini puts it well out of the range where I will ever find out. Is the level of quality a consideration if you can't consider paying for it? If it is super affordable, does that automatically mean the product is of low quality? Not necessarily. I'm reminded of the app Splice, which is a video editing app that I use on my phone. Granted, I pay for it (about $3 a week) but their regularity of updates and their method of continually improving the product makes it worth that expense for me. It's not s much that it is going to discourage me but it also provides me a value that makes me willing to keep paying for it.

Holistic Testing focuses on the idea that testing happens all the time. To that end, Janet is not a fan of the terms shift-left or shift-right testing. The real question is, "what do you mean you are not doing active testing at every stage of the process?" It does help to know all areas where testing makes sense to perform and why/when we would do it. It may honestly have never occurred to people that monitoring and analytics after a product is released fits into testing and that testing can actually learn from these areas to help improve the product. 

One of the best phrases a tester can use/encourage is "can you show me?" I find that when working with developers and testers, many misconceptions and miscommunications can be avoided just by asking this question.  Using AB/Testing, feature flags, or toggles to turn on or off features allows us to do testing in production without it being a scary proposition. We also get to observe what our customers actually do and use and from that we can learn wi=hich features are actually used, or for that matter even wanted in the first place. We may also discover that features we develop to serve one purpose may actually be used in a different manner or for a different purpose than we intended. With that kind of discovery, we can learn how to better hit the mark or to provide features that we may not even be totally aware are needed.

The key to realize is there are testing initiatives that happen at every level of software development. It's important for us as organizations, not just us as testers, to learn how to leverage that testing focus at all levels and be able to learn, experiment, confirm or refute, and then experiment again. It will take time, it will take involvement, it will take investment, and it will take commitment. Still, the more that we are abe to leverage these testing areas, the better our overall quality approach will have the potential to be.

Cutting Releases Cadence - a #PNSQC2019 Live Blog

OK, time to put on my Release Manager and Build Manager hat. For the past few years, outside of being a software tester this has been my most visible functionality within the team. There are a lot of moving parts in this process that I had to come to grips with and get a feel for what exactly it was that I was doing to make releases and deploy them. We do well with Continuous Integration. Continuous Delivery and Deployment are areas we can certainly do better, hence why I am here :).

We have some rules in place at my company and its parental units that mean that true push-button Continuous Delivery and Deployment will likely never happen in actual production. Well, saying "never" may be a bit of hyperbole but much would need to change before our organization would be OK with doing it that way. Still, just because there are limitations to CI/CD, that doesn't mean that in other cases we couldn't or shouldn't be able to do it. We have development environments, staging environments, and integration environments. They need to be provisioned and set up just like any customer site. Those steps are not exactly changing day to day if you get my drift :). Thus, it makes perfect sense to think that we should be able to do CI/CD on a more frequent basis, even if we are the only ones (the engineering team) who reap the everyday benefits.

I can totally feel how And Peterson's organization went through the processes he did to try to wrangle this monster to get a system in place that required less hand-holding and allowed for more time to work on genuinely interesting challenges.

Also, just because you have a process that is push-button does not mean that you always have to do it that way. All that it means is hat the parameters necessary are well understood and repeatable. If you can repeat them, you can standardize them. If you can standardize them, you can package them. If you can package them, you can set them into containers or other structures that allow us to maximize the amount of information that replicates and doesn't change, speeding up our deployments and limiting the time we have to wait between a release build finishing and th time when an environment is up and running with our application in a usable state.

Even with this approach, we are still limited to other teams in our company and what they can and will be able to release. Again, just because your product may not go out every day, there is no reason to not be able to create a staging environment that will benefit from these changes. While we may have a quarterly release cadence, there is nothing stopping us from getting into a daily cadence to push features in fully qualified builds to our staging server. Granted, this does mean that we have to go back and do a little bit of repeating to see if pushing a lot of the changes to a numbered build introduces anything unusual. Still, we have had a chance to see everything working in the staging environment so this shouldn't be a barrier in practice. I say that now, but let's see how that works in practice ;).