Into the Depths of Risk Analysis to Improve Your Testing: an #InflectraCON2023 Live Blog

It has been a while since I've done a blog update. Granted, it's been a while since I've been anywhere so reality has been much of the same but I am currently at InflectraCON and taking notes, so you can all come along for the ride if you'd like :).

Bob Crews

CEO, Checkpoint Technologies

Our first talk is with Bob Crews and is covering Risk Analysis to improve testing. Interestingly, we have seen the complexity of software development explode over the past couple of decades. Web sites and apps have matured significantly and what they can do has increased exponentially and continues to do so. By virtue of that, sites and apps are becoming more challenging to test every day. We can't test everything, no matter how delusional we believe ourselves to be. Thus, we have to apply a different metric. We have to consider what is critical and of most importance, and then work our way down from there to "nice to have" long before we ever get remotely close to "we've done verything" (trust me, no one gets to that point, ever).

With this, we need to make sure that we have a clear understanding of what areas are most important, what risks we face, and how we are able to mitigate those risks, to the best of our ability. We can't prevent risk but we can do some mitigation in the process. By analyzing what the potential threats are, we can make sure that we put the most important situations at the forefront. Elisabeth Hendrickson often led with the idea of waking up and seeing your company on the front page of the local newspaper. What would be the most terrifying thing you could see in those headlines? If you can envision that, then you can envision what the potential risks are if your product were to fail. Odds are, we will never face anything that dire but it illustrates the critical elements that we should be alert to. By putting those horrorshow examples front and center, you have done a simple risk analysis of what could go wrong. From there, you can start to consider what would be next in line, and then consider how to mitigate those potential issues.

To be clear, risk assessment is a time-consuming process and can be as formal or informal as you want to make it. It can be an enterprise-level operation and exercise, or it can be a personal and singular effort just for our own benefit. I'm not sure how many people have pipeline CI/CD systems but much of the time, we have created tests that are independent and can run in any order. That's great for parallelization and speed but it may not be the best approach for risk mitigation. In a randomized, parallelized environment, every test is basically considered equal. Every test has the same potential to be a pass or fail and every test can stop the pipeline until it is resolved. How often do we find ourselves working on trivial tests that stop the system while something major doesn't even get run? There are possible ways to set up a prioritized run and make those the tests that get run first and cover the broadest area possible. By doing this, we can schedule and structure our tests so that they run in a criticality order. Think of it as placing your tests in folders, where those folders are rated by priority. We would of course want to run the tests in folder #1 before we run the tests in folder #9. To determine what those tests are based on that kind of hierarchy, we would need to evaluate and assign a risk assessment to each test.

By taking the time to look at a test, giving it a risk impact score, a likelihood that it might happen, and the possible frequency that it might happen, we can determine which bucket an item falls into. Also, high impact is subjective much of the time, but there are places where that subjectivity can rise from annoyance to a critical issue. Over time, we can get to the point where we might assign a weight to these tests, let's say that 99 is a top weight and 10 may be a minimal weight (I'd argue anything less than 10 may not even be worth running, at least not daily or as part of the full CI/CD commitment).

The fact is, we often look at risks as being "Acceptable". For years, Accessibility and Inclusive Design are low priority items unless legal action pushes them to the forefront. Accessibility may be seen as a low-priority item unless a big client demands it to buy your product. Then Accessibility rapidly rises to the top of your risk list. Security is always a top-level and critical area but how much is critical? If everything security related is critical, then nothing really is. Of course, we want to keep the system secure but what level is intelligent and prudent coverage and what level is overkill? To that end, we create a Risk computation, based on the classic four quadrants (urgency and impact, meaning we have at level 1 high risk and high impact, level 2 being low risk and high impact, Level 3 being high risk and low impact, and level 4 being low risk and low impact). Level 1 is of course the most important and arguably Level 3 is the next most important. Level 4 is probably not even worth our time but again, circumstances can move any of these situations into a different quadrant. This is why risk assessment is never a "one-and-done" thing.

There's a phrase called the "wisdom of the crowd" where the idea is a large group of people can determine what is important. If enough people consider an issue to be an issue, it will be addressed. It may or may not make a lot of sense on the surface but if enough people consider it important and make known the fact it is important, best be sure it will be considered and worked into whatever process is necessary to have it be addressed. The crowd is not always right but it is often a good indication of conventional wisdom. Usability often falls into this. While we may decide that a process is logical and rational, if enough users disagree with us and decide they will not use our product because of it, it will become a talking point and possibly a critical one if enough people voice their displeasure. 

Over time, we can get pretty good at looking at the risk areas we face and weigh them in order of how critical they are. We may never get to a perfect level, but we will come closer to a workable risk assessment that will help us address the most needful things and prioritize those areas over just trying to be thorough and cover everything.

From Fear To Risk: An #InflectraCON Live Blog

Next up is Jenna Charlton with a realistic look at the rhetoric of Risk-Based Testing. As many may well be aware, there's a lot of branding and promises that surround a variety of terms. Many of the phrases that we like to use have a certain comforting ring to them. Risk-Based Testing is one of them. Think about what it promises. If we identify the areas of greatest risk and test around those areas, we can deliver the best bang for the buck quality and we can do it so much faster because we are not testing every single thing. 

Sounds great, right? However, what does this ultimately tell us? We have said we care about risk but what actually is risky? We are only alert to the risks if we have thought about them. The biggest fear I have when I think about doing a risk assessment is that I have made risk assumptions from what I know and can anticipate. Is that really a good risk assessment? It's an okay and workable one. However, if I'm not able to consider or understand certain parameters or areas that may be blind spots to me, I cannot really do a great risk assessment, so my risk assessment is incomplete at best and flying blind at worst. 

One of the first things that can help ground us in these considerations is to start with a simple question... "what am I most afraid of?" Understand, as a tester, what I am most afraid of is missing something important. I'm afraid of having shallow coverage and understanding. That's not necessarily something that a general risk assessment is going to focus on. How many of us have said, "I don't know enough about the ins and outs of this system to give a full risk assessment here"? I certainly have. What can I do? Much of the time, it's a matter of bringing up my concerns about what I know or don't know and being up-front about them. "I have a concern about this module we are developing because I do not feel I fully understand it and thus, I have foggy spots here and here". Sound familiar? What is the net result of this? Do we actually get a better understanding of the components and that leads to a more lean testing plan because now we know the items better? Do we double up our coverage and focus model so we can "be sure" we've addressed everything? Here's where risk assessment breaks down and we fall back into the "do more testing, just to be sure" approach.  

Something else that often doesn't get addressed is the fact that what is a risk at one point in time, as the organization matures and they have covered these areas, risk in those areas actually goes down. Still, how many of us have continued focusing on the "riskiest areas" because tradition has told us that they are, even though we have combed through every aspect of this area we consider so risky. If you have made tests for a risky area, you've run them for an extended period, and no problems have been found (the tests pass all the time), what does that tell us? It could tell us we have inadequate tests (a real risk, to be sure) or it could also tell us that this area has been thoroughly examined, we've tested it vigorously and now we have a system in place to query multiple areas. In short, this area has been moved into an area where it might be risky if something blows up but as long as it doesn't, the risk is actually quite low. Thus, we now have the ability and the need to reassess and consider which risks are the current ones, not yesterday's.

We have to come to grips with the fact we will never cover every test possible and as such, we will never fully erase the risk. Also, we will never get it perfect. Still, we often operate under the assumption that we will be blamed if something goes wrong, or that we made bad assumptions, and of course, we fear the retribution if we get it wrong. Thus, it helps to see how we can mitigate those fears we have. If we can quantify the risk and define it, then we can look at it objectively, and with that, we can better consider how we will address what we have found. Are afraid of an outcome (nebulous) or are we addressing the risks we can see (defined and focused)? To be clear, we may get it wrong, or we may make a mountain out of a molehill. Over time, we might get better at that. Our goal is to deal with the molehills effectively but miss the entire mountain. 

Again, there's a chance that we will miss things. There's a chance something that matters to our organization will not get the scrutiny it deserves. Likewise, fear may be making us focus on solidly functioning software over and over again because "it just pays to be safe" only to realize we are spending so much time on an older risk that isn't as relevant now. It's more art than science but both are improved with practice and observation. 

Risk Based Testing - a #PNSQC Live Blog

Its a fact of life. We can't test everything. We can't even test a subset of everything. What we can do is provide feedback and give our opinion on areas that may be the most important. In short, we can communicate risk and that's the key takeaway of Jeny Bramble's talk. By the way, if you are not here, you are missing out on Dante, the deuteragonist of this presentation (Dante is Jenny's cat ;) ).

Jenny points out off the bat that, often, words are inadequate when it comes to communicating. That may sound like unintentional irony but I totally get what Jenny is saying. We can use the same words but have totally different meanings. One of the most dangerous words (dangerous as in its fluidity) is "risk". We have to appreciate that people have different risk tolerances, often in the same team. I can point to my own team of three testers and I can feel in our discussions that risk is often a moving target. We often have to negotiate as to what the level of risk actually is. We get the idea that risk exists, but how much and for whom is always up for discussion.

Jenny points out that risk has a variety of vectors. There's a technical impact, a business impact, and a less tangible morale impact. When we evaluate risk, we have to determine how that risk will impact us. What is the likelihood that we will experience failure in these scenarios? I often have these discussions when it comes to issues that I find. Rather than just come out and say "this is a bug!", instead, I try to determine a consensus of how bad this issue might be. This is often done with discussions with our product owner and asking questions like "if our customers were to see this, what would your impression be?" I likewise have similar discussions with our developers and often, just asking questions often prompts people to look at things or to have them say "hey, you know what, give me a couple of hours to harden this given area".

Risk isn't always limited to the feature you are developing at the given moment. A timetable changing is a risk. Third party interactions can increase risk, sometimes considerably. If your infrastructure is online, consider where it is located (Jenny is from North Carolina and as many are probably aware, we recently had a hurricane sweep through that made a mess of Eastern North Carolina. Imagine if your co-lo was located there.

Ultimately, what it comes down to is being able to perform an effective risk assessment and have a discussion with our teams about what those risks are, how likely they are to happen, and ultimately how we might be able to mitigate those risks.

Jenny has a way of breaking down a risk matrix to make it a numerical value. By looking at the level of likelihood with the level of impact, multiply the two numbers and that gives you the risk factor. A higher number means higher risk and higher efforts to mitigate. Lower values mean lower risk and therefore lower cost to mitigate.

"This feature has been drinking heavily and needs to go to rehab!" Best. Risk. Based. Metaphor. Ever (LOL!).

This is my first time seeing Jenny present, though I see her comments on Twitter frequently. If you haven't been to one of her presentations, may strongly suggest that, should she be speaking at a conference near you, that you make a priority to see her speak? Excellent, my work here is done :)!

Customer Quality Dashboard: #PNSQC Live Blog

OK, the late night flight arrival into Portland and the early morning registration is catching up to me. I will confess that I'm hunting down Diet Coke and alertness aids of the non-coffee or tea types. If I seem a little hazy or less coherent, well, now you will know why ;).

In any event, it's time for John Ruberto's "How to Create a Customer Quality Dashboard" and he started with a story about losing weight, and a low-fat diet that he was on for awhile. The goal was to get less than 20% of his calories from fat. He soon realized that having a couple of beers and eating pretzels helped make sure that the fat calories consumed were less than 20%. Effective? Well, yes. Accurate. In a way, yes. Helpful? Not really, since the total calories consumed went well above those needed for weight loss, but the fat calories were under 20% all the time ;).

This story helps illustrate that we can measure stuff all day long, but if we aren't measuring the right things in context, we can be 100% successful in reaching our goals and still fail in our overall objective.

To create a usable and effective dashboard, we need to be able to set goals that are actually in alignment with the wider organization, focusing on what's most important to stakeholders, providing a line of sight from metrics to goals, and build a more comprehensive view of our goals.

Let's put this into the perspective of what might be reported to us from our customers. What metrics might we want to look at? What does the metric tell us about our product? What does the metric not tell us about our product?

Some things we might want to consider:

Process metrics vs Outcomes: Think vines of code per review hour vs. defects found per review.
Leading Indicators vs. Lagging Indicators: Think code coverage vs. delivered quality.
Median vs. Average: Median page load vs. average page load. Average can skew numbers
Direct Measures vs Derived Measures: Total crashes vs reported crash codes

There are a lot of potential issues that can cause us problems over time. One is gaming the system, where we set up metrics that we can easily achieve or otherwise configure in a way that is not necessarily supported in reality. See the example of the fat percentages. We could adjust our total intake so our fat calories were below 20%,

Confirmation Bias: is a preconceived notion of what things should be, and therefore we see or support results that help us see that reality. 

Survivor Bias: The act of seeing the surviving instances of an aspect or issue as though it's the whole of the group.         

Vanity Metrics: Measuring things that are easily manipulated.


                                                                                                                   

Keep Your Project off the Autopsy Slab: #PNSQC Live Blog

All right, my talk is over, and from what I can tell from the comments back it seems to have gone over well :). Lunch was good and we got to get into the 35th Anniversary cake. The committee also put out cupcakes and I'm pretty sure this had to have been put out there as a joke:

I am happy to hear that no Vegans were harmed in the making of these cupcakes ;).

The first talk for the afternoon is to talk about Pre-Mortems, with the subtitle of "Keep Your Project off the Autopsy Slab". To make it simple, there'are three basic things to consider in a pre-mortem:

1. Something has failed.
2. You and a group have determined and found the issue.
3. Steps are put in place so that the issue doesn't happen again.

Put simply, projects frequently fail, and they fail for predictable reasons. Sometimes they are event-based, such as a person quits, or the schedule changes, or an integration is needed. Sometimes it's people related, such as a skill set is missing, or someone is not dedicated to the project. At times the culture works negatively, such as fostering overt competitiveness or being overly skeptical. Additionally, the goal to get it out there overrides the willingness and ability to do the job right. Technology can be a problem, as in external third party software elements seem great on paper but we end up struggling to try to use it.

One additional issue that can sabotage a project is a lack of knowledge, specifically with the idea of "unknown unknowns" as in "we don't know it and we don't even know we don't know it".

If we are looking at a pre-mortem as a way to get a feel for where things are likely to go, we first have to identify as many risks as possible, but just as important is determining if, and how, it will be possible to mitigate those risks. Mitigation is an oddball term, in that it's not entirely clear as to what we are doing. Are we minimizing risk? Reducing risk? In general, it's lessening the impact of risk as much as possible.

There are eleven ideas to consider with pre-mortems:

1. What is the scope?
2. How do we determine the logistics?
3. What instructions do you send?
4. How can you imagine the project failing?
5. What are the reasons for potential failure (brainstorm)?
6. Now, let's create a master list (consolidate, de-duplicate and clarify).
7. What items can we remove from the list due to low impact (filtering)?
8. What reasons of potential failure could be real risks to the real project?
9. What does the group consider important? Reorder based on importance (VOTE).
10. Action Items to mitigate the top risks.
11. Action items need to actually be done. They are the whole point of the pre--mortem.

Understand Your Business and Customer Needs: 99 Ways Workshop #93

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #93: Understand your business and customer needs, not just the requirements - Mike Hendry

I was so tempted to put this into a combined post with the last one, because a lot of the suggestions would be the same. However, a  funny thing happened on the way to throwing in the towel and saying "OK, this could use the same advice as the one that I just wrote". I participated in a conversation on Twitter with another tester that was frustrated with the fact that "meaningless metrics" were getting their team nowhere. While they were still being asked for measurements, there was a chance that management might be open to another conversation. In the process of that conversation, we came down to a piece of advice that I realized answered this suggestion pretty well. I love when things like that happen :).

Workshop #93: Sit down with your management/product team and have a frank discussion about risk. What are the real risks that your product faces? Get specific. Map out as many areas as you can think of. Identify key ares that would be embarrassing, debilitating, or even "life threatening" for an organization. Develop a testing strategy that focuses on one thing, mitigating those risks.

Most organizations couch testing in terms of coverage or bugs found. It's a metric, and it may even be a meaningful metric, but very often, it's not. "Coverage" is vague. What do we mean? Are we talking about statement coverage? Branch coverage? Usually, the term "test coverage" is used, and again, probe deeper and see if that word means what the speaker thinks it means. If someone asks if "all possible test cases have been identified", we've got problems. At this point, it would be helpful to instruct and show that complete, exhaustive testing of all scenarios is impossible (not to mention infinite).


In most of the places I have worked, "risk" has not been directly assessed, and there are valid reasons for this. True and honest risk assessments are hard to do. They are subjective. Risk to who? Risk in what way? Under what circumstances would something we consider to be low risk become high risk?

Risk is not a sure thing. It's the probability that something could go wrong. The more something is used, the higher the probability that something will go wrong. Not all risks are weighted the same. Some risks are trivial and easily shrugged off ( typos and cosmetic errors) because the "damage" is minor. Other risks are much more important, because the potential for damage is very high (an iFrame can open you up to cross server scripting attacks). Risks are identifiable. They may or may not happen, but you have a handle on how they could happen. 

Here's a process you can use. The next time that you sit down for a story workshop (or whatever you refer to an initial exploration of a new feature idea and implementation), take the time to ask the following questions:

- What would be the downside if we didn't deliver this feature?

- What would be potential problems that would prevent us from implementing this feature?

- What other areas of the code will this feature associate with?

- In what capacity could we run into big trouble if something isn't configured or coded correctly?

- What are the performance implications? Could this new feature cause a big spike in requests?

- Is there a way that this new feature could be exploited, and cause damage to our product or customers?

Yes, I see some of you yawning out there. This is a blinding flash of the obvious, right? We all do this already when we design tests. I used to think the same thing… until I realize how much we were missing at the end of a project and we opened it up to  much larger pool of participants. Then we saw issues that we hadn't really considered become big time bombs. We all started working on interrupts to fix and close the loops on areas that were now bigger issues. It's not that we hadn't tested (we had, and we did lots of testing), but we had placed too much of our focus on areas that were lower risk, and not enough focus on areas that were higher risk. Our test strategy had been feature based, instead of risk based.

A risk analysis should consider a variety of areas, such as:

- defects in the features themselves and customer reaction to those defects

- performance of the feature under high load and many concurrent users

- overall usability of the feature and the user experience

- how difficult will it be to make changes or adapt this feature based on feedback

- what could happen if this feature were to leak information that could allow a hacker to exploit it.

Each of these areas are risks, but they do not have the same weight. At any given time, these risks can change in weight based on the feature, platform and audience. Security may be a middle-weighted issue for an internal only app, but much more heavily weighted if it is a public facing app. Performance is always a risk, but have we considered certain times and volume (healthcare.gov being a perfect recent example)?

Additionally, identify features that are critical to the success of a project, their visibility to users, frequency of use, and whether or not there are multiple ways to accomplish a task or only one way. The more an audience depends on a feature, the greater the risk, and the more focused the testing needs to be. Likewise, if the item in question is infrequently used, is not visible to a large audience, and there are other avenues or workarounds, then the risk is lower, and the need for voluminous tests lessened. 

Ideally, we would spend most of our time in areas that are high risk, and very little time in areas with little to no risk. Making that small shift in thinking can radically alter the landscape of your test cases, test coverage and focus.


Also, we would like to prevent another insidious risk factor that can change this equation and balance, and that's time pressure. If we have to compress a schedule, we radically alter the risk profile. Issues that are relatively low risk given enough time to test become much higher risk when there is time pressure and a "death march" mentality to getting the product out the door.

Bottom Line:

Everyone's risks are going to be different. Every organization has different fears that keep them up at night. Making a check list to consider every potential risk would be pointless, but a framework that allows us to examine the risks that are most relevant to our product owners, company and customers will help us to set priorities that are relevant, and place our efforts where they will have the most potential impact. 

Again, risk isn't black and white. It might happen. We might do something that could cause us damage down the road. We might be performing lots of "cover our butt" tests that, really, have a very low likelihood of actually happening in the wild, while missing important areas that have a much higher chance of occurring. Shift the conversation. Move away from "how many tests have we performed" to "how many risks have we mitigated, and are we focused on mitigating the right risks?"

Product Review: Acronis True Image 2014

One of the fun things about writing a regular blog is that, over time, you get some interesting opportunities come your way. Many of them, for various reasons, I've not been able to follow up on, but every once in awhile, something either looks interesting, meets a need, or a part of me that might otherwise be reluctant says "oh what the heck, why not?"

It was in this guide that I received a request from Acronis Software and asked if I'd be interested in checking out and reviewing their newest product, Acronis True Image 2014. I figured "sure, why not?" If nothing else, it may be an interesting experience to try something new, and see what I could discover in the process.

I decided that this might be a fun experiment for a few reasons. First being the fact that I've been playing fast and loose with my PC. I used to be much more meticulous about things like anti-virus software, firewalls, backups, imaging, etc, but it's been awhile (since the MacBookPro became my main system). Still, there's a lot of stuff I do on the PC that I'd be rather upset if I were to completely lose ,so I considered this a timely and fortuitous call to action.

Installation was relatively quick, even on my now somewhat long in the tooth Windows 7 PC (a 2009 era Toshiba Satellite for those interested). It was with this well seasoned machine that I figured a more nuanced and interesting test could be conducted.

There's two levels to this application. The first is the stripped down, almost bare bones display of the most critical functions. For those who want to get in, do what they must and then get out, this is a nice setup. Clean lines, uncluttered, easy to navigate and understand. Ah, but what fun is a backup and restoration app if it doesn't give fine control to the one setting it up? If you are one easily smitten by such things... Adornis has you covered there, too.

Some nice features that True Image offers are as follows:

 

Complete Control of the level and Density of the backup.

If you want to create a very specific and cherry picked set of files that you need to back up, along with a set of partitions on your disk, you can do that. The control options are geared towards the simplest and quickest path, and if you want to "set it and forget it", it's very easy to do so. If you would like to go in and perform fine control and very specific file level interactions, you can do that as well.


Individual file selection from back-ups.

In addition to backing up either full disks or individual file level, the user has the options of performing a file level recovery from a disk partition, if that is desired. rather than have to go and pull down a full backup, the user can go in and select a single file and just pull that one option. Yes, this has been available for systems for years, but being able to do it this simply and this directly is new and, frankly, kind of nice.


Create bootable media that can interact with the True Image backups.

This option still requires a bit of futzing with the BIOS to set the boot order, but if you do, then you are just a USB stick or a CD/DVD away from getting your system up and restoring a True Image backup. again, the system allows for both "drop in simple" and "fine motor control" options, depending on how involved or detailed you choose to be.


Back up to the cloud.

Acronis gives the user a number of "DropBox" level options for online storage, and in this capacity, users of Acronis can utilize the True Image equivalent of Dropbox. While drop box create and drag options are cool and widespread, True Image has added the ability of pushing and pulling full backup images from the cloud. this way, if you'd like to get a dedicated hard drive out of the equation entirely, you can certainly do that. In addition, users can also use the sync functionality with multiple devices and keep all of their items up to date. Click "Publish" and the whole world can see it, or just those who you designate access to.

Some other interesting tidbits:

Try and Decide - Curious to see if an app will cause problems? try and decide lets you give it a run, and if it turns out to be more trouble than it's worth, you can roll it back.

Secure Zone - creates a special secure partition for your backups on your disk.

Boot Sequence Manager - Lets you boot your computer from a disk image if desired.

Backup Conversion - Move your backups from windows to Acronis and vice versa

File Shredder and Other Security Tools - exactly what it sounds like


Bottom Line:


The tool isn't magic, but as far as having a balance between ease of use and flexibility/control, Acronis hits more notes than it misses. Systems are of course going to be limited based on the hardware are features. My test runs were not what I would call "zippy", but I am perfectly aware of the fact that that may have more to do with my own system limitations and hardware than anything having to do with the Acronis application.

Also, I've only spent a few days with this app, and there's so much that I'm curious to play with and tweak with. I suppose the best endorsement I can give is that, after several days of poking and prodding, I feel like I want to know more and do more. In these days of short attention span theater, that says a lot.

tl;dr: nice app, simple to operate for the basics, but plenty of firepower to geek out with if users choose to. A nice balance overall.

Book Review: The Practice of Network Security Monitoring

This certainly fell into my lap at an opportune time. With the various revelations being made about the NSA and its tactics, as well as the upsurge in attention being paid to network and application security in general, this book was a welcome arrival in and of itself. 

There's a lot of attention paid to the "aftermath" of security breaches. We see a lot of books that talk about what to do after you've been hacked, or tools that can help determine if your application can be penetrated, along with tools and recommendations for performing that kind of testing. 

Less often asked (or covered) is "what can we do to see if people are actually trying to get into our network or applications in the first place?" While it's important to know how we got hacked, I'd like to see where we might get hacked, and sound an early warning to stop those hackers in their tracks.

To that end, Network Security Monitoring (NSM) makes a lot of sense, and an important line of defense. If the networks can be better monitored/protected, our servers are less likely to be hacked. We cannot prevent all breaches, but if we understand them and can react to them, we can make it harder for hackers to get to anything interesting or valuable. 

It's with this in mind that Richard Bejtlich has written "The Practice of Network Security Monitoring", and much of the advice in this book focuses on monitoring and protecting the network, rather than protecting end servers. The centerpiece of this book (at least from a user application standpoint) is the open source Security Onion (SO) NSM suite from Doug Burks. The descriptions and the examples provided (as well as numerous sample scripts in the back of the book) help the user get a good feel for the operations they could perform (and control) to collect network data, as well as how to analyze the collected data. 

The tools can be run from a single server, but to get the maximum benefit, a more expansive network topology would be helpful. I can appreciate that my ops people didn't quite want to see me "experiment" on a broader network for this book review. After reading it, though, they may be willing to give me the benefit of the doubt going forward ;).

There are lots of individual tools (graphical and command line) that can be used to help collect and analyze network traffic details. Since there are a variety of tools that can be used, the author casts a broad net. Each section and tool gets its own setup, and an explanation as to how to use them. The examples are straightforward and easy enough to follow to get a feel as to how they can be used.

The last part of the book puts these tools into action, and demonstrates examples as to how and where they can be used. The enterprise security cycle is emphasized (planning, resistance, detection, and response), with an emphasis on the  last two items. NSM uses its own process flow (collection, analysis, escalation, and resolution). By examining a variety of server side and client side compromises, and how those compromises can be detected and ultimately frustrated, we get a sense of the value and power of this model.

Bottom Line:

My approach to learning about NSM in general comes from being a software tester, and therefore I'm very interested in tools that I can learn and implement quickly. More important, though is the ability to apply a broad array of options. Since I don't really know what I may be called on to test, this varied model of NSM interests me greatly. From an understanding level, i.e. an ease of following along and seeing how it could work and where, I give the book high marks. I'm looking forward to when I can set up a broader and more varied network so I can try out some of the more expansive options. 

On the whole, "The Practice of Network Security Monitoring" gets the reader excited about getting deeper into the approach, and looking to where they can get more engaged. As tech books go, it's a pretty fun ride :).

When a Wish Falls into Your Lap

I AM THE PUMPKIN KING by ~ROFLSOLDIER

I cannot say the word "PumpKing" without this image

popping into my head ;).

This is a bit amusing, a bit serendipitous, and a whole lot of awesome all rolled into one. I don't typically talk about my company in this blog unless it's to make a general point or to highlight something I find interesting, and today's comments fall squarely into the latter.


For quite awhile now,  I've been hoping for a window of opportunity to look at and open up some venues of more technical engagement for both myself and the software testing team, in ways that go beyond writing automation scripts. One of those opportunities literally fell into our laps a couple of weeks ago.


My company has a long running and enterprise level product that was originally written in Perl. As part of the development process, over the years, we have had a movable role of build manager, code health manager, deployment specialist, front line troubleshooter, and whip cracker all rolled into one. The term (that has a lot of history in the Perl community) for this role is the "PumpKing" (holder of the pumpkin, they who keeps the system afloat, the puller of the strings, themz who pulls the taps to keep the good stuff flowing, call it what you will).

The responsibility of PumpKing has been handed off week after week to the various programmers in a round-robin fashion. A few weeks ago, I made an aside about one of the programmers (who I happen to be good friends with) not living up to their responsibility as "PumpKing". This had to do with running stand-up that morning, i.e. it starting late, and my comment was totally in jest. Their response was (and with a smile, I might add):


"Well, Michael, if you would like to get into the PumpKing rotation, we can certainly arrange that!"


I chuckled, thinking the comment was likewise in jest.


It wasn't.


At our next Engineering team meeting, said programmer made a motion that "all software testers should join the PumpKing rotation". I snickered again, thinking this was follow-up on the joke. What I was not prepared for was when our VP of Engineering said "I think that's a great idea!"


After I did my double take, and realized that I was not being punked, I stopped and thought about what an awesome opportunity had just been handed to us. Why is this an awesome opportunity? Because in one fell swoop, the potential number of "active duty build and deployment engineers" just went up considerably. For a group of software testers looking for an excuse/opportunity for more technical engagement with the development team, we were just given the keys to a gold mine.


Needless to say, there's a lot of learning to be done so that we can do end to end builds, merges, CI, testing, deployment to various environments and interacting with Ops to make the final pushes to production. Additionally, the PumpKing runs stand-up during their week, is on call for issues, interacts with front line support for rapid response of issues, etc. It's kind of a big deal!


I'm excited because this is a real opportunity for those of us that want to have an avenue to better understanding of the technical underpinnings to really get into it. We have the potential to do hot fixes if needed, we are responsible for pushes in our development and staging environments, as well as with pushes to production. It's visible, it has the potential to get wild, and it's a great way to get knee deep in the muck and grime of the real code base and understand how everything fits together. In short, it's a dream come true for a software tester wishing there were some way to more effectively blur the lines between programmer and tester.


These past couple of weeks have been all about learning the ropes, practicing along with the current PumpKing, and getting ready for my turn in the saddle... which starts this coming Monday morning and extends to the following Monday morning.


Let the festivities begin ;).

What's the Risk? Weekend Testing Americas Returns August 10, 2013

Again, I figured some of you would like to have a break from the barrage of 99 Things posts, and what better way than to announce our next Weekend Testing Americas session.

First, the formalities:


Weekend Testing Americas - Session 42

Date: Saturday, Aug 10, 2013

Time: 12 p.m. - 2 p.m. (Eastern Daylight Time).

World Time: http://www.timeanddate.com/worldclock/fixedtime.html?msg=Weekend+Testers+Americas+%2342&iso=20130810T12&p1=179&ah=2

Facilitator: JeanAnn Harrison


Risk Based Testing for Mobile Application

All will need to download onto their mobile device a mobile email application. You can use either Yahoo Email app or Gmail app. Set up an email account if you do not already have one. Be sure to have notifications set up to be sent to phone when a new email is sent.


Agenda:


1) Brief discussion of risk.

2) Group participates in creating a risk based plan/approach for testing

3) Group implements testing (we can split up into groups to cover more testing)

4) Debrief: includes did we follow our risk based testing plan, how much test coverage do we feel we achieved


Logistics:


Add "weekendtestersamericas" to your Skype contact list if you have not done so already.


About 20 minutes prior to the start of the session, contact us on Skype and let us know you would like to participate.


Please let us know if you have any questions.

Keep Your Eye on the Ball (The End Goal): 99 Ways Workshop #14

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions.

Suggestion #14: Keep Your Eye on the Ball (The End Goal) - Kate Paulk

What is the point of software testing? Why do we do it? In the simplest sense, as I've said previously, software testing is about providing information about the state of a product or project. It provides guidance to all parties in an organization as to the quality of an application and its functionality. It illuminates risks to value. It highlights areas that may need to be improved. All of these require a variety of skills and attributes, many of which would go well beyond a specific "do this one thing and you'll get better" approach.

For this section, I'm going to focus on the statement "Keep your eye on the ball" as it pertains to software testing in general. Very often, we find ourselves looking at a piece of functionality. As we are asking questions and exploring, we can veer off into directions that, while interesting, can easily get us off track. Don't get me wrong, exploration is part of the job. It's often the most fun part of software testing. Keeping a focus on the areas that matter, at least for that given time, is a skill that all testers should develop.

Workshop #14: Session Based Test Management (SBTM) and Thread Based Test Management (TBTM)

Many testers have heard the term "Session Based Test Management" (SBTM) at one point or another. If you have participated in Weekend Testing, you have seen first hand how SBTM is applied. SBTM is a structured and focused session of exploration. The goal is to make sure that you know where you plan to go, and record what you do along the way. Jonathan Bach has written an excellent overview of this approach. 

The main idea behind SBTM is that you set a time to focus on something specific within an application. The time is pre-determined. I prefer sessions to be short, roughly an hour in total duration, though they can be longer or shorter as needed. I also prefer to create very specific missions and charters for these sessions. A mission is the overall goal of the session, and the charter(s) are specific tasks I want to accomplish within the scope of that mission. The more specific the mission and charters, the better my chances of successfully completing them within the given time frame. 

Taking notes or capturing what I am doing is important. This can range anywhere from just having a simple text editing application open in the corner while I test, using a dedicated SBTM tool like Rapid Reporter to gather my approach and findings, to actually recording a screen cast of my testing and capturing both my exploration and my commentary of what I am doing. How the session is structured is up to you, but structure the session, take notes of your exploration and your findings, and be sure that you can explain what you did and what you found along the way.

What's this "Thread Based Test Management" (TBTM) think I am referring to? It likewise has its roots in STBM, but think of it as a particular tangent you might take if you were to get involved in while testing. Have you found yourself talking with someone and, as you are discussing a particular area, it would lead you down a path that you might not have considered at the outset? Sometimes those paths are short, and they provide good information to the main topic. Sometimes those paths are much longer, and they seem to take you away from the main point of the discussion, but they are still valuable. What do you do when you find yourself exploring a path that fits your mission or charter, but opens up into areas that you might consider to be too broad or too long for your session. Do you terminate it? Do you go back to the main mission and charters? Using TBTM, you note that you are following a thread, and you see where it takes you. Again, Jonathan Bach has written about his own experiences with using TBTM here.

On a personal level, I like to use the TBTM approach and apply the "rule of three" when it comes to specific threads. If I find myself making more than three steps away from the main mission and charter I am working on, I note that this may be a thread worth exploring later on, but I try to not "rabbit hole" on a given area if I see it's taking me out of the area I've scoped for that particular session. Rabbit holes do, however, offer some great exploration opportunities, so we don't want to forget them. When I find myself taking three or more steps away from the main mission and charter, that's a good indication to me that that thread probably deserves its own session, its own mission, its own charter and dedicated time to focus on it. 

ADDENDUM: This was written some time ago and the state of the tooling world has changed a bit. I was asked if I would include the following list of up[dated tools, so if you are seeing this post in 2022 or later, first off, thank you for reviewing my earlier work :) but also, here's a listing of 10 Additional Tools that you can use to support your Exploratory Testing efforts (happy to see Rapid Reporter is among the list :) ). 

Bottom Line:

Both Session Based Test Management and Thread Based Test Management offer opportunities to structure or explorations, and reference how we got to key areas in applications s that we can communicate what we found, how we found it, and why it maters to the stakeholders. Above all, it provides a way to keep focus while we are testing. Exploratory testing is often seen as mesh and unstructured, a lark that happens to sometimes find something interesting. Those of us who use SBTM and TBTM approaches know that we can provide a great deal of concrete evidence of where we have tested, what we have learned and how that learning can benefit the team that is developing an application.

The key part of Exploratory Testing is that we learn along the way, and we adapt and focus our efforts base on what we learn. Rather than formulating our plan up front and slavishly following a list of scripts and to-do items. We loosely develop missions and charters, and then try out our ideas based on those missions and charters. By keeping track of the threads that we follow, we can determine which areas might require follow up and additional testing.

Understand the Business Model and Business Challenges: 99 Ways Workshop #13

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions.

Suggestion #13: Understand the business model and business challenges/context before determining the testing challenges. Its not all clicks and buttons. - Mohan Panguluri

Over the years, I have worked for a variety of companies and with a variety of organizations. Each has had different goals and needs, and have been part of different spaces and industries. During my software testing career, I have worked for a very large networking company (and during my tenure watched it go from a relatively small networking company to one of the world's biggest), a company that made virtualization software, a company that specialized in touch devices and peripherals, a video game publisher, an immigration law software application developer, a television aggregator, and a maker of social collaboration tools. 

Were I to have approached my testing in each of those places in a "cookie cutter" way, there is no way I would have been successful across the board (it's likely I might not have been successful at all). Each company made decisions that were unique to its environment, and each worked to woo new customers, as well as work to keep current customers happy. Taking the time to get to know those customers and what they want is always a good investment. It also helps guide you on what areas are really critical.

Workshop #13: Take a "Gemba Walk" with Someone in Sales and Support

For those not familiar with the term, Gemba is a Japanese word and it means "the real place". It is used in connotation of "the scene of the crime" or "live from this spot" or "the place of interest", but they all annotate the same thing; here's where the action is. If you really want to understand the business model and what's important, you need to go where the action is. In most companies, the "action" is in Sales and Support. They deal with both sides of the business. First, the sales people are the ones selling. They have the ear of both the current and potential customers, and they know very well what is bringing people to part with their money, as well as what is needed to keep them happy and active customers. Second, the customer support people know intimately well what is causing customers discomfort. They also know which issues are tolerable, and which ones are deal breakers.

Taking a "Gemba Walk" means to go and explore what's going on in these critical areas. See if you can spend part of a day with the people in sales (preferably multiple people, if possible). Be there to observe, listen in on calls, take notes, and see where your learning takes you. Don't go in with set expectations or an idea of what you want to discover. Additionally, spend some time as well with the support team (multiple people, if possible). Sit in on calls if you can. Take notes of what you experience. Like with Sales, do not go in with predetermined questions or expectations. Be open to what you learn,

After taking these walks go back and read what your documentation, marketing literature and company website have to say. If the materials you have support what you have seen on your Gemba walk, that's a good place to be. If you're marketing and website materials are not reflected in the Gemba walk, I recommend you give priority to what you see and hear on the Gemba walk. The marketing and website literature may be idealized. The Gemba Walk will reflect reality.

With the insights these sessions give you, take a look at your overall testing strategy. Do your testing efforts align with the reality of your business model, the *REAL* business model? Do you understand what the real goals and the real risks of your organization are? Based on what you have learned from Sales and Support, what would you want to change and do differently with regard to your testing?

Bottom Line:

Companies tend to put their best foot forward when marketing themselves. They also tend to cherry pick the feedback they get from their customers and share with the public. They may share a bit more with the development team and with the testing team, but we should not be surprised when they don't go out of their way to share more than the peaks of the highs and lows. Many important and under served areas often go unrecognized because they are in that indeterminable "middle" area. Those issues in isolation may not rise up to be seen as threats. When taken together, and examined with a different perspective, patterns may emerge that can highlight risks that we don't consider. Just because they don't rise to the level of "urgent", doesn't mean they aren't important.

Sales and Support have vital information, but they may not be sharing it because no one has really expressed interest in understanding it in a way beyond quarterly targets and trouble ticket resolutions. These are both areas where a keen tester could learn a lot, and possibly cause them to dramatically change where to focus time and attention.

Question the Way You Work Everyday: 99 Ways Workshop #6

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.


My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions.

Suggestion #6: Question the Way You Work Everyday - Kinofrost

Sometimes, it's easy for us to think that things are set in stone, that we cannot really change things We don't have a voice or the authority to change up what we do. Generally speaking, unless you are an inmate in a penitentiary, that is not true. You can do things to change the way that you work and the way that you are perceived in your workspace. What is usually the case is that you (and me), like many others, are waiting to be given permission to make a change. Stop it! You don't have to ask permission. Asking permission is anathema to being a tester ;).

Workshop #6: Add Some Disorder To Your Day

Pick a project you are working on. Consider its protocols, its "essential elements", its mandated deliverables, whatever process exists to do what you need to do to complete the job as it's defined. Got that? Cool. Now, for one day, look at every single protocol and rule, and try to subvert it. I don't mean break the law, I mean try to see if you can either change up the way you do things, or, if you believe you can be effective in chucking something entirely, do so… but make notes of what you do in their stead.

For many years, I had a spreadsheet, and that spreadsheet was an ironclad "do this, do this, observe this, mark yes or no". It was a totally by the numbers thing. I had to turn this sheet in every day, and it was reviewed and commented on by the lead tester. It was efficient. It was accurate. It was also soulless and joyless. One day, I decided to simply call a personal mutiny. I decided to drop the spreadsheet for a day and see if, by ranging into different places and ignoring the set script, I could discover something interesting. Often, I did, and often, the things I discovered actually were game changers. I would then go through and look at the "canned script" and see what areas I covered and give the yes or no, and sure enough, what I found was often not part of the specified test cases, and would not have been found by the set test cases. 

Occasionally, I'd be told "that's out of scope, do it by the book", but over time, as I found other interesting oddities, and continued to do so, I was able to demonstrate that some "exploration" was more valuable than following the mandatory checks (note: at this point in my career, I don't think that the term "exploratory testing" was known by many, but that's definitely what I was doing).  

Think about the way that you write bug reports. If you walked in blind, would you be able to tell what's happening? Could you recreate the issue described? If not, how could you change what you write to be more effective?

Have you ever thought to yourself "This would be a cool tool to use to (fill in the blank), but our infrastructure is so different, I don't see a good way to integrate that into our environment"?

Interestingly, the last one was something I was discussing with our development team today. I was telling them that I was exploring some open source tools for doing security testing and load/performance testing. The problem was that the tools used different languages than we were using. I was afraid that any tests I created would exist on their own little islands, so maybe they weren't worth the time to dig deeper. I just figured that that was the case, so it didn't make much sense for me to talk about it further. I did bring it up with my Director, though, and mentioned that I thought they might be novel things to look at. 

During our group meeting, said Director brought up the tools I was talking about, and my perceived "shortcomings" as far as them working in our environment. One of our developers said "Hmmm… that might not be so hard to implement after all. Since we use Jenkins to do a lot of our CI and functional tests, Jenkins can scaffold just about anything. Let's take a lot later and see if we could work those tools in". After I showed him what I was considering, he said "Oh yeah, that's no so out there. We could definitely explore this!"

Bottom Line:

As testers, we are trained to ask tough questions of the products we interact with, and the methods and tools we use. We should be just as willing to ask the tough questions when we "test" our own organizations. If something seems odd, out of place, wasteful, or less than helpful, we should be willing to experiment and see if we can do better with something else. Perhaps it really is the best way, but I'm willing to bet, a lot of the time, thinking differently and approaching situations with a different point of view will prove to be very enlightening, both to you and the organization. There is, of course  a danger if there is politics or status associated with some decisions and policies (and a sense of losing face for an individual if your changes are implemented), but even then, it's possible to start changing hearts and minds a little at a time. Be willing to "think different", but be prepared to show how your different thinking is better than what's already in place. It's possible you won't be able to… then again, maybe you will ;).

Learn When to Automate and When Not To: 99 Ways Workshop #5

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions.

 
Suggestion #5: Learn When to Automate and When Not To -- Christopher George

Automation… I figured we'd get to this topic soon enough.

I'm going to say it, and if this frustrates anyone, I apologize. If you want to become good at software test automation, you will have to become good at software programming. That's it. Where you do that programming is largely left up to need, purpose and application, but there's no way around it. Good automation requires good programming skills, and consistent use of them. Abstraction frameworks help some, and may ease the way in for some testers to help in the automation process, but those frameworks will only carry you so far. If they break, need service, or otherwise don't meet the actual need, then what will you do? The likelihood is that you will have to grab a programmer and have them help you fill in the blanks. There is nothing wrong with that, but it's a reality too few people understand, and too few tool vendors ever mention.

Now, having gotten that little downer out of the way... what I really want to do is reset the discussion at this stage. For our purposes, let's drop the phrase "automation" because it's too broad a description and it covers way too much to be a practical term for this workshop. Instead, let's bend the vocabulary a bit and use a different term… computer aided testing. Every time we do anything with a computer system, we are using computer aided testing. Any time we determine there's a pattern that we can repeat and, by bundling the steps together and running them in sequence, we can either save time or prevent human error, at its most elementary level, that is automation, and that is something anyone can start doing.

Workshop #5: Focusing on Patterns of Repetition

If you are working with a Linux/UNIX based system, this is something that you are probably already somewhat familiar with. The whole point of the UNIX paradigm is that utilities take work from one process and direct them to another process. This pipelining, redirecting, copying, sorting and trimming is at the core of most one-off shell scripts that are written. I've written one-off scripts for many years, but didn't stop to really consider what I was doing until relatively recently. Many software testers have similar steps that need to be performed. We start environments, we get them to a particular state, we pose questions that we hope will yield certain answers and if we see what we expect to see, we break down the environment or just exit the shell.

Chances are, many of the steps that are performed to do this are the same every time that you do them. If you can identify those steps, and the sequence those steps are performed, then you have identified a critical area where "computer aided testing" can be of benefit to you. Likewise, if at the end of a test run, you have a large amount of data that you need to go through, parse and cut out key areas to look at more closely, that is also another area where "computer aided testing" can be exceptionally helpful.

These areas are not glamorous. They are not what we traditionally consider to be "automation". They certainly aren't what we normally look at as being "automated testing". Still,  everyone has to start somewhere, and if you need to automate anything, I would strongly suggest automating those steps that you know will always, or at least a great percentage of the time, act the same way.

Another area that frequently comes into play is checking input fields. Much of the time, we know in advance what values we want to use, or a representation of what values we want to use. Filling in those fields manually is murder, but there are small tools that you can use that can ease this process considerably. On Mac, I use a tool called aText. On PC, a longtime standby for this purpose is Texter. Both work in similar ways; they allow you to create shortcut keys so that you can input text in predictable order. Both tools support tabbing, so that you can fill in forms directly. Again, this isn't classic hands off automation, but they streamline and simplify many time intensive steps, and that's often a very good thing.

Many people want automate their browser interactions, and for that purpose, using direct tools like Selenium IDE is a "gateway drug" to test automation. While I think Selenium IDE has a limited benefit, it can be helpful to do certain things, and it's a good way to get started with repeatable testing at the UI level. The disadvantages are also many, and there are plenty of blog posts out there that explain exactly why.

Most important, though, is taking the time to stop and think about what the actual problem is. As I wrote on the SmartBear Software Quality Matters blog, when we focus too much attention on a tool (or a need to automate) we often miss the obvious point, which is "what is the problem that we want to solve?" A tester might want to create a data driven test that walks through and adds 100 users to a system using the front end form. If the point of the test is making sure that the form handles a variety of input parameters effectively, then yes, a front end, data driven test might make sense. If the goal is to populate the database with 100 entries, then a much more effective approach (and likely less error prone) would be to create a shell script that inserts records to your test machine's SQL database (or the appropriate nomenclature for your system of choice).

Bottom Line:

For anyone looking to take their first steps into automation, start small, but start with what will give you the biggest immediate benefit. Make it a habit to become "more lazy". Find the tools that will allow you to repeat the repeatable, fill the fillable, and click the clickable. Once you get a taste of how to do that, then you may find you have an appetite to get deeper into more traditional automation frameworks and approaches. As you make that transition, you will need to boost your programming chops to meet the skill needs, but as you do that, you will also be getting better at recognizing real duplication patterns, and the more familiar you are with what those patterns are, the more effective you will be at making automation tests that will serve your needs.

Cascading Fail: A Crash That Hits Close to Home

As many of you are aware, there was an airliner coming from Seoul, Korea that crashed upon landing at San Francisco International Airport Saturday, July 6, 2013. Much has been said in the news about the crash, and much will probably be said in the following days and weeks. My point for this post is not  to talk about the crash, or the myriad of systems that were or were not available. It doesn't change the fact that an airline crashed, scores of people were injured, two girls were killed, and a major International Airport was brought to a grinding halt while the events that followed played out.

This crash had a direct impact on me, though, in two ways. First, it is the cause of my daughter, who has spent the last eight days in Japan, being unable to come home yet. Because of the shut down of the airport, many flights had to be diverted to smaller airports, which were quickly overwhelmed by the increased load. Many flights that were scheduled were cancelled, my daughter's included. Thankfully, due to some herculean and determined efforts by the team of chaperones, as well as the good will and kindness of the city of Narita, Japan, they were able to weather the hiccup. Note, this "hiccup" meant that their return was delayed by three days, including, at the current time, a rerouting through Seattle and an almost 24 hour layover.

The second way that it impacted me is knowing that, for the two girls who were confirmed dead, both were students coming over on an exchange trip from China. Both were mid teenagers. In other words, both were mirroring my own daughter's trip. I am a realist, and I understand Black Swan events, and the likelihood of a repeat performance is way less than a million to one, but that doesn't calm a father who now has new uncertainties and anxieties. Needless to say, it was a little too close to home.

Note, I'm not blaming anyone for this, but in the tester's world that I inhabit, rarely is there such a thing as a "isolated problem". Usually, when something goes wrong, it affects entire systems, and those systems also affect entire systems. The net results of an error could cascade out and cause devastating problems, and the after effects not seen until well after this issue has occurred. Yes, I am one of those people who can see a testing story in everything, but today I am doubly reminded of how early mistakes not caught can ripple out, and we really have no way of knowing just how far the problems  can cascade. We may have a momentary interest, as long as the issue doesn't affect us. Once it does, though, we start to see a much broader world of issues and problems. A good reminder to me in my workaday world to try to find problems early, while the course correction options are much wider and still possible.

Aedificamus: The Defining Point

For some of my readers, this post may seem all over the map. There is a very good reason for it.

Last Thursday, I went in for what I hope will be one of the last chapters of what has proven to be a rather long ordeal. I had surgery on Thursday, June 20, 2013, to remove the steel plate and pins/screws that were in my right tibia. These were remnants of an injury I suffered on August 29, 2011, when I broke clean through both bones in my lower right leg.

The past two years have been times of incredible highs and smashing lows. From the excitement of having my first full talk accepted for a major conference, to having to send my regrets to that conference because I would be flat on my back for the next month and physically unsuited to attending.  That month was one of the longest I ever went through, but it was also cathartic. It was the month I worked through several books. It was the month I spent doing all I could to make "Mordor", a smoke testing framework written in Cucumber, RSpec and Capybara (with a lot of help from Sidereel developers) something that was actually effective and reliable. It was a period where I realized that the world didn't stop turning because I wasn't running 100 miles an hour to do everything I needed/wanted/had to do. I had to rely on others, and trust they would do what had to get done. In most cases, they got done.

Sometimes, we don't know what we miss until we can't do it any longer. I came to the conclusion that I missed running. Note, I wasn't much of a "runner" before, but I took for granted that, if I wanted to go for a run, or play a game of Frisbee or touch football with some friends, I could. Then this accident happened, and even after I healed, my ability to run was greatly compromised. The orthopedic surgeon was impressed with my range of motion, was glad that I could walk for long distances, and looked to him otherwise vigorous and strong. Still, I couldn't run. I could jog at a slow clip, but if I tried to do a full out run, I looked like a drunk man staggering. I lost 25 pounds after the accident, but since I was inactive, much that weight loss came from losing muscle. Sadly, when I gained 35 pounds back, the majority of it wasn't muscle, either. I'm now, like it or not, the heaviest I've ever been (256 pounds). Granted, when you're almost 6'3", that may not seem like much, but for a guy that was used to being a vigorous, active, and fairly solid 225 pounds for the bulk of his adult life, this has been frustrating.

Thursday, June 20, 2013, was a defining and pivotal point in my life. The stainless steel is now gone. The "zipper", i.e the long scar and staples in my leg, has made a return engagement, and I have to do all I can to keep it covered, clean, and elevated. My lymphatic system is still recovering, and my circulatory system, by association, still has a way to go. Also, my tibia now has about 14 holes of various lengths extending from just above my ankle to about 40% up the length of the bone. As a friend pointed out, those holes make my leg, for now, just like the perforations on a saltine cracker. Overall, the bone is solid, but if I hit it just the right way, until those holes fill in with new bone, I run a risk of breakage and going back to square one. The good news? That should take, maybe, six or eight weeks, and then my bone should be "good as new". No more plate, no more irritation and threats of infection, no more torsion modulus if I happen to mis-step or flex my leg quickly. In short, with the exception of a big scar, two remodeling bones, and some nerve damage that may never recover, I have a choice to rehab and get back into my preferred "fighting shape".

I've decided I'm going to share that journey with everyone, or at least those who want to follow along. You will notice a tab that is, for now, mostly empty. That tab says Aedificamus. Much like my "Practicum" is where I practice my craft of testing in various ways, Aedificamus is where I will test and refine the theories needed to rebuild the most important project I have... ME :).

I want to rebuild. I want to get stronger. Most important, I want to run again, and I want to do this in a way that is meaningful to me. This could become the "boldest boast" of my career, maybe even my life. It could work out splendidly. It could go down in flames.

There's only one way to find out ;).

For Sanity's Sake, Allow For Slack

I had booked my flight weeks ago, had the time frame dialed in just right, and at the last minute, an opportunity presented itself. A number of the participants at STPCON decided to rent a box at Petco Park (where the San Diego Padres play). My flight came in at 5:30 p.m., the game started at 7:00 p.m., with pre-festivities starting at 6:00 p.m. If everything worked out right, I should be able to make that.

I think you all know where this is going, right ;)?

Well, as I'm sitting here, waiting for the flight, I have just heard over the loudspeakers "we are sorry for the inconvenience, but the flight to San Diego has been delayed by..." well, it's not really relevant how much. The point is, the game will go on without me there. Ah, well!

Last year, I did my best to book flights to get me to New Orleans, and likewise, that flight got delayed. This delay, however, was a much more troublesome one. I actually missed my connection flight, and no other flights would be available until the following morning. Net result was that I slept on the floor of the airport in Houston to get on the first flight out the next morning.

Both situations suffer from the same issue... planning without allowing for slack. We want to believe that everything will go well, that everything will work in our favor, and that we will be able to do what we need to so that we can get to our respective places, do what we want to do, or otherwise meet some goal. The simple fact is that, in our very regulated world where things are designed to run like clockwork, when things go wrong they tend to go "very wrong".  If we plan for things to go very wrong all the time, we're not going to get anything done, but if we plan for a little slack time in the things that we do,  then we find that we can absorb many of the issues that come our way, with much less need to to have to alter plans when things don't go our way.

As for my plans for tonight, I've already sent the news along, so if you hadn't signed on for the Padres game tonight, and you are at STPCON, and now wish you could go, have a chat at the registration desk and say "Michael Larsen can't make it, can I get his ticket?"

Updated: turns out there's a wait list for tickets, so it'll be claimed for sure by someone. I'll see you all as soon as I get there ;).

A Different Stress Test: My Wife as Tour Guide

This may seem an odd title, but as you all know, I can find ways of making a testing allegory to just about anything, if given a chance.

Last weekend, my wife, Christina,  and I decided to take a weekend away. Our kids are old enough now that they can look after themselves for a night, and we wanted to celebrate our 20th anniversary (albeit delayed) in a more "festive" location and manner, so early Saturday morning, we boarded a plane, and took a short flight to Las Vegas.

First, a couple of things to know about us. Neither of us drink, we're not "partiers", and neither of us gambles or frequents strip clubs, etc. Some might say that, with that kind of a list (or anti-list), Las Vegas would be the most boring place on Earth. Not so. Christina considers Las Vegas to be a second home, especially the Strip with all its glitzy hotels, resorts and amenities available for all ages. She's kind of an expert on it. Ask her to point out the ten best shows on the strip and she can tell you without hesitation. Best restaurants? She can tell you dozens off the top of her head. Architectural designs unique to each hotel? Yep, she knows all of them and has combed just about every inch of every hotel from Mandalay Bay to the Stratosphere (there's a few she hasn't been to yet, but rest assured, she'll get to them soon enough :) ).

My point is, if anyone wanted to consult a "domain expert" on the places to see and things to do in Las Vegas that didn't revolve around drinking, gambling, or strip clubs, Christina is it. Yet even with that, we discovered that you can throw curve balls even to the best of experts and make them have to adapt.

A little context might help. Christina and her mom have been going to Las Vegas once or twice yearly as long as I have known them. I think this tradition started when Christina was a teenager, and without fail, at least once a year (sometimes twice), exploratory trips are planned, conducted, and travelogues of sorts are reported. Typically, they go mid week and they go "off season", usually meaning February or October. In that situation, they have plenty of time to go where they want to go, do what they want to do, and see what they want to see. For our trip, however, timing just happened to be that we were going to go on a weekend, and in particular, St. Patrick's Day weekend.

The net result was that we discovered the stress test that was getting from our hotel (at the Palazzo) to our dinner reservation (at THEMix at THEHotel at Mandalay Bay) and then to our show (Blue Man Group at Monte Carlo) and then negotiate how to get back to the Palazzo.

First, a bit about the Las Vegas Strip. If you look at it from a map, it looks like everything is close together. In reality, it's not. The hotels are just so big that they look like they are next door. The distance from Mandala Bay to the Stratosphere (the traditional boundaries of "The Strip") is about six miles. To get from place to place, cabs are important. On this night, we discovered that the queue to get a cab from the Palazzo was about 300 people thick. WOW! Not something we had anticipated. We did get our cab, and we did get to our dinner reservation... at which we had to wait a bit to be seated, and wait even longer to get our dinner (almost an hour... was it because I ordered Wagyu?).

From there, we made our way down and, caught another cab to get us to the Monte Carlo for our show (with a little time to spare, but not much). After the Blue Man Group Show (which was amazing, btw, and totally worthy of its own post :) ), we walked out of the hotel hoping to catch a cab... and saw the queue was triple the size it was at the Palazzo. Doing the math and realizing how long it would take, we decided to walk and have Christina play "tour guide" to me and show me her favorite spots. However, even this was stymied because of the sheer volume of people everywhere we went. Just moving from place to place was like salmon swimming upstream. Even with all of that, it was a lot of fun watching Christina, even under pressure, modify her approach and where we would go, and the angles she'd choose to take to avoid the crowds when possible. In short, she could adapt quickly because she knew just about every angle of the Las Vegas Strip there was. Were I to try to figure it out on my own, well, we probably would have made it back, but a lot later, I'd bet.

So what's my testing analogy? We may think we know an area, and we may think we understand how something works, but we don't really know how well or what parameters we know until we put some stress on the system. The fact is, pathways that are well known and easy to navigate may become considerably less so when our environment changes, and on this night, changes radically. If you have done your homework in advance (in this case, about two decades worth) you can adapt and move more freely. Likewise, when a system is under stress, it's vital we know as many avenues to get around as possible to allow us to see where people might go and interact with. Quiet times are good times to explore and find them, but rest assured, under load and stress, even quiet areas of an app will take on more load and pressure, and the results may be less than ideal.

One bright point... the odds of you running into several revelers puking in the bushes... probably higher in Las Vegas the night before St. Patrick's Day than in your app... but you never know ;).