OK, now that my current VP has had a chance to brief the troops of the situation, I can stop being cagey. I pledged to let him decide how he wanted to break the news and when, and since he decided to tell everyone today, I can now talk a little more, too :).
So what would prompt someone to go for a very public and much focused refresh and reboot of his web skills if it were not the prospects of a new job? Well, those who speculated that this was due to a new job were exactly right! I gave Tracker, my current company, 40 days notice (basically six weeks), enough time I felt to help them get the word out to get a new QA Engineer(s) to take my place. On January 17th, 2010, I will be starting at SideReel, a company that makes tools to aggregate and track Internet television properties.
It’s a fun company with a fun product, but unlike the more business and Legal/HR focus of Tracker Corp, SideReel has a lot more of a focus on the new and flashy web components that are out there, much more befitting of a company where entertainment is their primary market. When I made the decision to sign on with them, I asked them specifically what they recommended I make an effort to focus on before I got there. Their answer, unequivocally, was “learn all you can about cutting edge web components and technologies, because we use most, if not all of them!”
In the days of old, when I had a question about web technology and languages, my first point of reference was W3Schools, and I’m happy to say that it has enough details and information to still be one of my key points of reference, especially when it comes to HTML, HTTP, CSS, AJAX, JQUERY, JavaScript and any other number of components. One of the cools things about the W3Schools interface is that it allows the user to plug in the styles or code they want to experiment with and see it in action on the page side by side with the code they are modifying. A very nice service and definitely worth a visit.
To add to this mix, how timely was it to see posted to Twitter a listing of ten Firefox plug-ins, a couple of which I was already familiar with, such as Firebug and IE Tab, but also several I’d never worked with before, including ScribeFire, a blog editing tool, and Window Resizer, which allows the user to try out a bunch of different resolutions to view a page without having to modify the screen resolution (having a requirement to test Firefox 2.0 at 800x600 for an application under test right now has me really appreciating this little utility).
So that’s a little bit more, but it’s time to get my fingers dirty with Ruby, so that will be my focus tomorrow. Stay tuned :).
Friday, December 10, 2010
TWiST #23 with Mark Crowther
Is it noticeable? The new mic went into effect for this episode, so for the first time in 15 shows I had to re-record the intro and outro. Felt a little jarring at first; I’d gotten used to the other intros and their cadence. I also have to adjust to the sensitivity of the standalone mic; it’s much more live, so every pop, click and exhale is amplified and articulated way more distinctly. I even pulled out my old Pop Screen attachment I used to use when I recorded vocals in my home studio. It proved to be very helpful.
For today’s episode, Matt talks with Marc Crowther, a QA/Test Manager and tester who has had experiences in both the software side of things and the actual manufacturing side. Much of the conversation was based around software development and software testing trying to shoe-horn standardization and auditing tools designed for manufacturing into the development and test space, and how that often becomes a frustrating path (and perhaps not even a very lucrative one for time and energy invested). I liked his discussion at the end of the piece about decreasing unnecessary and repetitive artifacts, and I found myself really appreciating his perspective on this. If you’d like to listen along, please go here to listen to Episode 23.
Standard disclaimer:
Each TWiST podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to the podcasts and the entire library whenever you want to :). In addition, Pro membership allows you to access and download to the entire archive of Software Test and Quality Assurance Magazine, and its issues under its former name, Software Test and Performance.
TWiST-Plus is all extra material, and as such is not hosted behind STP’s site model. There is no limitation to accessing TWiST-Plus material, just click the link to download and listen.
Again, my thanks to STP for hosting the podcasts and storing the archive. We hope you enjoy listening to them as much as we enjoy making them :).
Thursday, December 9, 2010
Day 2 of 40: TESTHEAD BOOT CAMP: Rails, Firebug, and IE Tab
Here's my day 2 report for TESTHEAD BOOT CAMP. I just realised that my TESTHEAD BOOK CLUB and TESTHEAD BOOT CAMP posts are going to look really cluttered if they bunch up together, so I'm changing the format of the BOOT CAMP posts. With that, let's talk about today...
It figures, I announce I’m going to do something dramatic, and what happens? Mother Nature gives me a back hand! I woke up on Thursday feeling like someone shoved a cork in my nose and swabbed my throat with sandpaper. In short, not feeling good at all, but with a ticking countdown, I couldn’t just do nothing.
Day 2 of boot camp consisted of me de-cluttering, de-junking and installing tools that I hope will get me towards my goal of “supa-fly web test ninja”… hey, I don’t have a better phrase at the moment, but I’m of course open to suggestions :). Seriously though, taking the time to de-clutter and de-junk a work area, room or system is imperative; it sets the stage and tells your body and mind that you are serious and going for broke here!
So what was the first job I decided to tackle? Get Ruby On Rails running on my laptop. Since I am running Windows 7, that meant I had to see what steps had to be jumped through to get the software installed. As is often the case with applications and environments not necessarily developed for Windows, I expected there’d be a few steps that would be bulky and require me to dig around and do modifications. I was wrong. The Ruby installer worked right out of the box as did the Development Kit, and from there the steps to install the necessary Ruby Gems, install Rails, install SQLite (just to keep things relatively easy; I’ll be playing with MYSQL a little later).
So what’s my first share item here? For starters, my tip of the hat to the good people who have created and maintain the Rails Wiki and their steps to making a darn near painless experience for installing on Windows. The steps are relatively straightforward and describe with the needed details without becoming overwhelming or mired in minutiae the important steps:
Installing Ruby and RubyGems
Installing Rails
Installing your Database engine of choice (decided to keep it simple and used SQLite)
Setup your first project
Open the URL to your Rails project
All of these are spelled out admirably in the Rails Wiki, so for those looking to install and get up and running with Rails, allow yourself about 30 minutes and you’ll be good to go.
Additionally, I have to give some props to another tool that was also recommended as an “immediate first" and that’s Firebug. This is a nice little tool that allows the user/developer the ability to monitor, edit and debug HTML, CSS and JavaScript in real time.
It seems many of the really cool tools are likewise limited to Firefox, which makes for a mild drag when applications are optimized for IE, which a few I’m testing are… ah, but there is still hope, young ninja to be. There’s an extension to Firefox called IE Tab, which allows you to run Internet Explorer only pages inside of Firefox. It’s not a perfect fit, there are some oddities with layout, but so far, it’s letting me use a number of my favorite Firefox tool and still let me test the apps that have the IE limitation, so that’s a definite bonus! Now if I can find a way to run a “virtual browser” of Firefox 2.0 inside of an up-to-date version of Firefox, my quest would be complete (ummm, well, not really, but it would have been a neat cap for today :) ).
So there you have it, Day 2 of Boot Camp was focused on installing and identifying tools, putting them in place, and making sure they fired up effectively. Of course, installing and starting is the (relatively) easy part. Putting them to good use? That takes time and with that, basic training really begins!
It figures, I announce I’m going to do something dramatic, and what happens? Mother Nature gives me a back hand! I woke up on Thursday feeling like someone shoved a cork in my nose and swabbed my throat with sandpaper. In short, not feeling good at all, but with a ticking countdown, I couldn’t just do nothing.
Day 2 of boot camp consisted of me de-cluttering, de-junking and installing tools that I hope will get me towards my goal of “supa-fly web test ninja”… hey, I don’t have a better phrase at the moment, but I’m of course open to suggestions :). Seriously though, taking the time to de-clutter and de-junk a work area, room or system is imperative; it sets the stage and tells your body and mind that you are serious and going for broke here!
So what was the first job I decided to tackle? Get Ruby On Rails running on my laptop. Since I am running Windows 7, that meant I had to see what steps had to be jumped through to get the software installed. As is often the case with applications and environments not necessarily developed for Windows, I expected there’d be a few steps that would be bulky and require me to dig around and do modifications. I was wrong. The Ruby installer worked right out of the box as did the Development Kit, and from there the steps to install the necessary Ruby Gems, install Rails, install SQLite (just to keep things relatively easy; I’ll be playing with MYSQL a little later).
So what’s my first share item here? For starters, my tip of the hat to the good people who have created and maintain the Rails Wiki and their steps to making a darn near painless experience for installing on Windows. The steps are relatively straightforward and describe with the needed details without becoming overwhelming or mired in minutiae the important steps:
Installing Ruby and RubyGems
Installing Rails
Installing your Database engine of choice (decided to keep it simple and used SQLite)
Setup your first project
Open the URL to your Rails project
All of these are spelled out admirably in the Rails Wiki, so for those looking to install and get up and running with Rails, allow yourself about 30 minutes and you’ll be good to go.
Additionally, I have to give some props to another tool that was also recommended as an “immediate first" and that’s Firebug. This is a nice little tool that allows the user/developer the ability to monitor, edit and debug HTML, CSS and JavaScript in real time.
It seems many of the really cool tools are likewise limited to Firefox, which makes for a mild drag when applications are optimized for IE, which a few I’m testing are… ah, but there is still hope, young ninja to be. There’s an extension to Firefox called IE Tab, which allows you to run Internet Explorer only pages inside of Firefox. It’s not a perfect fit, there are some oddities with layout, but so far, it’s letting me use a number of my favorite Firefox tool and still let me test the apps that have the IE limitation, so that’s a definite bonus! Now if I can find a way to run a “virtual browser” of Firefox 2.0 inside of an up-to-date version of Firefox, my quest would be complete (ummm, well, not really, but it would have been a neat cap for today :) ).
So there you have it, Day 2 of Boot Camp was focused on installing and identifying tools, putting them in place, and making sure they fired up effectively. Of course, installing and starting is the (relatively) easy part. Putting them to good use? That takes time and with that, basic training really begins!
Wednesday, December 8, 2010
Day 1 of 40: TESTHEAD BOOT CAMP: Superfly Web Ninja in 40 Days?!!
If one thing has been made abundantly clear to me over the past couple of weeks working through How We Test Software at Microsoft
, I have decided that I must overcome my aversion to and phobia of coding. I can read code, I can write code, but it's just not something I'm all that totally psyched about doing, so it often ends up low on my priority list. It's like saying I dig epic Greek poetry, but won't open or try to read the epics in their original Greek. I can still talk about them, I can still appreciate them, heck I can even criticize them, but unless I can open a book with the original Greek and sound out and savor those words, I'm doomed to a second rate experience, doomed to have others savor the richness that I cannot. Thus, no more excuses, it's time to get into shape! What's going on inside my brain at this very moment, in Godinese (Seth Godin speak), is my Lizard Brain screaming at the top of its lungs, and The Resistance working all of its magic on me. In the past, it has often succeeded, but now the time has come to shout it down, and since my "credibility" is my ultimate accountability (see Monday's blog for my comments on that :) ), it's time to do what I often find the most effective way to really commit to doing something... shout like a fool from the rooftops and give regular updates. So that's what I'm going to do!
Thus, starting today, I hereby challenge myself to become a "supa-fly ultra-modern web ninja tester" and I will give myself 40 days to do it!
WHOOMP! There it is!!! (stands cockily with arms crossed and head cocked to the side for semi-thug-geek comic relief).
No seriously, I have an initiative that requires that I freshen up my web testing skills, like big time. No more excuses, no more Lizard Brain, no more Resistance, it's time to get serious and it's gonna happen publicly, and you can follow along. For the next 40 days, I hereby commit to a DAILY commentary (or as close to daily commentary as I can so that I can meet this goal, or at least get as close to it as I possibly can). What's more, I encourage those who want to follow along and practice with me to do so. If I highlight books, resources, tutorials, or anything else, I'll reference them here and I'll let you see the thrills of my victories and the agonies of my defeats.
The goals: implement Selenium, Cucumber, Firebug, Rails and Agile methodologies, learn and implement tests with a unit testing framework, create projects to practice these technologies, explain what I do and have learned, share my pain points and be willing to share in any way what pearls of wisdom I learn along the way.
TESTHEAD BOOT CAMP is now officially in session.
FALL IN, MAGGOTS! DROP AND GIVE ME TWENTY!!!
Tuesday, December 7, 2010
BOOK CLUB: How We Test Software at Microsoft (6/16)
This is the sixth installment of the TESTHEAD book club covering How We Test Software at Microsoft. This is a continuation of Section 2, which deals with the philosophy of testing and the testing concepts that Microsoft embraces and teaches its SDE’s and SDET’s. This section covers the area of Structural testing, and again, the density and detail provided could take several blog posts to cover. Rather than give a run down and bit by bit summary of the entire chapter, this is a high level view and will naturally only be able to skim the surface.
Chapter 6. Structural Testing Techniques
Bj starts out this chapter with an anecdote that deserves to be left as is:
A kindergarten teacher asked her students, "Who knows the color of apples?" A student dutifully raised his hand and the teacher said, "Ethan, what is the color of apples?" Young Ethan proudly stood up and stated, "Apples are red." Then, several other hands raised and the teacher picked another child. "Yes, Emma?" queried the teacher. Emma replied, "But some apples are green." "Correct!" exclaimed the teacher. Then, several more hands went up, and the teacher again selected an anxious student who was vigorously waving his hand. But young Caroline couldn’t control herself and before the selected student could answer she shouted out, "And some apples are yellow." "Yes, that’s right! Apples are red, green, and yellow," stated the teacher. But before she continued with the lesson, the teacher noticed a quiet little girl in the corner of the room with her hand still raised. The teacher called upon the child and the little girl said, "Apples are also white." The somewhat puzzled teacher politely replied, "Elizabeth, there are red apples, and green apples, and yellow apples, but I have never seen a white apple." The other children in the room began to laugh. Elizabeth stood up, peered over her glasses at the teacher and bluntly declared, "All apples are white on the inside!"
This is a great analogy, because very often we look at our applications from the external view, and our testing is focused primarily on what we see and how we interact with it. As the girl in the Kindergarten class accurately points out, the skin is less than 1% of the total volume and consistency of an apple. It’s the same with software. What happens underneath the skin is every bit as important as what we observe on the surface (if not lots more so).
This is where structural testing techniques come into play. Structural testing is necessary because the surface level behavioral tests and the direct function level unit tests don’t cover everything. There are lots of things that are not even touched by these two approaches. Structural testing is a white box testing approach. Intimate knowledge of the code is needed to make it effective. This means if a program is coded in C# or VB, it’s important that we also be very conversant in that language as well.
The big question Bj confronts, and one I’ve long wondered about, is “do testers who have intimate knowledge of the code end up biasing their testing to the functions themselves, meaning they prove that the functions do what they are meant to do, and that’s it?". I’ve long feared that I would be biased in this way, but Bj makes the case that here is where test rigor and focus is required, and the same exploratory mindset that is used for behavioral and black box testing methods should be applied here as well, and if they are applied, then those biases will likely not come into play.
Bj makes a comparison to testing done at Microsoft over several years, and has determined that when scripted testing and exploratory testing from a behavioral and black box perspective are applied, the difference in issues discovered by percentage and code coverage provided are relatively the same, with a nominal increase in coverage by combining both methods. Using structural testing in addition to behavioral methods brings the tester to close to 91% test coverage. Key takeaway: test tools are tools, and each does their jobs differently. To get the most out of them, make sure you are using the right tool for the right job at the right time. Structural tests will give you clues that no scripted or exploratory testing will hint at.
Block Testing
I have to admit that this is about the point that I feel myself getting out of my league, but I found the descriptions for talking about block coverage vs. statement coverage to be actually understandable (of course, that may just be repeat exposure, too). Statement coverage basically means the number of statements or lines of code that have been executed in a given round of testing. Block coverage means tests that measure groups of statements that don’t branch. Flow control mechanisms and the decisions made determine how much coverage is actually provided. In short, it’s possible to execute all statements, but not actually cover all of the blocks, and vice versa. You have to be aware of the differences to make sure that you are covering what you think you are covering. Even then, statement coverage or block coverage does not necessarily mean effective tests. This is where that nose for trouble that makes exploratory testing such an effective technique can come in handy while devising structural tests.
One place where structural testing really shows its benefits is with error control or exception handling. Black box/behavioral testing methods do not typically give much insight into where to find them, or to guarantee they have all be loaded and shown at least once (referencing “Whittaker Attack #1” again). Therefore, knowing what it takes to trigger these exceptions and generate errors is an important part of structural testing.
Decision Testing
Decision testing takes simple Boolean expressions (yes or no, true or false, 0 or 1) and branch one way or another based the condition being true or false. goal of decision testing is so that the tester can verify both true and false outcomes of Boolean expression. The benefit of this type of testing is that flow control can be closely monitored and branching can be observed based on a simple yes/no answer. For more complicated conditions, or multiple conditions, that’s where Condition testing comes in.
Condition Testing
Condition testing comes into play when there are multiple options to evaluate, such as AND or OR clauses in statements, where some things may be true while others are false, and containing two or more Boolean expressions. Condition tests evaluate true and false outcome of each subexpression within a compound conditional clause.
Basis Path Testing
Path testing tries to cover and exercise every possible path through a program. In reality, the sheer potential number of tests, especially in code paths that incorporate nested loops, can become astronomical. Performing exhaustive testing is perhaps not impossible, but it’s incredibly impractical in many cases, especially those that end up with trillions of test cases that would take a thousand years to complete, even at their top speeds.
Bj takes a diversion here to describe what cyclomatic complexity is.
Cyclomatic complexity measures the decision logic used to control flow through a module or a function. The formula to calculate cyclomatic complexity is v(G) = Edges – Nodes + 2, but if you must calculate cyclomatic complexity by hand an easier formula is to simply count the number of conditional clauses and add 1 (v(G) = p + 1).
A basis path is defined as a linearly independent path through a function. For example, in a function with one conditional clause there are two independent paths through it.
The simplified baseline path technique is described and applied in the following steps:
1. Identifies the shortest baseline path (the path with the fewest number of conditional clauses) through a function from the entry point to the exit point of the function being tested.
2. Return to the entry point of the function.
3. Trace control flow from the entry point to the first conditional clause that has not been evaluated to both its true and false outcomes.
4. Change the conditional clause to its alternate outcome.
5. Follow the shortest path from that conditional clause to the exit point.
6. Repeat steps 2 through 6 until all basis paths (equal to the cyclomatic complexity metric) are defined.
While the simplified baseline path technique can identify identifying your basis paths, it can also result in a set of paths that are either too restrictive or otherwise considerably unlikely. To address this there is also the Practical Baseline Path Technique, which differs in the following ways:
1. Identify a likely functional baseline path through the function that is representative of highly probable control flow through the function that is most commonly encountered at run time, or the most important or critical path through the function.
2. Return to the entry point of the function.
3. Trace control flow from the entry point to the first conditional clause that has not been evaluated to both its true and false outcomes.
4. Change the conditional clause to its alternate outcome.
5. Follow a path to the exit point that includes the maximum number of conditional clauses traversed by the baseline path.
6. Repeat steps 2 through 6 until every conditional clause has been evaluated to both its true and false outcomes and all basis paths are defined.
Basis path testing requires each outcome for each conditional statement be tested independently. In the event there is a flow control that has an option for a yes or no answer, basis path tests will require the user to follow those commands and the structures they lead to, including tests that bypass code structures. Net result, more testing than block or decision testing will provide on their own.
Key to this whole discussion is, of course, the fact that this approach to white-box, code aware testing is a technique, and it’s one of many. It isn’t the be-all and end all technique, just as Exploratory or Scripted testing aren’t the be all and end all either. Context, timing and focus all come into play. Structural techniques are great when the focus is code, statement and block coverage to make sure that you are addressing areas that black box testing will not give you clues about. Risk goes way up if we don’t actually realize that we are traveling down pathways that we would not know to test or even understand would need to be tested without that level of knowledge. Make no mistake, any white-box testing technique takes time and patience to master (again, I freely admit this is an area I need help in, and I appreciate the detail in which it was covered).
Chapter 7 coming this Friday.
. This is a continuation of Section 2, which deals with the philosophy of testing and the testing concepts that Microsoft embraces and teaches its SDE’s and SDET’s. This section covers the area of Structural testing, and again, the density and detail provided could take several blog posts to cover. Rather than give a run down and bit by bit summary of the entire chapter, this is a high level view and will naturally only be able to skim the surface.Chapter 6. Structural Testing Techniques
Bj starts out this chapter with an anecdote that deserves to be left as is:
A kindergarten teacher asked her students, "Who knows the color of apples?" A student dutifully raised his hand and the teacher said, "Ethan, what is the color of apples?" Young Ethan proudly stood up and stated, "Apples are red." Then, several other hands raised and the teacher picked another child. "Yes, Emma?" queried the teacher. Emma replied, "But some apples are green." "Correct!" exclaimed the teacher. Then, several more hands went up, and the teacher again selected an anxious student who was vigorously waving his hand. But young Caroline couldn’t control herself and before the selected student could answer she shouted out, "And some apples are yellow." "Yes, that’s right! Apples are red, green, and yellow," stated the teacher. But before she continued with the lesson, the teacher noticed a quiet little girl in the corner of the room with her hand still raised. The teacher called upon the child and the little girl said, "Apples are also white." The somewhat puzzled teacher politely replied, "Elizabeth, there are red apples, and green apples, and yellow apples, but I have never seen a white apple." The other children in the room began to laugh. Elizabeth stood up, peered over her glasses at the teacher and bluntly declared, "All apples are white on the inside!"
This is a great analogy, because very often we look at our applications from the external view, and our testing is focused primarily on what we see and how we interact with it. As the girl in the Kindergarten class accurately points out, the skin is less than 1% of the total volume and consistency of an apple. It’s the same with software. What happens underneath the skin is every bit as important as what we observe on the surface (if not lots more so).
This is where structural testing techniques come into play. Structural testing is necessary because the surface level behavioral tests and the direct function level unit tests don’t cover everything. There are lots of things that are not even touched by these two approaches. Structural testing is a white box testing approach. Intimate knowledge of the code is needed to make it effective. This means if a program is coded in C# or VB, it’s important that we also be very conversant in that language as well.
The big question Bj confronts, and one I’ve long wondered about, is “do testers who have intimate knowledge of the code end up biasing their testing to the functions themselves, meaning they prove that the functions do what they are meant to do, and that’s it?". I’ve long feared that I would be biased in this way, but Bj makes the case that here is where test rigor and focus is required, and the same exploratory mindset that is used for behavioral and black box testing methods should be applied here as well, and if they are applied, then those biases will likely not come into play.
Bj makes a comparison to testing done at Microsoft over several years, and has determined that when scripted testing and exploratory testing from a behavioral and black box perspective are applied, the difference in issues discovered by percentage and code coverage provided are relatively the same, with a nominal increase in coverage by combining both methods. Using structural testing in addition to behavioral methods brings the tester to close to 91% test coverage. Key takeaway: test tools are tools, and each does their jobs differently. To get the most out of them, make sure you are using the right tool for the right job at the right time. Structural tests will give you clues that no scripted or exploratory testing will hint at.
Block Testing
I have to admit that this is about the point that I feel myself getting out of my league, but I found the descriptions for talking about block coverage vs. statement coverage to be actually understandable (of course, that may just be repeat exposure, too). Statement coverage basically means the number of statements or lines of code that have been executed in a given round of testing. Block coverage means tests that measure groups of statements that don’t branch. Flow control mechanisms and the decisions made determine how much coverage is actually provided. In short, it’s possible to execute all statements, but not actually cover all of the blocks, and vice versa. You have to be aware of the differences to make sure that you are covering what you think you are covering. Even then, statement coverage or block coverage does not necessarily mean effective tests. This is where that nose for trouble that makes exploratory testing such an effective technique can come in handy while devising structural tests.
One place where structural testing really shows its benefits is with error control or exception handling. Black box/behavioral testing methods do not typically give much insight into where to find them, or to guarantee they have all be loaded and shown at least once (referencing “Whittaker Attack #1” again). Therefore, knowing what it takes to trigger these exceptions and generate errors is an important part of structural testing.
Decision Testing
Decision testing takes simple Boolean expressions (yes or no, true or false, 0 or 1) and branch one way or another based the condition being true or false. goal of decision testing is so that the tester can verify both true and false outcomes of Boolean expression. The benefit of this type of testing is that flow control can be closely monitored and branching can be observed based on a simple yes/no answer. For more complicated conditions, or multiple conditions, that’s where Condition testing comes in.
Condition Testing
Condition testing comes into play when there are multiple options to evaluate, such as AND or OR clauses in statements, where some things may be true while others are false, and containing two or more Boolean expressions. Condition tests evaluate true and false outcome of each subexpression within a compound conditional clause.
Basis Path Testing
Path testing tries to cover and exercise every possible path through a program. In reality, the sheer potential number of tests, especially in code paths that incorporate nested loops, can become astronomical. Performing exhaustive testing is perhaps not impossible, but it’s incredibly impractical in many cases, especially those that end up with trillions of test cases that would take a thousand years to complete, even at their top speeds.
Bj takes a diversion here to describe what cyclomatic complexity is.
Cyclomatic complexity measures the decision logic used to control flow through a module or a function. The formula to calculate cyclomatic complexity is v(G) = Edges – Nodes + 2, but if you must calculate cyclomatic complexity by hand an easier formula is to simply count the number of conditional clauses and add 1 (v(G) = p + 1).
A basis path is defined as a linearly independent path through a function. For example, in a function with one conditional clause there are two independent paths through it.
The simplified baseline path technique is described and applied in the following steps:
1. Identifies the shortest baseline path (the path with the fewest number of conditional clauses) through a function from the entry point to the exit point of the function being tested.
2. Return to the entry point of the function.
3. Trace control flow from the entry point to the first conditional clause that has not been evaluated to both its true and false outcomes.
4. Change the conditional clause to its alternate outcome.
5. Follow the shortest path from that conditional clause to the exit point.
6. Repeat steps 2 through 6 until all basis paths (equal to the cyclomatic complexity metric) are defined.
While the simplified baseline path technique can identify identifying your basis paths, it can also result in a set of paths that are either too restrictive or otherwise considerably unlikely. To address this there is also the Practical Baseline Path Technique, which differs in the following ways:
1. Identify a likely functional baseline path through the function that is representative of highly probable control flow through the function that is most commonly encountered at run time, or the most important or critical path through the function.
2. Return to the entry point of the function.
3. Trace control flow from the entry point to the first conditional clause that has not been evaluated to both its true and false outcomes.
4. Change the conditional clause to its alternate outcome.
5. Follow a path to the exit point that includes the maximum number of conditional clauses traversed by the baseline path.
6. Repeat steps 2 through 6 until every conditional clause has been evaluated to both its true and false outcomes and all basis paths are defined.
Basis path testing requires each outcome for each conditional statement be tested independently. In the event there is a flow control that has an option for a yes or no answer, basis path tests will require the user to follow those commands and the structures they lead to, including tests that bypass code structures. Net result, more testing than block or decision testing will provide on their own.
Key to this whole discussion is, of course, the fact that this approach to white-box, code aware testing is a technique, and it’s one of many. It isn’t the be-all and end all technique, just as Exploratory or Scripted testing aren’t the be all and end all either. Context, timing and focus all come into play. Structural techniques are great when the focus is code, statement and block coverage to make sure that you are addressing areas that black box testing will not give you clues about. Risk goes way up if we don’t actually realize that we are traveling down pathways that we would not know to test or even understand would need to be tested without that level of knowledge. Make no mistake, any white-box testing technique takes time and patience to master (again, I freely admit this is an area I need help in, and I appreciate the detail in which it was covered).
Chapter 7 coming this Friday.
Monday, December 6, 2010
Is Credibility The Ultimate Accountability Partner?
This is something I’ve been musing about lately, and I’ve had success with it enough in the past to know that, at least for me it is true.
Why is it that, when we shout something out to the World, we are much more likely to follow through than when we keep something private? Has anyone else found the converse to be true, that when you are quiet about your goals, you do better than if you announce them to the world?
I’ve been thinking about this the past several weeks, especially when it comes to the forum of posts that I make on this blog. I feel honor bound to post three to four times a week, even though there’s no official expectation of my doing that. I specifically felt it the most when I was doing a book review every week. When it got to the point where I couldn’t do one every week any more (or at least not in a manner that I would consider “in depth” enough to be worthwhile, I actually got a little depressed. Why? I felt like I was letting people down… people I have never met and in many cases never will meet, yet I still felt like I was committed to producing for them. Why? Because these were the people who were establishing my credibility as a tester and a writer.
This really made me think about the idea behind credibility. Who ultimately bestows on us the mantle of “credible”? Is it our boss? Our company? Our Peers? Our Community? I think it’s all of these at different times, but in some ways, I think the farther away the interaction, the more motivated someone becomes to establish and keep their credibility. When we are at work, and we interact with people every day, there are many excuses we can use to side-slip the ultimate issue of being accountable. I’m not saying this is a good thing at all, but it seems to be an inevitable thing. We expect leeway from the people we interact with every day. For those who don’t know me at all, or for those who only know me through TESTHEAD, my motivation is very different. I may only get one chance to build credibility with those eyes out there, so I only have one chance to make a good impression. If I didn’t chances are they won’t come round again.
I think in some ways this is what makes the things that I do on TESTHEAD and other places ultimately drive me and motivate me; my credibility is ultimately tied to my following through on the things I say I am going to do, and in the web space, I have no room for slipping or being late. Of course, I find that I cannot deliver exactly on time in many ways, and it’s entirely possible that those people out there that do read this blog really don’t care if I’m on time or not, but again, it matters to me, because I’ve chosen this platform to represent me, and whether it’s my best friend a block away or a complete stranger half a world away, TESTHEAD is me, and I am TESTHEAD (and we are all together… OK, bad and old pop culture reference, sorry :) ).
I’m serious with the above questions I posed, though, so I’d appreciate some feedback. Who else finds their accountability to be closely tied to their credibility, and does anyone else have similar reactions to it the way that I do? Really, I’m curious!
Why is it that, when we shout something out to the World, we are much more likely to follow through than when we keep something private? Has anyone else found the converse to be true, that when you are quiet about your goals, you do better than if you announce them to the world?
I’ve been thinking about this the past several weeks, especially when it comes to the forum of posts that I make on this blog. I feel honor bound to post three to four times a week, even though there’s no official expectation of my doing that. I specifically felt it the most when I was doing a book review every week. When it got to the point where I couldn’t do one every week any more (or at least not in a manner that I would consider “in depth” enough to be worthwhile, I actually got a little depressed. Why? I felt like I was letting people down… people I have never met and in many cases never will meet, yet I still felt like I was committed to producing for them. Why? Because these were the people who were establishing my credibility as a tester and a writer.
This really made me think about the idea behind credibility. Who ultimately bestows on us the mantle of “credible”? Is it our boss? Our company? Our Peers? Our Community? I think it’s all of these at different times, but in some ways, I think the farther away the interaction, the more motivated someone becomes to establish and keep their credibility. When we are at work, and we interact with people every day, there are many excuses we can use to side-slip the ultimate issue of being accountable. I’m not saying this is a good thing at all, but it seems to be an inevitable thing. We expect leeway from the people we interact with every day. For those who don’t know me at all, or for those who only know me through TESTHEAD, my motivation is very different. I may only get one chance to build credibility with those eyes out there, so I only have one chance to make a good impression. If I didn’t chances are they won’t come round again.
I think in some ways this is what makes the things that I do on TESTHEAD and other places ultimately drive me and motivate me; my credibility is ultimately tied to my following through on the things I say I am going to do, and in the web space, I have no room for slipping or being late. Of course, I find that I cannot deliver exactly on time in many ways, and it’s entirely possible that those people out there that do read this blog really don’t care if I’m on time or not, but again, it matters to me, because I’ve chosen this platform to represent me, and whether it’s my best friend a block away or a complete stranger half a world away, TESTHEAD is me, and I am TESTHEAD (and we are all together… OK, bad and old pop culture reference, sorry :) ).
I’m serious with the above questions I posed, though, so I’d appreciate some feedback. Who else finds their accountability to be closely tied to their credibility, and does anyone else have similar reactions to it the way that I do? Really, I’m curious!
Saturday, December 4, 2010
BOOK CLUB: How We Test Software at Microsoft (5/16)
This is my fifth installment in the TESTHEAD BOOK CLUB for the book How We Test Software at Microsoft, written by Alan Page, Ken Johnston and Bj Rollison.
In Chapter 5, Bj takes the lead and discusses Functional Testing Techniques. By the way, a side note about how I’m writing these reviews and summaries. There are times where my take on the details in the book will not suffice, and the actual text of the area or really close summation (Where it’s obvious that it’s not my words) needs to be differentiated. That’s what the red text represents. Whenever you see red text, it is pretty much verbatim what’s in the book. I do this mostly with checklists and tables that I think are key to understanding the chapter details, and I do it when I make direct quotes to text in the book. So now you know.
Chapter 5. Functional Testing Techniques
Bj describes a Christmas morning when he received a gift of a Demolition derby slot car kit. Setting up the track and having the cars crash into each other was great fun, and Bj credits this with possibly starting him down the road towards testing, or as he put it, his “penchant for breaking things”.
The play was fun, but more to the point, his curiosity got him to wondering how the cars actually worked. HE admits to enjoying this, but writes with chagrin that unraveling the motor winding wasn’t a good idea because he didn’t know how to put it back together again, and so the car was permanently broken (a trip to the toy store and a new car fixed that). The takeaway is that you can learn a lot about something just by taking it apart and dissecting it, and likewise putting it back together again (if of course, you can :) ). This curiosity is a hallmark of testers, and therefore it should be encouraged. Likewise with software, if we want to learn about the program beyond the surface level user experience testing, we need to be willing to unscrew the top and look at “the wiring under the board” or in this case, the code structures that hold the application together.
One approach is to break the product down to its specific, discrete feature sets, and then test the specific functional attributes and capabilities of those specific pieces. This chapter focuses on ways that testers at Microsoft do exactly that.
The Need for Functional Testing
No question about it, software today is more complex than ever before, and it’s just going to continue on that way with the demand for more features and functionality. The tester’s job will not get any easier as time goes by, so testers have to make choices. How can they define a set of tests that will effectively examine a product or feature set while knowing full well that they will never be able to do an exhaustive run of all possible tests? More to the point, how can we hone in on those high value tests and provide the best coverage possible with the limited amount of testing we will realistically be able to do?
To this end, and to my great pleasure, Bj champions the use of Exploratory Testing (ET). ET is a great way to get a handle on an application, especially when a tester is new to it. My favorite metaphor for Exploratory testing is “testing without a map” (and credit for that phrase rightfully belongs to Michael Bolton since he wrote an article with that title, hence why I associate that phrase with ET :) ). Bj argues that Exploratory Testing can be sufficient for small software projects, or software with limited distribution, or software with a limited shelf life. But that it doesn’t scale well for large-scale, complex or mission-critical applications. I may disagree with that personally, but I get what he’s trying to say, in that Exploratory Testing techniques will not be the be all and the end all with testing (nor should it be; all tools have their right time and their right place). Exploratory Testing from a black box perspective also will not answer what BJ is referring to here, which is performing in-depth and detailed tests specific to what the functions themselves are coded for. Bj makes a good point in that black box behavioral testing, while certainly a key element, cannot be the sole approach to testing.
The question he proposes is this: How can we increase the effectiveness of our tests to limit redundancy and reduce our team’s overall exposure to risk?
Weinberg’s Triangle Revisited
Remember how I said that Alan Page emphasized that the stories and anecdotes will probably stick around with the users more than the book prose will? BJ proves that point again in this chapter with an example of “Weinberg’s Triangle”. What’s that? It’s a simple program based on Gerald Weinberg’s triangle problem to establish a baseline skill assessment. The goal is to use their existing skills and knowledge within 15 minutes to explore the application and to define tests necessary to determine if the program satisfies functional requirements. The requirement states that the program reads three integer values representing the lengths of the sides of a triangle and then displays a message: the triangle is scalene, isosceles, or equilateral. This time box limitation forces the tester to try to identify the most important tests. In reviewing over 5,000 samples, they discovered that the majority of SDETs included only one test in which valid integer inputs would result in an invalid triangle type, one test for equilateral, one test for scalene, and one test for isosceles. These four tests exercise approximately cover only 50 percent of the paths in the most critical method in the software.
So how can functional techniques help us here? Functional techniques allow us to systematically look at the feature sets capabilities. Note: this implies white box techniques and understanding of the code, but it doesn’t have to. It’s entirely possible to perform black box user interface level tests using Functional techniques. The key is to use them in the proper context; when you test and why you test are just as, if not more important, than what you test.
Bj takes another diversion with a comment he calls the “Pesticide Paradox”. In this he describes growing food in his backyard and the fact that, once planted in the open, numerous pests come after the food, specifically slugs. While there are various efforts he uses (beer, salt, copper stripping, wood ash, etc.) no matter how many he uses, invariably some slugs get through.
Likewise, when we test, if we rely on just one method, bugs will get through. Even if we vary our approach and use all sorts of different testing methods, we may catch or stop some bugs, but guess what? There’s still going to be some bugs that get through, because no matter how thorough we are, bugs will prove resistant to one form of testing or another. So diversifying the testing and using different tactics will, if not eliminate all bugs, at least give the tester more of a fighting chance to knock out a larger percentage of them.
Functional testing techniques can also help by zeroing in on key areas and eliminating redundant or needless tests. The rub is that it requires a certain amount of domain knowledge and understanding to do that effectively. Ultimately, any testing technique is a tool, just like a screwdriver and a reciprocating saw are tools for a house builder. Knowing when to use the right tool will make all the difference in the effectiveness in completing the job.
So what are some key functional testing techniques? What are our reciprocating saws and screwdrivers, to carry the metaphor further? Some examples are:
• Boundary Value Analysis (BVA)
• Equivalence Class Partitioning (ECP)
• Combinatorial Analysis
• State Transition Testing
Used in the correct context, testers can focus on specific features and methods. Still, a tool is just that; it won’t do the work for you, and it won’t be able to evaluate or reason through what it finds. For that, you need a skilled, aware, alert, “sapient” tester (to borrow a much adored term from James Bach and others) who knows where, how and when to use the appropriate tools to accomplish the task at hand.
Bj goes through and gives a thorough breakdown of these test approaches, and making a thorough summary of any of them would be deserving of its own blog post, or several. Therefore, I can only give a high level view of each of them.
Equivalence Class Partitioning
Bj considers Equivalence Class Partitioning (ECP) to be foundational to understanding and using other testing techniques. Actually, most testers already do this, they just don’t realize that they are doing it, or may not know that what they are doing iis ECP.
For ECP to work, need to break down the possible variable values for each input and/or output parameter, and then determine a way to define valid and invalid inputs for each. The ECP tests are made up of groups of valid until all of them have been used, and then going through and examining the invalid values individually. The trick is that this requires a solid grasp of the underlying structure of the application so as to correctly define all valid and invalid data groups. With this information, we design positive and negatives tests to evaluate functionality, as well as the ability of the system to handle errors (sound familiar?).
The first benefit of ECP is that it helps us systematically reduce the number of tests while still providing confidence that the variable combinations we use will produce the expected result repeatedly.
For example, assume you are testing a text box control that accepts a string variable of Unicode characters between upper case A and Z with a minimum string length of 1 and a maximum string length of 25 characters. Exhaustive testing would include each letter one time (261) and each letter combination for every possible string length. So, to test for all possible inputs the number of tests is equal to 2625 + 2624 + 2623 . . . + 261.
Yeah, that’s a huge number, and that’s exactly Bj’s point. Instead of testing every character combination imaginable, we determine that the following would be a logical way of breaking it down:
• uppercase Unicode characters between uppercase A and Z
• any string of characters with a string length between 1 and 25 characters
In short, any combination of those two criteria will work as a positive input, and anything that falls outside of that range would be considered a negative input.
Randomly selecting valid or invalid values randomly is meant to help increase the odds of exposing unusual behavior that might not otherwise be exposed using “real-world valid class static data”.
Decomposing Variable Data
So how can we effectively break these variables down into a good enough subset of valid/invalid data groups? Bj states that a solid familiarity with the data and the underlying structures is key. It also requires an understanding of “historically problematic variables” (or areas that have been known to cause problems in the past). Scope and intended audience can also give clues as to the proper context for the data. The danger with this is twofold; first, if we overgeneralize, we will not get a data set likely to find anomalies. If our variable data is too specific (hyperanalysis), we run the risk of too many redundant test cases. Finding the balance comes with use and practice.
Some of this is going to seem like a blinding flash of the obvious, but the point that Bj makes here is subtle. Valid data should create a normal or expected response. Invalid data should produce errors. To help me with the subtlety, I was reminded of James Whittaker’s Attack #1 from How to Break Software: determine every potential error message in the code, and force the system to display it at least once.
Bj uses the example of reserved words for system variables in a PC to demonstrate how an error was uncovered. Certain terms like LPT1 and COM1, COM2, etc. cannot be used as file names because they are handled at the system level. During a training course and showing how to test these examples, they discovered something in Windows XP. Some of the values if they were used to save a file with the reserved name (LPT and COM 1 through 4) showed one error message, but entering other reserved names (LPT and COM 5 through 9) displayed an entirely different error. With EQP, it might be seen as simple to assume that LPT and COM values would be treated the same. In this particular case, that would have been incorrect, and choosing too few names would not have uncovered the issue.
Glenford Myers proposes four heuristics in the book The Art of Software Testing when looking to partition data values. BJ defines a heuristic as “a guideline, principle, or rule of thumb that is often useful in performing tasks such as decision making, troubleshooting, and problem solving.”
• Range: A contiguous set of data in which any data point in the minima and maxima boundary values of the range is expected to produce the same result. For example, a number field in which you can enter an integer value from 1 through 999. The valid equivalence class is >=1 and <=999. Invalid equivalence class ranges include integers <1 and >999.
• Group: A group of items or selections is permitted as long as all items are handled identically. For example, a list of items of vehicles used to determine a taxation category includes truck, car, motorcycle, motor home, and trailer. If truck, car, and motor home reference the same taxation category, that group of items is considered equivalent.
• Unique Data: in a class or subset of a class that might be handled differently from other data in that class or subset of a class. For example, the date January 19, 2038, at 3:14:07 is a unique date in applications that process time and date data from the BIOS clock and it should be separated into a discrete subset.
• Specific: The condition must be or must not be present. For example, in Simple Mail Transfer Protocol (SMTP) the at symbol (@) is a specific character that cannot be used as part of the e-mail name or domain name.
Bj uses an example of a program called NextDate that is based off the Gregorian Calendar and uses this as a a jumping off point for several pages of code and examples. While all of them are very instructive, the volume of information is too large to include in a single chapter review (again, might make for a great series of posts another time).
The key takeaway with ECP is that it allows the tester to winnow down test variables in a systematic way and reduce the total number of tests required to perform an exhaustive test of all possible values. When applied correctly ECP lets the tester do the following:
• Force the tester to perform a more detailed analysis of the feature set and the variable data used by input and output parameters
• Help testers identify corner cases they might have overlooked
• Provide clear evidence of which data sets were tested and how they were tested
• Increase test effectiveness systematically, which helps reduce risk
• Increase efficiency by logically reducing redundant tests
Boundary Value Analysis
Boundary value analysis (BVA) is possibly the best known and most often misused functional test tool because it is assumed to be a trivial process. In short, a lot of issues pop up at the boundaries of variable values. Analyzing and determining these boundaries is important. Using BVA along with EQP (yep, Alan wasn’t kidding when he mentioned the proliferation of TLA’s (three letter acronyms).
BVA or boundary testing is especially useful for detecting the following types of errors:
• Incorrect artificial constraints of a data type
• Erroneously assigned relational operators
• Wrapping of data types
• Problems with looping structures
• Off-by-one errors
BVA is helpful with examining fixed-constant values (such as date conventions or common constants used in mathematics) and fixed-variable values (such as the X-Y axes used in creating and resizing a window.
Defining Boundary Tests
Paul Jorgensen in “Software Testing: A Craftsman’s Approach” uses the formula 4n + 1 (or 6n + 1, which would include the minimum value –1 and maximum value +1), where n is equal to the number of independent parameters. This should not be seen as the be all and end all to boundary condition tests. The actual boundary conditions mav vary depending on the actual implementation, so check your code to be sure.
What about those values that are not at the extreme edge? Do we use the same tools or something else? BJ suggests that, after identifying all boundary conditions the minimum set of test data can be calculated with a simple formula: 3(BC), where BC is equal to the number of specific boundary conditions.
To use the example for the NextDate program referenced in the text, there are three hard limits:
Jorgensen’s 6n + 1 formula makes for (6 * 3) + 1, or 19 test cases (see book for table breakdown). There are additional values to consider, though. How about months that only have 30 days (or that pesky month that sometimes has 28 and sometimes 29 days)? Applying the 3(BC) formula for boundary testing reveals the test set required to analyze more potential boundary conditions. 3(BC) is (3 * 18), or 54 possible cases (again, see book).
Combinatorial Analysis
Put simply, it’s nice to have the ability to test simple structures and perform tests where only one variable at a time changes. However, in the real world, dynamic data entry, saves and access methods cause lots of variable values to be changed simultaneously. What’s more, many of these variables depend on each other. What to do? Combinatorial analysis to the stage!
Combinatorial analysis has a number of benefits, including:
• Identifies most defects caused by variable interaction
• Provides greater structural coverage
• Has great potential to reduce overall testing costs (when used appropriately)
Note: Combinatorial analysis is not an effective technique for testing independent parameters with no direct or indirect interaction, mathematical calculations, ordered parameter input, or inputs that require sequential operations.
Combinatorial Testing Approaches
There are a number of ways to apply Combinatorial testing, including:
• random or ad hoc methods: (best guess, random selection, rely primarily on intuition and luck of the tester).
• each choice: (EC, or simply testing each variable at least once).
• base choice: (BC, identifies a combination of variables as the base test. This is usually the happy path or the most commonly used combinations of variable states).
• orthogonal array: (OA, where each array variable is linked to a state for each interdependent parameter, and those states are mapped into an array).
• combination tests: (pair-wise through n-wise or t = n),
• exhaustive testing: (just doing every possible test combination, likely not possible in anything but the most trivial programs).
Chapter 6 will be covered on Tuesday.
, written by Alan Page, Ken Johnston and Bj Rollison. In Chapter 5, Bj takes the lead and discusses Functional Testing Techniques. By the way, a side note about how I’m writing these reviews and summaries. There are times where my take on the details in the book will not suffice, and the actual text of the area or really close summation (Where it’s obvious that it’s not my words) needs to be differentiated. That’s what the red text represents. Whenever you see red text, it is pretty much verbatim what’s in the book. I do this mostly with checklists and tables that I think are key to understanding the chapter details, and I do it when I make direct quotes to text in the book. So now you know.
Chapter 5. Functional Testing Techniques
Bj describes a Christmas morning when he received a gift of a Demolition derby slot car kit. Setting up the track and having the cars crash into each other was great fun, and Bj credits this with possibly starting him down the road towards testing, or as he put it, his “penchant for breaking things”.
The play was fun, but more to the point, his curiosity got him to wondering how the cars actually worked. HE admits to enjoying this, but writes with chagrin that unraveling the motor winding wasn’t a good idea because he didn’t know how to put it back together again, and so the car was permanently broken (a trip to the toy store and a new car fixed that). The takeaway is that you can learn a lot about something just by taking it apart and dissecting it, and likewise putting it back together again (if of course, you can :) ). This curiosity is a hallmark of testers, and therefore it should be encouraged. Likewise with software, if we want to learn about the program beyond the surface level user experience testing, we need to be willing to unscrew the top and look at “the wiring under the board” or in this case, the code structures that hold the application together.
One approach is to break the product down to its specific, discrete feature sets, and then test the specific functional attributes and capabilities of those specific pieces. This chapter focuses on ways that testers at Microsoft do exactly that.
The Need for Functional Testing
No question about it, software today is more complex than ever before, and it’s just going to continue on that way with the demand for more features and functionality. The tester’s job will not get any easier as time goes by, so testers have to make choices. How can they define a set of tests that will effectively examine a product or feature set while knowing full well that they will never be able to do an exhaustive run of all possible tests? More to the point, how can we hone in on those high value tests and provide the best coverage possible with the limited amount of testing we will realistically be able to do?
To this end, and to my great pleasure, Bj champions the use of Exploratory Testing (ET). ET is a great way to get a handle on an application, especially when a tester is new to it. My favorite metaphor for Exploratory testing is “testing without a map” (and credit for that phrase rightfully belongs to Michael Bolton since he wrote an article with that title, hence why I associate that phrase with ET :) ). Bj argues that Exploratory Testing can be sufficient for small software projects, or software with limited distribution, or software with a limited shelf life. But that it doesn’t scale well for large-scale, complex or mission-critical applications. I may disagree with that personally, but I get what he’s trying to say, in that Exploratory Testing techniques will not be the be all and the end all with testing (nor should it be; all tools have their right time and their right place). Exploratory Testing from a black box perspective also will not answer what BJ is referring to here, which is performing in-depth and detailed tests specific to what the functions themselves are coded for. Bj makes a good point in that black box behavioral testing, while certainly a key element, cannot be the sole approach to testing.
The question he proposes is this: How can we increase the effectiveness of our tests to limit redundancy and reduce our team’s overall exposure to risk?
Weinberg’s Triangle Revisited
Remember how I said that Alan Page emphasized that the stories and anecdotes will probably stick around with the users more than the book prose will? BJ proves that point again in this chapter with an example of “Weinberg’s Triangle”. What’s that? It’s a simple program based on Gerald Weinberg’s triangle problem to establish a baseline skill assessment. The goal is to use their existing skills and knowledge within 15 minutes to explore the application and to define tests necessary to determine if the program satisfies functional requirements. The requirement states that the program reads three integer values representing the lengths of the sides of a triangle and then displays a message: the triangle is scalene, isosceles, or equilateral. This time box limitation forces the tester to try to identify the most important tests. In reviewing over 5,000 samples, they discovered that the majority of SDETs included only one test in which valid integer inputs would result in an invalid triangle type, one test for equilateral, one test for scalene, and one test for isosceles. These four tests exercise approximately cover only 50 percent of the paths in the most critical method in the software.
So how can functional techniques help us here? Functional techniques allow us to systematically look at the feature sets capabilities. Note: this implies white box techniques and understanding of the code, but it doesn’t have to. It’s entirely possible to perform black box user interface level tests using Functional techniques. The key is to use them in the proper context; when you test and why you test are just as, if not more important, than what you test.
Bj takes another diversion with a comment he calls the “Pesticide Paradox”. In this he describes growing food in his backyard and the fact that, once planted in the open, numerous pests come after the food, specifically slugs. While there are various efforts he uses (beer, salt, copper stripping, wood ash, etc.) no matter how many he uses, invariably some slugs get through.
Likewise, when we test, if we rely on just one method, bugs will get through. Even if we vary our approach and use all sorts of different testing methods, we may catch or stop some bugs, but guess what? There’s still going to be some bugs that get through, because no matter how thorough we are, bugs will prove resistant to one form of testing or another. So diversifying the testing and using different tactics will, if not eliminate all bugs, at least give the tester more of a fighting chance to knock out a larger percentage of them.
Functional testing techniques can also help by zeroing in on key areas and eliminating redundant or needless tests. The rub is that it requires a certain amount of domain knowledge and understanding to do that effectively. Ultimately, any testing technique is a tool, just like a screwdriver and a reciprocating saw are tools for a house builder. Knowing when to use the right tool will make all the difference in the effectiveness in completing the job.
So what are some key functional testing techniques? What are our reciprocating saws and screwdrivers, to carry the metaphor further? Some examples are:
• Boundary Value Analysis (BVA)
• Equivalence Class Partitioning (ECP)
• Combinatorial Analysis
• State Transition Testing
Used in the correct context, testers can focus on specific features and methods. Still, a tool is just that; it won’t do the work for you, and it won’t be able to evaluate or reason through what it finds. For that, you need a skilled, aware, alert, “sapient” tester (to borrow a much adored term from James Bach and others) who knows where, how and when to use the appropriate tools to accomplish the task at hand.
Bj goes through and gives a thorough breakdown of these test approaches, and making a thorough summary of any of them would be deserving of its own blog post, or several. Therefore, I can only give a high level view of each of them.
Equivalence Class Partitioning
Bj considers Equivalence Class Partitioning (ECP) to be foundational to understanding and using other testing techniques. Actually, most testers already do this, they just don’t realize that they are doing it, or may not know that what they are doing iis ECP.
For ECP to work, need to break down the possible variable values for each input and/or output parameter, and then determine a way to define valid and invalid inputs for each. The ECP tests are made up of groups of valid until all of them have been used, and then going through and examining the invalid values individually. The trick is that this requires a solid grasp of the underlying structure of the application so as to correctly define all valid and invalid data groups. With this information, we design positive and negatives tests to evaluate functionality, as well as the ability of the system to handle errors (sound familiar?).
The first benefit of ECP is that it helps us systematically reduce the number of tests while still providing confidence that the variable combinations we use will produce the expected result repeatedly.
For example, assume you are testing a text box control that accepts a string variable of Unicode characters between upper case A and Z with a minimum string length of 1 and a maximum string length of 25 characters. Exhaustive testing would include each letter one time (261) and each letter combination for every possible string length. So, to test for all possible inputs the number of tests is equal to 2625 + 2624 + 2623 . . . + 261.
Yeah, that’s a huge number, and that’s exactly Bj’s point. Instead of testing every character combination imaginable, we determine that the following would be a logical way of breaking it down:
• uppercase Unicode characters between uppercase A and Z
• any string of characters with a string length between 1 and 25 characters
In short, any combination of those two criteria will work as a positive input, and anything that falls outside of that range would be considered a negative input.
Randomly selecting valid or invalid values randomly is meant to help increase the odds of exposing unusual behavior that might not otherwise be exposed using “real-world valid class static data”.
Decomposing Variable Data
So how can we effectively break these variables down into a good enough subset of valid/invalid data groups? Bj states that a solid familiarity with the data and the underlying structures is key. It also requires an understanding of “historically problematic variables” (or areas that have been known to cause problems in the past). Scope and intended audience can also give clues as to the proper context for the data. The danger with this is twofold; first, if we overgeneralize, we will not get a data set likely to find anomalies. If our variable data is too specific (hyperanalysis), we run the risk of too many redundant test cases. Finding the balance comes with use and practice.
Some of this is going to seem like a blinding flash of the obvious, but the point that Bj makes here is subtle. Valid data should create a normal or expected response. Invalid data should produce errors. To help me with the subtlety, I was reminded of James Whittaker’s Attack #1 from How to Break Software: determine every potential error message in the code, and force the system to display it at least once.
Bj uses the example of reserved words for system variables in a PC to demonstrate how an error was uncovered. Certain terms like LPT1 and COM1, COM2, etc. cannot be used as file names because they are handled at the system level. During a training course and showing how to test these examples, they discovered something in Windows XP. Some of the values if they were used to save a file with the reserved name (LPT and COM 1 through 4) showed one error message, but entering other reserved names (LPT and COM 5 through 9) displayed an entirely different error. With EQP, it might be seen as simple to assume that LPT and COM values would be treated the same. In this particular case, that would have been incorrect, and choosing too few names would not have uncovered the issue.
Glenford Myers proposes four heuristics in the book The Art of Software Testing when looking to partition data values. BJ defines a heuristic as “a guideline, principle, or rule of thumb that is often useful in performing tasks such as decision making, troubleshooting, and problem solving.”
• Range: A contiguous set of data in which any data point in the minima and maxima boundary values of the range is expected to produce the same result. For example, a number field in which you can enter an integer value from 1 through 999. The valid equivalence class is >=1 and <=999. Invalid equivalence class ranges include integers <1 and >999.
• Group: A group of items or selections is permitted as long as all items are handled identically. For example, a list of items of vehicles used to determine a taxation category includes truck, car, motorcycle, motor home, and trailer. If truck, car, and motor home reference the same taxation category, that group of items is considered equivalent.
• Unique Data: in a class or subset of a class that might be handled differently from other data in that class or subset of a class. For example, the date January 19, 2038, at 3:14:07 is a unique date in applications that process time and date data from the BIOS clock and it should be separated into a discrete subset.
• Specific: The condition must be or must not be present. For example, in Simple Mail Transfer Protocol (SMTP) the at symbol (@) is a specific character that cannot be used as part of the e-mail name or domain name.
Bj uses an example of a program called NextDate that is based off the Gregorian Calendar and uses this as a a jumping off point for several pages of code and examples. While all of them are very instructive, the volume of information is too large to include in a single chapter review (again, might make for a great series of posts another time).
The key takeaway with ECP is that it allows the tester to winnow down test variables in a systematic way and reduce the total number of tests required to perform an exhaustive test of all possible values. When applied correctly ECP lets the tester do the following:
• Force the tester to perform a more detailed analysis of the feature set and the variable data used by input and output parameters
• Help testers identify corner cases they might have overlooked
• Provide clear evidence of which data sets were tested and how they were tested
• Increase test effectiveness systematically, which helps reduce risk
• Increase efficiency by logically reducing redundant tests
Boundary Value Analysis
Boundary value analysis (BVA) is possibly the best known and most often misused functional test tool because it is assumed to be a trivial process. In short, a lot of issues pop up at the boundaries of variable values. Analyzing and determining these boundaries is important. Using BVA along with EQP (yep, Alan wasn’t kidding when he mentioned the proliferation of TLA’s (three letter acronyms).
BVA or boundary testing is especially useful for detecting the following types of errors:
• Incorrect artificial constraints of a data type
• Erroneously assigned relational operators
• Wrapping of data types
• Problems with looping structures
• Off-by-one errors
BVA is helpful with examining fixed-constant values (such as date conventions or common constants used in mathematics) and fixed-variable values (such as the X-Y axes used in creating and resizing a window.
Defining Boundary Tests
Paul Jorgensen in “Software Testing: A Craftsman’s Approach” uses the formula 4n + 1 (or 6n + 1, which would include the minimum value –1 and maximum value +1), where n is equal to the number of independent parameters. This should not be seen as the be all and end all to boundary condition tests. The actual boundary conditions mav vary depending on the actual implementation, so check your code to be sure.
What about those values that are not at the extreme edge? Do we use the same tools or something else? BJ suggests that, after identifying all boundary conditions the minimum set of test data can be calculated with a simple formula: 3(BC), where BC is equal to the number of specific boundary conditions.
To use the example for the NextDate program referenced in the text, there are three hard limits:
- month (1 to 12)
- day (1 to 31)
- year (1582 to 3000)
Jorgensen’s 6n + 1 formula makes for (6 * 3) + 1, or 19 test cases (see book for table breakdown). There are additional values to consider, though. How about months that only have 30 days (or that pesky month that sometimes has 28 and sometimes 29 days)? Applying the 3(BC) formula for boundary testing reveals the test set required to analyze more potential boundary conditions. 3(BC) is (3 * 18), or 54 possible cases (again, see book).
Combinatorial Analysis
Put simply, it’s nice to have the ability to test simple structures and perform tests where only one variable at a time changes. However, in the real world, dynamic data entry, saves and access methods cause lots of variable values to be changed simultaneously. What’s more, many of these variables depend on each other. What to do? Combinatorial analysis to the stage!
Combinatorial analysis has a number of benefits, including:
• Identifies most defects caused by variable interaction
• Provides greater structural coverage
• Has great potential to reduce overall testing costs (when used appropriately)
Note: Combinatorial analysis is not an effective technique for testing independent parameters with no direct or indirect interaction, mathematical calculations, ordered parameter input, or inputs that require sequential operations.
Combinatorial Testing Approaches
There are a number of ways to apply Combinatorial testing, including:
• random or ad hoc methods: (best guess, random selection, rely primarily on intuition and luck of the tester).
• each choice: (EC, or simply testing each variable at least once).
• base choice: (BC, identifies a combination of variables as the base test. This is usually the happy path or the most commonly used combinations of variable states).
• orthogonal array: (OA, where each array variable is linked to a state for each interdependent parameter, and those states are mapped into an array).
• combination tests: (pair-wise through n-wise or t = n),
• exhaustive testing: (just doing every possible test combination, likely not possible in anything but the most trivial programs).
Chapter 6 will be covered on Tuesday.
Friday, December 3, 2010
TWiST #22 with Jane Fraser and More TWiST-Plus from PNSQC 2010
It’s funny, since I started doing the podcast, I have not really invested in any podcasting infrastructure. The whole thing is done on a Toshiba Sattelite Notebook PC running Windows 7 and I’ve been using a generic Verizon Wireless promotional headphone and boom mic set-up. It’s OK, but the connection points for the audio are shorting and so it’s become not so reliable. You might think that going out and getting a replacement microphone and headset would not be a big deal. Unfortunately, I’ve become annoyingly picky after doing this now for almost 20 entries. I tried two different headsets but neither of them had the sound quality I was after. I ultimately decided that I’d do a hybrid approach; using the ear buds that I use when I listed to music on my MP3 player and using a cool desktop microphone made by Logitech and connects via USB. You will hear the difference, I hope, in next week’s podcasts when I produce them (this week’s already went to press before I made the changes).
So for today’s episode, Matt sits down with Jane Fraser, a QA director over at Electronic Arts (the gaming powerhouse with office in Redwood Shores, CA, Canada, China, Vietnam, and India). Much of the talk for this episode had to do with the challenges of testing and managing testing between all of these locations, and how many techniques that work for smaller local groups just don’t work as well in these distributed and somewhat autonomous locations. Anyway, it was neat to hear her perspective and see the state of an industry I left five years ago. So click the link if you want to hearEpisode 22.
In addition, I also did a Twist-Plus episode with Media Monks regarding their Virtual Extreme Programming workbench project and with two High school participants of the Lego League robotics competition challenge, both recorded during the poster paper presentations at Pacific Northwest Software Quality Conference 2010. You can listen to the latest Twist-Plus episode here.
Standard disclaimer:
Each TWiST podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to the podcasts and the entire library whenever you want to :). In addition, Pro membership allows you to access and download to the entire archive of Software Test and Quality Assurance Magazine, and its issues under its former name, Software Test and Performance.
TWiST-Plus is all extra material, and as such is not hosted behind STP’s site model. There is no limitation to accessing TWiST-Plus material, just click the link to download and listen.
Again, my thanks to STP for hosting the podcasts and storing the archive. We hope you enjoy listening to them as much as we enjoy making them :).
Thursday, December 2, 2010
Corkboard.me: Your Personal Kanban/GTD/SCRUM/Whatever Board
Timothy Coulter has made a neat and simple little app that is effectively an access anywhere corkboard. Want to keep all of those sticky notes, but not have actual sticky note clutter? Then go to http://corkboard.me and make your own corkboard!
The first thing that happens when you go to the page is that a random ID gets generated and pasted into the URL. Bookmark that URL, and you now have your own virtual corkboard space. Click to create a note. Type to create a message. Make as many as you need, Use Zoom to pull out or zoom in, or click and drag to physically move the board around. Put in an image URL and hit enter; the image becomes a picture on the corkboard. Move items around to your hearts content.
Seriously, this is a really cool way to keep small pieces of information that heretofore would be stuck to my monitor, and be totally useless to me after I left my desk. This way, anywhere the page can be displayed, I have access to the sticky notes... and yeah, I think that pretty cool.
There does seem to be some dodginess with the system around reloading at a different location, but hey, it's been available for five whole hours, I can live it it for the time being :).
So if you want to check it out for yourself, go to http://corkboard.me/ and see how it works.
The first thing that happens when you go to the page is that a random ID gets generated and pasted into the URL. Bookmark that URL, and you now have your own virtual corkboard space. Click to create a note. Type to create a message. Make as many as you need, Use Zoom to pull out or zoom in, or click and drag to physically move the board around. Put in an image URL and hit enter; the image becomes a picture on the corkboard. Move items around to your hearts content.
Seriously, this is a really cool way to keep small pieces of information that heretofore would be stuck to my monitor, and be totally useless to me after I left my desk. This way, anywhere the page can be displayed, I have access to the sticky notes... and yeah, I think that pretty cool.
There does seem to be some dodginess with the system around reloading at a different location, but hey, it's been available for five whole hours, I can live it it for the time being :).
So if you want to check it out for yourself, go to http://corkboard.me/ and see how it works.
Wednesday, December 1, 2010
Next Weekend Testing Americas Session: 12/11/2010
Weekend Testing Americas Session number 3 will be here in just 10 more day. Since European Weekend Testers is doing a session this coming Saturday, December 4th, and since our times are so close, generally speaking, we decided to wait a week for WTA03.
Lynn, Joe and I have been thinking that, since we are testers, it would be kind of cool to test a test tool. The previous two sessions, Shmuel Gershon's Rapid Reporting Tool has been used, discussed, and lauded by all who have used it. It's a tool that allows users to quickly capture information about their testing and have a record of what happened and use it for reporting (hence the name, Rapid Reporter). Since this has proven to be such a boon to our testers participating in our sessions, I asked Shmuel if he'd be game to have us use Rapid Reporter as our next testing challenge. He was totally up for it, so that's the plan.
So what do you all need to do to participate? It's simple.
First, go to weekendtesting.com and make an account if you don't already have one (it's not mandatory, but if you participate in Weekend Testing sessions, it would certainly be sporting to be a member of the community, right :)?).
Second, send a message to WTAmericas@gmail.com saying that you will be participating in the "WTA03" sesion to be held Saturday, December 11, 2010.
Third, download Rapid Reporter from http://testing.gershon.info/reporter/.
Fourth, if you haven't already done so, add "weekendtestersamericas" to your Skype contacts.
Fifth, come and join us on Saturday, November 11th at 11:00 AM PST (which is 1:00 PM EST, and I'll let the rest of the world do the math from there :) ).
Our testing challenge has been determined but will not be announced until the start of the session (in keeping with Weekend Testing tradition) but rest assured, Rapid Reporter will be a key part of that challenge, and is the point of the session. Please make sure you have installed it and can run it before the session begins. Caveat: this does mean that this session will be limited to those with Windows computers running with .NET 3.5, although even if you do not have such a system, you may be able to pair with another tester who does. Let us know if you have any issues prior to the start of the session, and we'll see what we can do to help make the experience a good one for all involved.
My thanks to Shmuel Gershon (an active participant in WTA as well as the creator for Rapid Reporter) for his willingness to let us go to town on it. We look forward to seing you all on the 11th.
Subscribe to:
Posts (Atom)