It probably has not gone unnoticed that I am posting less frequently than usual. There's a few reasons for that, and one of them is that I am trying to get all of my ideas down for the education modules that we will be preparing for Summer QAmp. I had initially planned on writing everything out, and posting it, and then massaging it down to a useable format. The past several days have taught me that that is going to take a long time, and it will give a very limited amount of time for feedback and review from others.
Because of this, I've decided to take a different approach, and I hope that you all will join me. We are discussing the elements that we want to present in the education materials over on the Association for Software Testing's EdSIG forum. At the moment, you have to be a member of AST to post in the forums, but we may well change that for this initiative, so as to get as many of the interested contributors as possible to be able to post.
In the meantime, though, the forum is open for reading to anyone who wants to participate, and if you would like to have your ideas included, please reply and let me know your interest and areas you would like to contribute.
Additionally, if you would prefer to communicate with me via email, you are welcome to send a message to me directly at mkltesthead (at) gmail (dot) com.
Monday, May 14, 2012
Wednesday, May 9, 2012
Cucumber Nerds: Can Color and Pipes Coexist?
I think I have gotten to a point where I have done all I can do to try to figure out a problem, and I'm coming up empty handed, so to my wonderful tester friends out there, especially those well versed in Cucumber, I need your help.
In a perfect world, I would come up with hooks to handle weird exceptions, and I would also have clean running tests every time. However, this is not a perfect world, and right now, getting the tests to reliably work is my priority. Also, I want to be able to rerun tests that fail and see if they are spurious failures or if there's a real problem.
How did i do this? I made a shell wrapper that would tee the output of the console to a rerun file, and then I would capture the failures, recreate a test suite of just the failed scenarios, and run them individually. Repeat until we get a 100% clean run or we confirm that something is genuinely broken.
So what's my problem? The lovely red/green/yellow color scheme that shows me what's happening at a glance disappears, replaced with a monochrome representation of all the steps. Do the tests run? Sure? Does the rerun whittle down my errors until I get to the essential problems? Yep, it does. I just want to have my cake and eat it too, if possible.
If I run without the rerun script (i.e. sans tee):
If I run with the rerun script (i.e. with the tee):
I get that this is because the system is trying to be smart and not deal with colors because the output is going to a pipe and not to the traditional console. My question is, how can I make my system stop being so smart? the -c (force color) option doesn't cut it.
In a perfect world, I would come up with hooks to handle weird exceptions, and I would also have clean running tests every time. However, this is not a perfect world, and right now, getting the tests to reliably work is my priority. Also, I want to be able to rerun tests that fail and see if they are spurious failures or if there's a real problem.
How did i do this? I made a shell wrapper that would tee the output of the console to a rerun file, and then I would capture the failures, recreate a test suite of just the failed scenarios, and run them individually. Repeat until we get a 100% clean run or we confirm that something is genuinely broken.
So what's my problem? The lovely red/green/yellow color scheme that shows me what's happening at a glance disappears, replaced with a monochrome representation of all the steps. Do the tests run? Sure? Does the rerun whittle down my errors until I get to the essential problems? Yep, it does. I just want to have my cake and eat it too, if possible.
If I run without the rerun script (i.e. sans tee):
If I run with the rerun script (i.e. with the tee):
I get that this is because the system is trying to be smart and not deal with colors because the output is going to a pipe and not to the traditional console. My question is, how can I make my system stop being so smart? the -c (force color) option doesn't cut it.
Tuesday, May 8, 2012
Learning to Tell Different Stories
| Motoko Kusanagi from "Ghost in the Shell" |
I first became familiar with Anime through shows like Kimba the White Lion, Speed Racer and Gatchaman (which we knew in the states as “Battle of the Planets”). This was my early introduction to this style, and over the years, it’s become better known and more titles have become available in the U.S. over the past few decades. Unlike in the U.S., where animation is often seen as programming for kids, Anime is developed for all ages and some is as gripping and intense as the most well produced cable television series and, in some cases, motion pictures (for those interested, some of my favorites are "Ghost in the Shell", "Cowboy Bebop", "Fullmetal Alchemist", "Neon Genesis Evangelion", "Clannad" and "Wolf's Rain").
Why am I bringing this up on a testing blog? It’s because, as I’ve been sharing these stories with my children, I’ve enjoyed answering some of their questions or finding out more about some of the questions they ask. One of the things that I have been sharing with them is the fact that Japan, though in many ways a modern industrial culture, has very different cultural traditions as compared to the U.S. Key to that are their religious and social mores. Many of the references made in these shows are things that are very familiar to us in the U.S., but there are also subtleties that are very Japanese.
We are used to many of the stories that are told in the “Western” tradition being told from the perspective of Judeo-Christian values. These values, mixed in with Greek philosophy, make up the bulk of the stories and the legends that have come down to us. The fairy tales and literature that we are most familiar with have many things that we would look at naturally; good overcomes evil, the hero saves the day, the protagonist generally achieves his goal, and they all lived happily ever after.
The stories that appear in Anime do not always flow this way. Often evil does succeed. Often the protagonist dies.The resolution of many series is left to be ambiguous at best. Frankly, I enjoy that. I like Anime because it helps me look at different cultural stories and see what is common to the human experience, but more to the point, I like seeing what is different. Some of my friends who have grown up in Japan and who indulge me in talking about some of these stories have told me that there are maybe a hundred little things in any given series that would go over the head of an everyday Western viewer. There are inside jokes, idioms, aspects of the way characters are drawn, interact with each other, look at each other, and things that we would take as insignificant quirks. To those who have grown up with it, these small quirks are actually very significant. I like being clued into these things, because I can view the shows again after several years, and I see new aspects I never knew about. They were always there, but I was not trained to see them.
Today, we as testers have similar opportunities. When we make a shift from one product to another, we also get the chance to see the stories that are universal. But we also get to see those things that are very unique to their own industry, niche, market, or worldview. If we think that knowing a bit of testing “best practices” will carry us over to all circumstances, we are very much mistaken and we will miss a lot of stuff. We have to get used to those new stories, and learn the nuances that make them unique, and in turn understand the underlying culture and how that underlying culture informs the stories. Doing that will not necessarily make you an expert in every domain, but it will definitely give you a better chance of getting closer to that goal in the domain you are currently working in.
Thursday, May 3, 2012
Cross "Training": Shaking Up the Home Office
This may either turn out to be a corny diversion or the healthiest thing I've ever done for myself. I'm not entirely sure which :).
One of the factors that I discovered when I started pouring myself into doing a lot of testing things is that my fitness levels started to drop. I would get out and skateboard when I could or ride my bike, but these activities make my wife anxious. Needless to say, my breaking my leg last year using my skateboard as a commute vehicle did not give her much to boost confidence in my doing these things, and rather than risk friction in my marital bliss, I've complied. This does, however, mean that the opportunities to get out and do structured exercise are somewhat limited. Each time I think of things I want to write, test, prepare, read, or do, I have to decide to deliberately stop what I'm doing to go get some exercise. It's one or the other... or does it have to be?
There have been a variety of stories about the "brogramming" movement (if you're curious, put "brogramming" in your preferred search engine and enjoy or cringe at what you read). Regardless of the silliness of the various memes related to "brogramming" I will admit that the attention to fitness did appeal to me. What's more, it was through this that I started to see lots of links to a variety of "treadmill desks". Kitschy? Sure, but at the same time, very intriguing. After I got over poo-poohing the ridiculousness of a treadmill desk, I remembered that one of our Sidereel founders had a treadmill desk in our old loft on Jessie Street. With this thought in my head, I asked him about it, if he still used it, and if he had any thoughts about what to do.
He made a few recommendations. First, it's easier if the treadmill in question has support arms that are parallel to the floor and at a reasonable height. From there, a piece of wood, some C clamps and you are golden. One suggestion he recommended to me was that I'd want to invest in a high end treadmill, because the entry level models would not hold up to the abuse of using it for hours each day.
With that, I made some inquiries, checked out some models, and since I wasn't entirely sold on this idea, I figured that I would look to find a refurbished higher end treadmill rather than buy one brand new. I'd hate to spent a grand plus and then find out this is a white elephant of an idea, and then the "treadmill desk" becomes a "treadmill clothes rack", and a rather expensive one at that. I met up with a guy that goes to auctions and purchases items from gyms that have gone under, then he fixes them up and resells them. Thus, I was able to get a pretty good high end, though older, Precor unit for $175.00.
The next step? Making a desktop surface for as little as possible but still be functional. That was done by taking a trip to Lowes and picking up a 15" x 36" white melamine shelf, 4 eye screws, and two 24" bungee cables. The eye screws were put in an inch from the front edge on the sides, and an inch from the sides on the back. this way, I could criss-cross a par of bungee cords from the eyes and hold them down to the cross bars and make for a solid and stable work surface. I made one additional adjustment in that I used two bath towels and rolled them up on the bars so they could act as a dampening mechanism for the machines. All in all $175 for a refurbished treadmill, $25 for the parts to make the desktop, and now I have a way to read, study, dork around on Twitter or Facebook, and rather than feel like I have to choose getting up and walking around vs. sitting down and doing work, I now have the ability to do both. The benefit to the system that I've put together is that I can remove the desktop easily and use the arm bars for a more vigorous workout if I so choose.
I gave the system its maiden voyage last night, and ended up spending about two hours walking at 3 mph or so (maybe a little slower at spots; 3mph is about the maximum speed I can maintain and still type). Up side, I got a lot done (including editing the most recent TWiST podcast while walking :) ). Down side, I'm rather sore this morning. Also, I discovered that the better way to work with the walking desk is to have one system at a time up on the desk. By having two side by side, I was positioning myself at a bit of an angle depending on the machine I was working on, and it torqued the belt a little bit. Nothing serious, but I ended up skidding the belt off to the side and needed to adjust the treadmill to get it back into regular operating order again.
So will this be the holy grail to getting me back into being fit and healthy? Will it be a silly diversion that I don't follow through with? Time can only tell on that, but so far, it seems to be something that's workable, interesting, and enough of a novelty to shake up the system. Whether it will have staying power is anyone's guess, but I'm certainly willing to give it my best try.
One of the factors that I discovered when I started pouring myself into doing a lot of testing things is that my fitness levels started to drop. I would get out and skateboard when I could or ride my bike, but these activities make my wife anxious. Needless to say, my breaking my leg last year using my skateboard as a commute vehicle did not give her much to boost confidence in my doing these things, and rather than risk friction in my marital bliss, I've complied. This does, however, mean that the opportunities to get out and do structured exercise are somewhat limited. Each time I think of things I want to write, test, prepare, read, or do, I have to decide to deliberately stop what I'm doing to go get some exercise. It's one or the other... or does it have to be?
There have been a variety of stories about the "brogramming" movement (if you're curious, put "brogramming" in your preferred search engine and enjoy or cringe at what you read). Regardless of the silliness of the various memes related to "brogramming" I will admit that the attention to fitness did appeal to me. What's more, it was through this that I started to see lots of links to a variety of "treadmill desks". Kitschy? Sure, but at the same time, very intriguing. After I got over poo-poohing the ridiculousness of a treadmill desk, I remembered that one of our Sidereel founders had a treadmill desk in our old loft on Jessie Street. With this thought in my head, I asked him about it, if he still used it, and if he had any thoughts about what to do.
He made a few recommendations. First, it's easier if the treadmill in question has support arms that are parallel to the floor and at a reasonable height. From there, a piece of wood, some C clamps and you are golden. One suggestion he recommended to me was that I'd want to invest in a high end treadmill, because the entry level models would not hold up to the abuse of using it for hours each day.
With that, I made some inquiries, checked out some models, and since I wasn't entirely sold on this idea, I figured that I would look to find a refurbished higher end treadmill rather than buy one brand new. I'd hate to spent a grand plus and then find out this is a white elephant of an idea, and then the "treadmill desk" becomes a "treadmill clothes rack", and a rather expensive one at that. I met up with a guy that goes to auctions and purchases items from gyms that have gone under, then he fixes them up and resells them. Thus, I was able to get a pretty good high end, though older, Precor unit for $175.00.
The next step? Making a desktop surface for as little as possible but still be functional. That was done by taking a trip to Lowes and picking up a 15" x 36" white melamine shelf, 4 eye screws, and two 24" bungee cables. The eye screws were put in an inch from the front edge on the sides, and an inch from the sides on the back. this way, I could criss-cross a par of bungee cords from the eyes and hold them down to the cross bars and make for a solid and stable work surface. I made one additional adjustment in that I used two bath towels and rolled them up on the bars so they could act as a dampening mechanism for the machines. All in all $175 for a refurbished treadmill, $25 for the parts to make the desktop, and now I have a way to read, study, dork around on Twitter or Facebook, and rather than feel like I have to choose getting up and walking around vs. sitting down and doing work, I now have the ability to do both. The benefit to the system that I've put together is that I can remove the desktop easily and use the arm bars for a more vigorous workout if I so choose.
I gave the system its maiden voyage last night, and ended up spending about two hours walking at 3 mph or so (maybe a little slower at spots; 3mph is about the maximum speed I can maintain and still type). Up side, I got a lot done (including editing the most recent TWiST podcast while walking :) ). Down side, I'm rather sore this morning. Also, I discovered that the better way to work with the walking desk is to have one system at a time up on the desk. By having two side by side, I was positioning myself at a bit of an angle depending on the machine I was working on, and it torqued the belt a little bit. Nothing serious, but I ended up skidding the belt off to the side and needed to adjust the treadmill to get it back into regular operating order again.
So will this be the holy grail to getting me back into being fit and healthy? Will it be a silly diversion that I don't follow through with? Time can only tell on that, but so far, it seems to be something that's workable, interesting, and enough of a novelty to shake up the system. Whether it will have staying power is anyone's guess, but I'm certainly willing to give it my best try.
Monday, April 30, 2012
WOTA (Write Once, Test Anywhere)
In some ways, this has become the Holy Grail of my automated testing existence.
In the environment that I am currently testing, I have four distinct places where I can run our code. On a development system, on a demo machine, on a staging environment, and in production. I used to maintain four different systems, and those four different systems would float and require tweaking. In an attempt to try to get some sanity and reduce wasted effort, I decided that a better, more Don't Repeat Yourself (DRY) approach was to figure out how to move my scripts to a system that allowed me to create one set of scripts, and then optimize my environments and my test data so that those scripts could be run anywhere. This is the core behind my approach of Write Once, Test Anywhere (WOTA).
Now, when I say "my approach", I certainly don't mean that I'm the first one to think to do this, not by a long shot. I'm also having to make interesting tweaks to various config files so that I can effectively do this. Unlike the development team's tests, which really only have to focus on one environment to verify that their tests work (note, that's not a dig, it's a reality), mine have to work effectively on four different environments. This also helps in the sense that it allows me to see if Acceptance Tests and the methods used to write those tests (and their surrounding features) actually carry through as we apply them to consistently more complex systems. As we get closer to production, we abstract away from running tests on a dedicated workstation and a dedicated machine, and instead start calling on multiple machines that are structured in a cluster, with caching, external and distributed databases, load balancing, etc. The test themselves don't change, but often, we see different behavior with the same tests depending on the system we are testing.
Unlike the unit tests development uses, my scripts minimize specific JavaScript calls that can be made underneath the presentation layer. I use them if I must, but I try to avoid them so that I can focus on a more behavioral approach to tests, and checking to see if what I see would be similar to what a customer would see as we walk up the chain of environments. Keeping the scenario pool relatively simple, reusing accounts where possible, and using test data and persona criteria consistently on each system helps alert me when things don't appear correctly or if we have some investigation to do on different systems. The hope is that the tweaking necessary on production is minimal to non-existent; if I've done my job right, then tests really should just run on production and there should be little in the way of hiccups.
If this piece is a blinding flash of the obvious, well, it took me a bit of time to figure out a good way to do this. If you are still testing with different bases for each environment, seriously, consider looking at ways you can implement WOTA in your tests.
In the environment that I am currently testing, I have four distinct places where I can run our code. On a development system, on a demo machine, on a staging environment, and in production. I used to maintain four different systems, and those four different systems would float and require tweaking. In an attempt to try to get some sanity and reduce wasted effort, I decided that a better, more Don't Repeat Yourself (DRY) approach was to figure out how to move my scripts to a system that allowed me to create one set of scripts, and then optimize my environments and my test data so that those scripts could be run anywhere. This is the core behind my approach of Write Once, Test Anywhere (WOTA).
Now, when I say "my approach", I certainly don't mean that I'm the first one to think to do this, not by a long shot. I'm also having to make interesting tweaks to various config files so that I can effectively do this. Unlike the development team's tests, which really only have to focus on one environment to verify that their tests work (note, that's not a dig, it's a reality), mine have to work effectively on four different environments. This also helps in the sense that it allows me to see if Acceptance Tests and the methods used to write those tests (and their surrounding features) actually carry through as we apply them to consistently more complex systems. As we get closer to production, we abstract away from running tests on a dedicated workstation and a dedicated machine, and instead start calling on multiple machines that are structured in a cluster, with caching, external and distributed databases, load balancing, etc. The test themselves don't change, but often, we see different behavior with the same tests depending on the system we are testing.
Unlike the unit tests development uses, my scripts minimize specific JavaScript calls that can be made underneath the presentation layer. I use them if I must, but I try to avoid them so that I can focus on a more behavioral approach to tests, and checking to see if what I see would be similar to what a customer would see as we walk up the chain of environments. Keeping the scenario pool relatively simple, reusing accounts where possible, and using test data and persona criteria consistently on each system helps alert me when things don't appear correctly or if we have some investigation to do on different systems. The hope is that the tweaking necessary on production is minimal to non-existent; if I've done my job right, then tests really should just run on production and there should be little in the way of hiccups.
If this piece is a blinding flash of the obvious, well, it took me a bit of time to figure out a good way to do this. If you are still testing with different bases for each environment, seriously, consider looking at ways you can implement WOTA in your tests.
Thursday, April 26, 2012
Weekend Testing on May 5th: Something Different
Most of the time, when I make announcements about Weekend Testing, I only do it the week of the event, with little lead up time, and not a lot of publicity. This is because of a number of reasons, but the main reason is to keep the group to a manageable size. Often, if we get more than 25 attendees, it's hard to communicate and threads and participants get lost in the shuffle.
This time, I am making an exception, because we are doing something with a need for a little bit of prep and perhaps a bigger crowd to do it justice.
On Saturday, May 5th, 2012 at 10:00 a.m. Pacific, we will be testing some new functionality being rolled out on Wikipedia (yep, *that* Wikipedia). Cross browser testing will be a big part of it, so in this case, the more the merrier really does apply. Also, we are in the process of creating a number of smaller charters so that several testers can branch out and try different things based on their interest. Those are being posted to the wiki (and will be shared as soon as Chris McMahon says it's OK to share it :) ).
For those not familiar with how Weekend Testing works, here's the refresher:
1. Add “weekendtestersamericas” to your Skype contacts if you haven’t already.
2. Fifteen minutes prior to the start of the session, please message “weekendtestingamericas” and ask to be added to the chat session. Once we see you, we will add you to the session.
For more details, contact WTAmericas@gmail.com. Hope to see you there!
This time, I am making an exception, because we are doing something with a need for a little bit of prep and perhaps a bigger crowd to do it justice.
On Saturday, May 5th, 2012 at 10:00 a.m. Pacific, we will be testing some new functionality being rolled out on Wikipedia (yep, *that* Wikipedia). Cross browser testing will be a big part of it, so in this case, the more the merrier really does apply. Also, we are in the process of creating a number of smaller charters so that several testers can branch out and try different things based on their interest. Those are being posted to the wiki (and will be shared as soon as Chris McMahon says it's OK to share it :) ).
For those not familiar with how Weekend Testing works, here's the refresher:
1. Add “weekendtestersamericas” to your Skype contacts if you haven’t already.
2. Fifteen minutes prior to the start of the session, please message “weekendtestingamericas” and ask to be added to the chat session. Once we see you, we will add you to the session.
For more details, contact WTAmericas@gmail.com. Hope to see you there!
Tuesday, April 24, 2012
Seeing Things The Way We Are: A "Recycled Scoutmaster's Minute"
I wrote this a few years back around the time my son became actively involved in the Order of the Arrow, a service organization that I ma also part of, with ties to Native American traditions as well as to Scouting ideals. Since I've been recalling some of my previous Scoutmaster Minutes, I thought this one would be good to include as well. I think some of the comments are also applicable to testers. See if you agree :).
Seeing Things the Way We Are (October, 2008)
This weekend, my son and I will be heading up to the Santa Cruz Mountains to participate in the Ordeal Weekend for our Order of the Arrow Lodge. It’s an event we have done together twice, and now this will be our third time together. Last year, he went through for himself to receive Ordeal membership. Last spring, he served as an Elangomat (meaning Friend or Guide) to others going through the Ordeal for the first time. While this was happening, I was in another part of the camp going through the process to receive the Vigil Honor (and no, I’ll not tell what that process is, if you want to know, join and get there yourself ;) ).
This weekend, Nick will be acting as an Elangomat again, and he has the chance to seal his membership in Order of the Arrow as a Brotherhood member. What's more, he will be an Elangomat for fellow members of his Troop, so this time, he will actually be leading his own friends through the process.
So what does the title of this post have to do with the preceding paragraph? It always interests me that we have opportunities where we can get away from it all, think, ponder, pray, meditate, and learn a little bit more about ourselves and where we fit into the world. These actions allow us to open our eyes just a little bit more, and they let us see a little more clearly what we as people need to do.
It’s been a year since my son was elected to be an Ordeal candidate. In that time, he has learned a bit more about what it means to serve and be part of a bigger group, and to contribute to the success of that group. I’m proud of him and what he has been able to do in a short time. By contrast, my own involvement over the last year has changed somewhat with the receiving of the Vigil Honor. With it comes a greater expectation, and through that expectation, I’ve determined that I need to be more aware and alert to the things that I need to do and the example I need to set.
As an active member of my church, I am often instructed and counseled to read my scriptures daily. Oftentimes, I have had to ask myself “why am I being asked to rehash things I’ve already read a bunch of times before?” I’ve come to see that it is because there really is no one world, one absolute reality, but billions of them, and each reality is informed by the viewpoint and the vision of the individual living that reality. Unlike a stone crag that juts out into the ocean, impervious to all that buffet it, human beings are really very small boats that have very lightweight anchors. We get blown about all over the place. Likewise, we are also very swift and maneuverable; we can change course very easily and maneuver very quickly and with great agility in even the most treacherous of areas. Thus the words that we are taught, and the counsel that we always seek those words, is inspired because of the fact that we are such lightweight and agile but easily blown about vessels. We need to be reminded where our home port is, and we need to be reminded of what the windows on the bridge are supposed to see.
What's more, there is a lot of our world that is hidden from us because we are not ready, willing or able to see it for what it is. Those moments of clarity take time to develop, and they often come about because we've discovered that something we thought worked well for us really doesn't. If we do not question and ponder the things we read, but just accept them at face value, we also stagnate, and may not even realize that the world has changed under our feet, and we were too slow to observe and realize that the world had changed.
This weekend, a new batch of boys will be coming up to find out a little bit more about who they are and how they see the world. Here’s hoping it will be a good experience for them, and for us as well.
Seeing Things the Way We Are (October, 2008)
This weekend, my son and I will be heading up to the Santa Cruz Mountains to participate in the Ordeal Weekend for our Order of the Arrow Lodge. It’s an event we have done together twice, and now this will be our third time together. Last year, he went through for himself to receive Ordeal membership. Last spring, he served as an Elangomat (meaning Friend or Guide) to others going through the Ordeal for the first time. While this was happening, I was in another part of the camp going through the process to receive the Vigil Honor (and no, I’ll not tell what that process is, if you want to know, join and get there yourself ;) ).
This weekend, Nick will be acting as an Elangomat again, and he has the chance to seal his membership in Order of the Arrow as a Brotherhood member. What's more, he will be an Elangomat for fellow members of his Troop, so this time, he will actually be leading his own friends through the process.
So what does the title of this post have to do with the preceding paragraph? It always interests me that we have opportunities where we can get away from it all, think, ponder, pray, meditate, and learn a little bit more about ourselves and where we fit into the world. These actions allow us to open our eyes just a little bit more, and they let us see a little more clearly what we as people need to do.
It’s been a year since my son was elected to be an Ordeal candidate. In that time, he has learned a bit more about what it means to serve and be part of a bigger group, and to contribute to the success of that group. I’m proud of him and what he has been able to do in a short time. By contrast, my own involvement over the last year has changed somewhat with the receiving of the Vigil Honor. With it comes a greater expectation, and through that expectation, I’ve determined that I need to be more aware and alert to the things that I need to do and the example I need to set.
As an active member of my church, I am often instructed and counseled to read my scriptures daily. Oftentimes, I have had to ask myself “why am I being asked to rehash things I’ve already read a bunch of times before?” I’ve come to see that it is because there really is no one world, one absolute reality, but billions of them, and each reality is informed by the viewpoint and the vision of the individual living that reality. Unlike a stone crag that juts out into the ocean, impervious to all that buffet it, human beings are really very small boats that have very lightweight anchors. We get blown about all over the place. Likewise, we are also very swift and maneuverable; we can change course very easily and maneuver very quickly and with great agility in even the most treacherous of areas. Thus the words that we are taught, and the counsel that we always seek those words, is inspired because of the fact that we are such lightweight and agile but easily blown about vessels. We need to be reminded where our home port is, and we need to be reminded of what the windows on the bridge are supposed to see.
What's more, there is a lot of our world that is hidden from us because we are not ready, willing or able to see it for what it is. Those moments of clarity take time to develop, and they often come about because we've discovered that something we thought worked well for us really doesn't. If we do not question and ponder the things we read, but just accept them at face value, we also stagnate, and may not even realize that the world has changed under our feet, and we were too slow to observe and realize that the world had changed.
This weekend, a new batch of boys will be coming up to find out a little bit more about who they are and how they see the world. Here’s hoping it will be a good experience for them, and for us as well.
Monday, April 23, 2012
The Pen Game: Software Testing Encapsulated
While I was at STAR East in Orlando last week, I had a chance to spend a fair amount of time with fellow testers outside of track sessions or after conference hours. In fact, I don't think I got to bed before 2:00 a.m. Eastern once the entire time I was there. With an admission like that, you'd be forgiven for thinking that this was a hard partying bunch, but that isn't the case. In fact, much of our time was spent talking philosophy, the variety of practices out there that could be considered "good" and improving the ways that we talk about testing. It was during this session that I was introduced to "The Pen Test".
The Pen Test is a simple game, where a "presenter" or product owner holds up a pen and repeats some questions. There are only two possible answers; Yes or No. The "tester's" goal is to figure out what makes the answer Yes, and what makes the answer No.
If you think I'm going to give you the answer, you don't know me very well (LOL!). If you want to play the game, find someone who knows it and ask them to walk you through it. The game itself isn't the point of the post.
What I found interesting about the game was the fact that it helped me explain software testing and the actions we subconsciously perform in a much easier way. We start out with a program (being presented the pen). We are shown behavior (the words and actions to display the pen), and based on that behavior, we need to determine what "state" the program is in. Sometimes we can guess, and do very well. but there's a danger when we guess and get it right many times. We create a mental model that may not be accurate, and when something goes wrong, we are at a loss as to why.
I had this experience when this was presented to me. Not by skill, but by luck, I was able to guess the correct answer nine straight time. It was on the tenth try that I got it wrong, and then had to regroup and figure out again what the criteria for Yes/No could be.
For this game to be successful, the testers need to weed out as many non essential aspects of the game as they can. There are many aspects that we can see, and these aspects may or may not have any bearing on the trigger that makes the answer Yes or No. The effective tester uses as many tools at their disposal to weed out as many of these options as possible. We create a hypothesis, and then we test the hypothesis. If it holds up, we continue pressing, but if it doesn't, we should discard the model we have created (or at least the assumptions that underlie it) and try something different. Often, through this process, we are able to notice subtle differences, or eliminate things entirely from consideration.
During the game, the product owner is able to answer any of our questions, provided the answer isn't "write down what you say" or "tell me the answer". This is much like a black box level of testing. We have to determine the answers based on the behavior of the application. Through this process, we try out a number of different heuristics. They may work, they may not, but each time we hit a dead end, we have another variable that we can remove, and that, over time, gets us closer to a solution.
I had several conversations with people over the course of the week, and tried this out with a number of different people. The ability to have a conversation about what worked and what didn't helped greatly in explaining the way that we make assumptions, try out models and test their ability to be effective, discard theories that don't work, and keep honing our process until we nail down the correct item(s) that make for a Yes/No situation. The challenge, of course is that once a game gets well known, then it loses its effectiveness, because we focus in on the answer. This game, at least for right now, requires a lot of questions and inquiry to answer, so I think it's worth using as an exercise for the time being. If it gets too well known, I'll look for others. That's one thing I know that I never have to worry about; there's a lot of games that can be applied to software testing; so many that any given tester is unlikely to have seen or worked through all of them.
Friday, April 20, 2012
STAREAST Day 2 Recap
I have to admit, it's a challenge to separate the fact that I am on East Coast time from the realities of waking up at 7:00 AM each morning, which is actually 4:00 AM where I'm from. I do that from time to time, but not with such late evenings. Still it's been a great deal of fun to hang out with such a great group of people and to have such wonderful, serendipitous moments.
The day started out with breakfast and a keynote address from Dorothy Graham of Grove Consulting about "What Managers Think They Know about Test Automation–But Don't". Dorothy points out that often, Management is looking for the following aspects to happen when they talk about test automation. They believe:
James Bach gave an excellent talk on "The Dirty Secret of Formal Testing". What made this talk interesting was the fact that he was describing testing in highly regulated markets, dealing with a product that was very life or death (a medical device to help aid patients with heart problems). These environments are extremely formal, but the dirty secret of these environments is that informal testing goes on all the time. It has to. Otherwise there's no way to learn about what the product actually does. As described by James, formal testing is any testing that must be done a specific way or must check specific facts. Requirements as described by James (and I like this description) are ideas at the intersection between what we want, what we can have, and how we MIGHT decide that we got it. In complex and life-or-death systems, we have to test to find out if that has happened, and we have to test first to find out if we actually accomplished those goals. Thus, contrary to popular belief and conventional wisdom, good formal testing MUST first begin with informal testing.
In addition to the variety of sessions, I have to point out that many times, the best discussions and best ideas happen between the sessions, or happen when we decide to engage in conversations with a single person or a handful of people, effectively creating our own session on the spot. I confess, I had a hidden motive with coming to STAREAST beyond giving my talk and learning some cool new ideas to bring home. My goal was to see what I could learn from many different people about how to teach software testing, and make it fun, for 16-24 year olds. More to the point, I needed to recruit helpers willing to do the work, and I was not disappointed. Instead of me looking for people to discuss this idea, word preceded me, so I had people coming up to me to ask if they could take part. Based on the interactions I had with many people this week, I got what I came for and plenty more. I had some terrific discussions with people from many industries as well as educators who have helped me narrow down and distill some of the ideas I have, and it's given me many new directions to consider. They may not all be possible, but I'm looking forward to trying them out.
STAREAST Virtual was a fun and interesting way to communicate with the audience of participants that were not specifically at the conference, and it was fun to have Matt Barcomb participate with me in a conversation about Testing Roles and Approaches to Paired Development and Testing. In addition, we also discussed many of the aspects of collaboration and ways to break down the unintentional walls that we build. In this way, we can set up our environments and approaches so that we can become more effective.
The main conference came to a close with the awarding of a number of awards, including those who found interesting bugs in the Test Lab, and something that made me smile greatly. It was announced to everyone just before the closing keynote that my paper and presentation were chosen as Best Paper for the conference :). For someone who gave his first official full hour-long track talk at any conference, well, yesterday, that was a wonderful thing to see happen (I rank it up there highly with "firsts" of my career :) ). I appreciate the award and those who felt I deserved it. Thank you very much!
The final keynote of the conference was given by Theresa Lanowitz of voke, inc., and it discussed "Testing Trends: Cloud, Virtualization, and Mobility". It gave a forecast of what we might see in the next two to three years, and I must say, things look exciting for testers, regardless of what the "test is dead" people are saying :). Numerous examples of performance problems, unintended consequences, security breaches and other issues of integration were explored, and they all pointed to the same thing. Testing needs to be performed, and it needs to be performed at a level that people can make solid and intelligent decisions about the ramifications and consequences. The simple fact is that there are no end of authentic problems to be tackled and the scope and the landscape are expanding. There will come a time when we will need more testers, not less. Get ready, folks :).
My thanks to so many people for the time they took to get to know me, speak with me, hang out in the hotel lounge or go out to dinner with me and talk about ideas and concepts that we all deal with and the solutions we hope to achieve. My thanks to Scott Barber, Claire Moss, David Gilbert, Matt Barcomb, Lee Copeland, James and Jon Bach, Lanette Creamer, Rachelle Sawal, Janet Gregory, Allison Wade, Matt Barcomb, Randy Rice, Mirkaya Capellan, James Lyndsay, Michael Bolton, Zeger Van Hese, Griffin Jones, all the attendees of my talk about Weekend Testing, and everyone else that helped make this a memorable week.
'Til we meet again :).
The day started out with breakfast and a keynote address from Dorothy Graham of Grove Consulting about "What Managers Think They Know about Test Automation–But Don't". Dorothy points out that often, Management is looking for the following aspects to happen when they talk about test automation. They believe:
- automation can test everything
- automation will find all the bugs
- with automation we will get test coverage of 100%
- more automation means more confidence
- run tests overnight and weekends
- automation will prevent human error
James Bach gave an excellent talk on "The Dirty Secret of Formal Testing". What made this talk interesting was the fact that he was describing testing in highly regulated markets, dealing with a product that was very life or death (a medical device to help aid patients with heart problems). These environments are extremely formal, but the dirty secret of these environments is that informal testing goes on all the time. It has to. Otherwise there's no way to learn about what the product actually does. As described by James, formal testing is any testing that must be done a specific way or must check specific facts. Requirements as described by James (and I like this description) are ideas at the intersection between what we want, what we can have, and how we MIGHT decide that we got it. In complex and life-or-death systems, we have to test to find out if that has happened, and we have to test first to find out if we actually accomplished those goals. Thus, contrary to popular belief and conventional wisdom, good formal testing MUST first begin with informal testing.
In addition to the variety of sessions, I have to point out that many times, the best discussions and best ideas happen between the sessions, or happen when we decide to engage in conversations with a single person or a handful of people, effectively creating our own session on the spot. I confess, I had a hidden motive with coming to STAREAST beyond giving my talk and learning some cool new ideas to bring home. My goal was to see what I could learn from many different people about how to teach software testing, and make it fun, for 16-24 year olds. More to the point, I needed to recruit helpers willing to do the work, and I was not disappointed. Instead of me looking for people to discuss this idea, word preceded me, so I had people coming up to me to ask if they could take part. Based on the interactions I had with many people this week, I got what I came for and plenty more. I had some terrific discussions with people from many industries as well as educators who have helped me narrow down and distill some of the ideas I have, and it's given me many new directions to consider. They may not all be possible, but I'm looking forward to trying them out.
STAREAST Virtual was a fun and interesting way to communicate with the audience of participants that were not specifically at the conference, and it was fun to have Matt Barcomb participate with me in a conversation about Testing Roles and Approaches to Paired Development and Testing. In addition, we also discussed many of the aspects of collaboration and ways to break down the unintentional walls that we build. In this way, we can set up our environments and approaches so that we can become more effective.
The main conference came to a close with the awarding of a number of awards, including those who found interesting bugs in the Test Lab, and something that made me smile greatly. It was announced to everyone just before the closing keynote that my paper and presentation were chosen as Best Paper for the conference :). For someone who gave his first official full hour-long track talk at any conference, well, yesterday, that was a wonderful thing to see happen (I rank it up there highly with "firsts" of my career :) ). I appreciate the award and those who felt I deserved it. Thank you very much!
The final keynote of the conference was given by Theresa Lanowitz of voke, inc., and it discussed "Testing Trends: Cloud, Virtualization, and Mobility". It gave a forecast of what we might see in the next two to three years, and I must say, things look exciting for testers, regardless of what the "test is dead" people are saying :). Numerous examples of performance problems, unintended consequences, security breaches and other issues of integration were explored, and they all pointed to the same thing. Testing needs to be performed, and it needs to be performed at a level that people can make solid and intelligent decisions about the ramifications and consequences. The simple fact is that there are no end of authentic problems to be tackled and the scope and the landscape are expanding. There will come a time when we will need more testers, not less. Get ready, folks :).
My thanks to so many people for the time they took to get to know me, speak with me, hang out in the hotel lounge or go out to dinner with me and talk about ideas and concepts that we all deal with and the solutions we hope to achieve. My thanks to Scott Barber, Claire Moss, David Gilbert, Matt Barcomb, Lee Copeland, James and Jon Bach, Lanette Creamer, Rachelle Sawal, Janet Gregory, Allison Wade, Matt Barcomb, Randy Rice, Mirkaya Capellan, James Lyndsay, Michael Bolton, Zeger Van Hese, Griffin Jones, all the attendees of my talk about Weekend Testing, and everyone else that helped make this a memorable week.
'Til we meet again :).
Thursday, April 19, 2012
Do Or Do Not - A "Recycled Scoutmaster's Minute
This will be my last of these for awhile. With this I have exhausted the ones' that I wrote and posted for the blog. While I will definitely revisit this idea again later, they will not be "recycled Scoutmaster's Minutes", but up to date ones, I hope. Again, this one came out of my pondering some things I was reading by Larry Winget (I went on a Winget tear in 2009 and read five of his bocks in short order.
I think it's important to realize that, when we talk to scouts (or testers), these are kids that can generally take some "tough love", and actually thrive on it. That was an important lesson to learn. Make high expectations and you should not be surprised when people meet those expectations. Set low expectations, and the same thing applies :).
DO OR DO NOT!
While I was reading Larry Winget's book "It's Called Work for a Reason", I found a statement that I felt was very applicable and something common enough that everyone would be familiar with it.
"WHEN SOMEONE SAYS THEY WILL TRY, BET YOUR MONEY IT WON'T HAPPEN"
Whenever someone tells you that they will try something, they have given themselves an out. If they don't do what they said they would do, then they can always come back and say "oh well, at least I tried".
Personally, I would encourage every boy to purge the word "try" from their vocabulary. Try does not have the sense of commitment that the word "do" has. If I were to say to a Scout "I will try to hold a board of review for you after you have finished all of your requirements" they would be rightly upset with me if I did not follow through, but still, I can always fall back on those tired words "well, I tried" (notice how similar the words “Tired” and “Tried” are? I don’t think that’s an accident :) ).
There is a scene in the Star Wars film "The Empire Strikes Back" that always resonated with me. It's where the character of Yoda is speaking with Luke Skywalker and Yoda asked Luke to do a particularly hard task. When Luke answers that he will try to do it, Yoda's answer is direct... "DO, or DO NOT... there is no TRY!"
The reason this word is so popular is that we as people want to shield ourselves from failure. If we say that we will try to do something, and it didn’t happen, we didn't really fail, right? We just tried it and decided it wasn't for us. It's comforting to say it that way, but often it rings hollow. It is better to say "I intended to do this, but I found that I ran out of time and could not do it" or "I did the thing I was asked to do, and discovered that I was not prepared or not physically able to accomplish the goal". There's a lot more meat to those words, isn't there? When we do something, there is not always the guarantee of success. We may succeed, we may fail, but no matter what, we DO SOMETHING ABOUT THE SITUATION. If we succeed, then we enjoy the accomplishment. If we fail at it, we learn why we failed, adjust our approach towards the goal, and do it again. It's my belief that those who keep doing, even when they fail, will eventually succeed... or they will determine that the task at hand is one that they wish to no longer do and they will set it aside. Either way, they have DONE something about it!
So boys, when you are asked by life to accomplish goals and to do the work that you need to do, commit to DO IT, or to NOT DO IT. Either is fine, but no more trying. History honors the doers of the world. Be a doer :).
I think it's important to realize that, when we talk to scouts (or testers), these are kids that can generally take some "tough love", and actually thrive on it. That was an important lesson to learn. Make high expectations and you should not be surprised when people meet those expectations. Set low expectations, and the same thing applies :).
DO OR DO NOT!
While I was reading Larry Winget's book "It's Called Work for a Reason", I found a statement that I felt was very applicable and something common enough that everyone would be familiar with it.
"WHEN SOMEONE SAYS THEY WILL TRY, BET YOUR MONEY IT WON'T HAPPEN"
Whenever someone tells you that they will try something, they have given themselves an out. If they don't do what they said they would do, then they can always come back and say "oh well, at least I tried".
Personally, I would encourage every boy to purge the word "try" from their vocabulary. Try does not have the sense of commitment that the word "do" has. If I were to say to a Scout "I will try to hold a board of review for you after you have finished all of your requirements" they would be rightly upset with me if I did not follow through, but still, I can always fall back on those tired words "well, I tried" (notice how similar the words “Tired” and “Tried” are? I don’t think that’s an accident :) ).
There is a scene in the Star Wars film "The Empire Strikes Back" that always resonated with me. It's where the character of Yoda is speaking with Luke Skywalker and Yoda asked Luke to do a particularly hard task. When Luke answers that he will try to do it, Yoda's answer is direct... "DO, or DO NOT... there is no TRY!"
The reason this word is so popular is that we as people want to shield ourselves from failure. If we say that we will try to do something, and it didn’t happen, we didn't really fail, right? We just tried it and decided it wasn't for us. It's comforting to say it that way, but often it rings hollow. It is better to say "I intended to do this, but I found that I ran out of time and could not do it" or "I did the thing I was asked to do, and discovered that I was not prepared or not physically able to accomplish the goal". There's a lot more meat to those words, isn't there? When we do something, there is not always the guarantee of success. We may succeed, we may fail, but no matter what, we DO SOMETHING ABOUT THE SITUATION. If we succeed, then we enjoy the accomplishment. If we fail at it, we learn why we failed, adjust our approach towards the goal, and do it again. It's my belief that those who keep doing, even when they fail, will eventually succeed... or they will determine that the task at hand is one that they wish to no longer do and they will set it aside. Either way, they have DONE something about it!
So boys, when you are asked by life to accomplish goals and to do the work that you need to do, commit to DO IT, or to NOT DO IT. Either is fine, but no more trying. History honors the doers of the world. Be a doer :).
Subscribe to:
Posts (Atom)


