My Current Dilemma: Randomizing Cucumber Tests

I'm working on a situation where I am trying to describe how to balance out activities such as Acceptance Test Driven Development, GUI automation and exploratory testing, and explaining where we can take ideas from each and work with them to help us get a balance between the three. This is the subject of a talk I'll be giving and am currently writing (which also explains my being more quiet than usual on here :) ).

One of the ideas I discuss, and that I actually do, is that I put a little bit of "What if" into my acceptance level tests. I have a suite of tests that I run to check and verify basic functionality for every build that we make and push to our respective machines (development --> demo --> staging --> production). All of my tests are written so that they can run on any of these environments. These are Cucumber tests, running on top of Ruby in a Rails environment.

One of the limitations about running Cucumber as an acceptance testing tool is the fact that the tests all run in alphabetical order in the feature directory. This is assuming you run your tests using rake; my setup is configured so that I can run a suite of tests with "rake cucumber:[@tagName]:[machineName]". For some added analysis and review, I currently have a bash wrapper script that allows me to tee the output to a log file, parse the log file for errors, set up a new suite of tests, and then rerun them. The goal is to get to where I can run these tests in a single pass and have no errors (legitimately, of course :) ). Usually, though, that doesn't happen. I typically have to make three passes before all of the tests pass and there's no more errors to parse in the log file.

In a pinch, that's OK, but the nerd in me doesn't like the fact that I have to tweak things like this. This caused me to start asking "What if"... What if there's a dependency I'm not aware of? What if there's something about the way that the tests are run and the way that we authenticate certain accounts that might leave "rat droppings" in the state condition? What if I tweaked with the order of the tests? What if I totally randomized the test order every run?

I've been able to address the first three, but the fourth has been a bit of a mystery. How can I set up a process where, without going in and renaming directories or files every time, can I make it so that the sixty or so scenarios tests that I run are always in a different order? I've seen different ideas, but most of them require making a weight for tests (which isn't really random), or setting up some kind of  permanent table with file name and line number to designate the test to be run (again, has to be manipulated each time, and adding tests creates overhead), or using bash's $RANDOM option, but again, this is getting me into doing bash gymnastics, which isn't really my goal here.

So I ask my fellow testing friends, especially those who use Cucumber and Rails, what do you do?

TWiST turns 100!!!

It's been an interesting ride. Over true past two years, I've had the opportunity to produce, sequence, edit, and format Software Test Professional's 'This Week in Software Testing" podcast. We've changed up the format a few times, explored a number of different avenues, and seen our collective fortunes change considerably from when the idea was first proposed back in 2010, and when I signed on to actually produce the show.

Some things I've discovered along the way:

1. Perfection is asymptotic and impossible to achieve, but just a little bit of clean-up can reap tremendous benefits. The problem comes when we try to go beyond the basics. Stray "ums" and long spots of dead air are easy to achieve, and make a big difference when they are removed, but doing more advanced editing (re-sequencing, cutting phrases to make a more coherent flow, etc.) is really hard to get right. Voice inflection has a rhythm and a flow, and when that rhythm is broken, it's noticeable, possibly even more noticeable than the stray ums or stutters would be. Thus I've learned that, unless it's necessary (audio drop-outs or other actual "damage" or "distraction in the recording), "leave well enough alone" is a good rule of thumb.

2. Remote podcasts are the biggest headache. On one hand, it's great to get recordings of live environs that would be lost to time otherwise, but in some ways, it's tricky because it's hard to get a room balance unless you can plug into a sound board to record. Sometimes that's easy, sometimes not. Usually, Matt or I bring a portable MP3 recorder and we place it in a spot where it can pick up the audio. Sometimes we can extend them from a tripod or a stalk to get a live room feed, but more times than not, the answer is to place the recorder on the podium or table. Positive, it gets the center of the room and when the speaker is lined up, it's crisp and clear. If they pace the room (common with live speakers), we lose them when they are off-axis. Most frustrating is when they "palm the table" or place their hands on the podium. Yep, the recorder picks all that stuff up. If I'm lucky, they are between breaths or at pauses, meaning I can filter them out. If they are during their speaking, and it's an important point, there's nothing I can do but leave it in there and let the bumps and noises be.

3. We do our group meeting calls on Skype, and most of the time, everyone comes in at a slightly different volume. That means that I have to do separation and mixing (really time consuming), or I do track segment amplification, trimming, leveling and normalization. While I haven't gotten to the point where I can do a totally automated "clean and level" with one take, it's gotten to the point where, after I do a course re-leveling of the audio and get most of the audio into a rough volume range, then I can run a number of scripted actions on the file and get the sound as level as possible. This used to take me 30 minutes on a given podcast. I've shortened this down to about ten minutes now.

4. I've experimented with five microphones since this process started, and I've concluded that the Blue Snowball and the Blue Yeti make for the best podcast microphones. They sound warm, they allow for different configurations, and the Snowball in the "Ringer" just plain looks cool. It reminds me of doing a radio show in the 1950s. Also, along with using the Blue Snowball mic, I have decided that 4:00 a.m. really is the best time to record my voice. It sounds way deeper and more sultry (like a real announcer) at that time of day, so it's become a habit to just record my vocal part then when possible. Later in the day, I'm a lot more high pitched and reedier.

So what do we have in store for the next 100 shows? I'm guessing a lot of topics that you all will find interesting, a mix of presentation styles, and hopefully an evolving and cleaner presentation, one that you will enjoy listening to for the next 100 episodes, or more :).

Book Review: Poke the Box

As someone who has quoted and talked about Seth Godin quite a bit here, it's taken me a bit of time to get to "Poke the Box". It's a short volume, so it doesn't take a lot of time to read, but it does offer some interesting and quick insights, encapsulated in the format that Seth does so well on his blog.

The entries are short, and really, they can be read in any order. Picture it as sort of a "Lessons Learned in Software Testing" for your motivation and emotional focus. The metaphor for the book is a story about a friend of Seth's who was an engineer, and who made a large electric box that had lights, switches and buzzers that would do different things when played with. He stuck this big box into a  baby's crib. What did the baby do? He played with the knobs and the switches, he saw the lights and heard the buzzer. He laughed as he played and interacted with it. In short, he "Poked the Box".

It's a long form way to say "you have permission to experiment. You have permission to not be ordinary. You have permission to take control of your goals and your dreams, but once you do, you have to actually do something with it."

"Poke the Box" is less a cohesive book than it is a series of exhortations and pep talks. Again, its format allows you to skip around and read the sections that interest you at any given time. You can pop it open to any page and read something that will get you fired up. That's the intent, and Seth has the passion and the drive to get that across. His main goal is to get people to start things, work at it, fail early and often, and learn from the mistakes and try again. Much like in Linchpin, he focuses on the language of the Lizard Brain and the Resistance, two concepts that were key to the material in that previous book. They are back in force here, and they are meant to help give everyone the motivation to shout them down and prevail over them.

A good quote to sum up the purpose of Poke the Box:

"Please stop waiting for a map. We reward those who draw maps, not those who follow them."

Test First, Then Lesson

We are getting close to the make or break point for the educational initiatives we have been planning for SummerQAmp. As I've been working through a lot of material, ideas, and suggestions, I've been trying to think about the proper order of what we can use and how to apply it. Yet no matter how we massage the data, no matter how we put the modules together, no matter what rosy scenarios we paint, there is one undeniable fact… we have no idea how this is going to go over until we sit some 16-24 year olds down and try this out.

My prevailing theory, and the prevailing theory of most of the people I have talked to on this front, is that we are putting the cart before the horse in much of the training that we have developed to date. It works good for those people who have experience with doing testing, but not so well for those who don't have experience or don't really see how to connect the dots. 

It's with this in mind that I am trying to create modules that will, effectively, allow people interested in software testing to look at a number of activities, consider them, poke at them, see what happens. At the end of those activities, we discuss what they did, what they learned, and then fill in the details of what they have been doing. Rather than go through and discuss exploratory testing, let's give them an application to explore, fumble around with, "poke the box" (to borrow a Seth Godin metaphor) and otherwise have a play at the application. Then let's discuss all of the things they did, didn't do, and how what they did all maps to in that "theoretical" space.

We're coming down to the wire and it's time to see if we have something that will keep their attention. We'll know soon enough, I guess :).

Doing the Unstuck

Recently, I came to a conclusion that was unavoidable. I was paralyzed by too many things, too many commitments, and too many good intentions that I just couldn’t get any traction on. I am frequently the victim of “analysis paralysis” because so many things end up getting clogged and feel like they are impossible to make any real movement. This makes me feel more frustrated, and instead of making forward progress, I actually slip back and get even more mired. In short, I get stuck in my own psychic mud.

I find it most aggravating when I know someone else is waiting on something from me. What often happens is, I know we have something I need to do, I know I want to get it done, but I don’t really have the time to get it done, or so I think. What happens? I psychologically start distancing myself from that person or group, I start avoiding what I need to do, and thus I get farther and farther behind, and the vicious cycle continues.

Ah, but I’m not here to talk about the vicious cycle, I’m here to talk about breaking it. For me, the easiest way to do it is to include the person or group in the very thing I am stuck in and make sure they are part of the conversation. For me, recently, Skype has proven to be my best piece of “psychic WD-40”. How so? Instead of sitting around waiting for a problem to fester, I instead use Skype to reach out to the person or group that’s not getting the attention, ask them to work with me to set a time (or better yet, right then and there, if they can) and help me get unstuck.

For many people, this is a really hard step. Why? It means we have to admit that we are wrong, or that we are lacking in self-management or time-management skills, or some other deficiency that we have not been able to resolve. It means we have to admit to our fallible humanity. To all of that, my best recommendation is to say to yourself and everyone else… so what? I’m human, I’m fallible, I make mistakes, and occasionally, I bite off more than I can chew. Sometimes it comes down to realizing we don’t know something we think we should, and we will expose ourselves as being foolish or stupid. Well, guess what? We all are ignorant, foolish or stupid at some point.

Freely admitting it, and getting help from those who we are supposed to be doing something for is the best way to break that log jam. It has two major effects. First, it allows you to focus on the issue, and second, it allows the person or group who you are doing the service for to help make sure you are working with the correct information or understanding to be successful. Ultimately, people don’t care if you get it right the first time, as long as you get it right in a reasonable time frame and help solve the mutual problem.

So if you find yourself on the hook for something, and you feel stuck in trying to deliver it, stop… breathe, and then get on Skype, or chat, or pick up the phone, or whatever immediate tool is at your disposal (key word here is "immediate"), and work with that person to get unstuck. Sure, your pride will take a little beating, but seriously, would you rather be proud, or would you rather be done?

Hairballistics: An #AU4H and #WeekendTesting Introspection

We're back again at Agile Up 4 Here, and we were furiously getting things ready for presentation at what would be Weekend Testing Americas session #28. For those who didn't see my post yesterday, Agile Up 4 Here (#AU4H) is a code retreat that goes for one week. Elisabeth Hendrickson hosts it in her Agilistry Studio in Pleasanton, CA, and participants come from all over (Elisabeth and I being the most local; the other participants this time around are from the midwest, the East coast, Germany and Sweden).

The week started with a clean slate, no code and no idea what they would create. As the week progressed, the team decided to make a simple JavaScript based game called "Hairballistics". Imagine shooting a cannon, but in this case, the cannon is a cat and the ballistic in questions is a (wait for it...) hairball!

Our app, in real time.

That's what we have been up to during the week, and I got in on the action on Wednesday, going in and exploring the deployed application and features as they can available. What's more, we had a chance to work as a group and focus on the issues as a group and work them in real time up on the screen.

...and you thought *your* monitor was big!

there's been a mild sense of urgency today, since we want to make sure that we get as much wrapped up, committed and tested internally as possible before our deadline of 2:00 PM, since that's when the Weekend testing session will begin. What's going to be interesting about the "weekday" Weekend Testing session is that I'll be doing it up on the projector with the full development team looking, reacting, learning and making changes based on feedback. I think this is a first for Weekend Testing, in that an entire development team will be watching and reacting/learning in real time from  a weekend testing session (it's unique for Weekend Testing Americas, in any event).

2:00 PM came around and even with the short notice, we had about 12 participants join us and test the Hairballistics game. We had a fun session, with a lot of confusion, frustration, discovery, understanding, and feedback (lots of feedback) based on what was in the game at the time of delivery. We were able to cycle through and fix a number of the issues in real time and push two releases during the testing session. We also had a discussion at the end about the variety of testing possibilities that were available to the testers, even with a relatively simple game. We shared the github code base with the testers, and some availed themselves to the opportunity of checking out the code and seeing if they could manipulate the environment to better test certain sections.

The board below shows the issues that are/were in play during the testing session. Cards with pink tags on them are those that were either found by Weekend Testing participants or confirmed by them.

The working board. Pink tags denote issues Weekend Testing discovered or confirmed areas.

This was an interesting experience to do a Weekend Testing session with a full development team in earshot of everything, as well as having the product owner actively participating during the session. It gave me a chance to see better into the application under development, and to also help facilitate and understand the challenges the testers were facing as they worked through the product and tested various areas.

Your host for today.

A Little View Into "Agile Up 4 Here"

Today and tomorrow i will be getting the chance to do something really cool. I get to join an Agile team for a brief part of a code retreat and an active master's session on working with agile teams, Agile testing, and rapid software development.

Elisabeth Hendrickson of Quality Tree Software and Agilistry Studios has a group of people come together each year and participate in a code retreat called "Agile Up X Here" (as you might guess, the first one was called "Agile Up 2 Here" and this is the third iteration of this retreat. We have a number of cool people participating in this event, ranging from product owners, UX designers, programmers, and yes, in my case, dedicated testers.

One team, two tables and just a few days.

Our focus is to create from scratch a simple web game using HTML 5 and JavaScript. We've had a focus of starting from nothing and creating stories, defining acceptance criteria, doing test driven development, pulling stories from the board and working on them, and yes, testing! Today has been interesting in that I've spend the better part of the day directly pairing with one of the developers in a "driver/navigator" capacity. While he wrote code, I had the chance to consider areas we should test, as well as performing exploratory tests on other modules as they are delivered. We also expressed a profound propensity for spontaneous silliness, which made the entire experience a great deal of fun.

Me and Alex Bepple automating the deployment of the app.

For those who want to get in on the fun with us tomorrow, we will be holding a weekday edition of weekend Testing. It will be held tomorrow (Thursday, May 24, 2012) at 2:00 PM Pacific. For this session, we have a simple mission and charter to share: "Be prepared to share your approach to testing what may appear to be a very simple application. Simple does not mean little needs to be done. Single testers, pairs and teams can present their charters, what areas they decided to test, and what they saw along the way. If you are interested in participating, please contact "weekendtestersamericas" on Skype and tell me you would like to join us tomorrow.

My thanks to  Matt Barcomb, Kevin Baribeau, Alex Bepple, Jaime Camphorst, Elisabeth Hendrickson, Dave Liebreich, Rickard Lindberg, and Zee Spencer for participating in this great event. I'm hoping the Weekend Testing session will prove fruitful and successful, and  I look forward to having more opportunities to participate in events like this.

Kindle Fire: My Favorite Non-Network Device

Some of you may be wondering what I mean by this. Of course a Kindle Fire can be networked, it's got a built-in WiFi interface. Yes, but that's not what I mean. Instead, it's now become my favorite way to read both technical and non technical books.

Below is my typical setup at work:

Using a Kindle Fire as a standalone documentation window.

The Kindle Fire is the device sitting just below my large monitor. Why do i have it set like this? It makes for an excellent place to read something technical and place it right at eye level when I'm working. It makes it easier for me to see and compare what is on the page and what I am actively working with. It lets me shift from book to book without my having to clutter up my physical desk or my computer screen.

While those are all good, the best benefit actually turns out to be what I now call the "Zed Shaw Effect" or "Hard Way Effect" (ZSE, HWE, sorry, there's no cool acronym there). What are those? The fact that I can see the text, I can read the text, I can ponder the text, but I cannot copy and paste the text. From working through Learn Ruby the Hard Way, I am now a firm believer that, if you want to learn a technical concept, you have to do it, not just read it, and seriously, you need to type that sucker out. That means making typos. It means cussing a little under your breath. It means re-doing what you thought you did correctly. It means you actually learn it by struggling a bit with it.

There's lots of reasons why a Kindle Fire may be a good investment for you (or hey, fill in the blank with your favorite tablet, the Fire just happens to be what I have). This, above, is my #1 reason :).

Losing the Plot?

There's a long standing statement that we are all familiar with:

"Give a man a fish and he eats for a day, teach a man to fish and he can feed himself for life."
--Confucius

The part that often goes missing from this statement is "teach everyone how to fish… pretty soon you are out of fish and have to move elsewhere". 

Yeah, it spoils the message a little bit, but it's true.

Another statement some may be less familiar with, but one I like a lot, has to do with goals and measuring them:

"When performance is measured, performance improves. When performance is measured and reported back, the rate of improvement accelerates."  

--Thomas S. Monson

On the surface, I totally agree with this, except for one fact… when we measure performance, we can get so caught up with the measurement and fine tuning the measurement and extrapolating the measurement, that we completely kill the performance initiative we were aiming to fix in the first place.

I make no secret of the fact that I consider myself a RescueTime nerd. I use it for my own benefit to see where I am and where I put my time. I started out totally objective in my goals, and I put everything into nice easily defined piles. Software Development work? Highly Productive. Dorking around on Facebook? Incredibly distracting. Personal email? Incredibly distracting. Simple and easy to see what I should be focusing my energies on.

Except it wasn't. What happens when I am responsible for testing Frictionless Sharing on Facebook? Is it distracting or highly productive? When my scripts go out and perform tasks on Facebook, is that productive or distracting? How do I tell the system the difference? It's simple, I go in and I create sub categories that tell me "this is distracting, except for when it's not"… and if you are confused by that last statement, congratulations, it means you are paying attention :).

The point is, we run the risk of becoming myopic if we are simply focused on wringing out what  we think is the most effective use of our time without applying some context to the time we are using. I had this point drilled into me last week when I got my weekly report. Yes, I get these as little pep talks to myself to make sure I know where I'm spending my online time. The overall numbers looked pretty good, including the statement that I was 28% more productive than the average RescueTime user… ahhh, but don't pat me on the back just yet. Why not? I thought to look at the categories. One of the most prominent (and listed as "Very Productive")? Yep, RescueTime itself! I actually spent so much time justifying, checking, fine tuning, and categorizing my activities, that I registered all that categorization as one of my most prominent uses of my time. 

Dumb Dumb Dumb Dumb Dumb!!! 

So lets go back to that second quote again, shall we?

"When performance is measured, performance improves. When performance is measured and reported back, the rate of improvement accelerates."

Except that, if we get too caught up in measuring and reporting, we end up actually wasting time that could be used to actually improve real performance in (fill in the blank category). I'm not saying "don't measure". I'm not saying "don't report". I am saying make sure that the data you are collecting is in the correct context, and then make sure that you are not getting too caught up in the measurement and reporting aspects. Provide enough to make sure you are on track, but remember the whole point of what you are actually measuring and reporting.

Hacking Down a Backlog

Oftentimes, when we get into doing a lot of things, we find ourselves either dropping items we want to work on, or we seem to forget that there are things we still need to do or that we have committed to do. We also tend to get into a mode when we somehow feel that, if we can't devote all of our attention to something, it's out of our reach and it's just not worth doing.

For fans of Seth Godin's "Linchpin", this is a classic trick of "The Resistance". When faced with what seems like an insurmountable task, we throw our hands up in the air and we say "oh, this is just not something I'm going to be able to do" and we hide from the commitment. That may make us feel temporarily good, but in the long run, it doesn't get us closer to our goals or to finishing good work. It also makes us more prone to put off more things that are important, and puts us in the mode where we just work on the things that are urgent (and often not all that important).

So how can we get a handle on all of this? I'm going to dare to pontificate, and realize my pontificating on this is not me trying to say that I am somehow an expert at doing these things, just that I fall victim to them a lot, so these are healthy doses of "Physician, heal thyself" :).

First, I think it's critical to give yourself set times to do certain things and do them in a regular manner. An example is when I produce the TWiST podcast. I know very well that, for me, my most focused time is early morning, so two to three days before the podcast is due, I get up an hour early and devote time to editing, listening, and taking notes. Because this is a regular delivery of mine (i.e. every week we post a new TWiST), it's a standing set of early morning dates, and I know when it happens. The net result, I rarely have to fight for time to edit the podcast, because nothing interferes on those sessions. It's ingrained.

Second, I look at things that are perhaps larger commitments. For one of the Foundations classes, I committed to giving detailed feedback to every one of the students. I am still making good on that one, and it's been several weeks since the last class ended. Life intervened, and it taught me something important. Even the best laid plans can be derailed by other things we have little control over. I also fell prey to the fact that I figured I'd be able to do this in an evening's time. Nope, not gonna' happen. So I had to readjust my expectations and the reality of what I could do. I couldn't review everyone in one night, but I can review one person per day. That means a few students will be waiting for awhile, but at least this way, I can set the expectation, and now I can focus on the one person who needs my feedback at that given time.

Third, if you use a tool like RescueTime to see where your online time goes (and believe me, getting familiar with that can be very informative) you can set goals for yourself in particular areas so that you are alerted when you meet them or if you go beyond a threshold. Do you want to spend a certain amount of time learning Ruby from a particular site? Set a time goal and track it. Do you want to make sure you don't spend too much time on Facebook or Twitter? Again, set a time goal and track it (and pop up an alert that says when you've spent too much time).

Fourth, sometimes we just need to dedicate "now" time to something. I have experimented with two different time focus techniques, and I see benefits to both. there's the Merlin Mann Procrastination dash (10 minutes on, two minutes break, repeated five times an hour) or the Pomodoro Method (25 minutes of focus, with five minute breaks, for the duration of time you want to be focused). Each of these still requires you to commit to timed periods of focus, which leads us to...

Fifth, eliminate distractions if possible, or make time or physical constraints for them if you can. I remember working with a company called WebSense a number of years ago when, at the time, their primary focus was security and filtering of web traffic. One of the phrases I remember one of their engineer's often saying was "You can't have it now, but you can have it later". This was at a time before video and audio streaming were as ubiquitous as they are now, and doing these things could be hugely draining on a corporate network (they still are, but nowhere near as much as back in the late 90's when these technologies were still in their infancy). The idea they fostered was to either put a time constraint or a physical constraint. For me, this might be "it's OK to spend as much time on Facebook or Twitter as I want, but I have to be walking on the treadmill desk to do it". It could also mean "personal emails will only be answered between the hours of 5:00 p.m. and 7:00 p.m." or any other number of areas. By doing this, we force distractions into scheduled blocks, and then we free ourselves of having to deal with them at other times.

Again, I come back to these things time and time again because I'm often the most guilty when it comes to dealing with these areas (or not dealing with them). We can't have it all, we can't do it all, there are no more than 24 hours in a given day, and those hours go by way faster than we want to believe. Taking small steps to whittle away at a backlog will one day get it down to a reasonable size. Yes, one day. It's not going to all disappear at once. The sooner we all get used to that, the sooner we can accomplish the things we need, and want, to do.