Tuesday, June 19, 2012

My Current Dilemma: Randomizing Cucumber Tests

I'm working on a situation where I am trying to describe how to balance out activities such as Acceptance Test Driven Development, GUI automation and exploratory testing, and explaining where we can take ideas from each and work with them to help us get a balance between the three. This is the subject of a talk I'll be giving and am currently writing (which also explains my being more quiet than usual on here :) ).

One of the ideas I discuss, and that I actually do, is that I put a little bit of "What if" into my acceptance level tests. I have a suite of tests that I run to check and verify basic functionality for every build that we make and push to our respective machines (development --> demo --> staging --> production). All of my tests are written so that they can run on any of these environments. These are Cucumber tests, running on top of Ruby in a Rails environment.

One of the limitations about running Cucumber as an acceptance testing tool is the fact that the tests all run in alphabetical order in the feature directory. This is assuming you run your tests using rake; my setup is configured so that I can run a suite of tests with "rake cucumber:[@tagName]:[machineName]". For some added analysis and review, I currently have a bash wrapper script that allows me to tee the output to a log file, parse the log file for errors, set up a new suite of tests, and then rerun them. The goal is to get to where I can run these tests in a single pass and have no errors (legitimately, of course :) ). Usually, though, that doesn't happen. I typically have to make three passes before all of the tests pass and there's no more errors to parse in the log file.

In a pinch, that's OK, but the nerd in me doesn't like the fact that I have to tweak things like this. This caused me to start asking "What if"... What if there's a dependency I'm not aware of? What if there's something about the way that the tests are run and the way that we authenticate certain accounts that might leave "rat droppings" in the state condition? What if I tweaked with the order of the tests? What if I totally randomized the test order every run?

I've been able to address the first three, but the fourth has been a bit of a mystery. How can I set up a process where, without going in and renaming directories or files every time, can I make it so that the sixty or so scenarios tests that I run are always in a different order? I've seen different ideas, but most of them require making a weight for tests (which isn't really random), or setting up some kind of  permanent table with file name and line number to designate the test to be run (again, has to be manipulated each time, and adding tests creates overhead), or using bash's $RANDOM option, but again, this is getting me into doing bash gymnastics, which isn't really my goal here.

So I ask my fellow testing friends, especially those who use Cucumber and Rails, what do you do?

2 comments:

James Marcus Bach said...

I don't use bullshit tools like Cucumber. I use Perl. Hence, I can do what I like, and I can do it quickly.

Stop using tools that limit you in dumb ways.

Michael Larsen said...

Heh heh, I kinda' figured I'd get an answer like that from you, James ;).

I've gotten some suggestions as to how to do this by going directly to Ruby and using the system() call, so I have a few things to experiment with. I use Cucumber because my team uses Cucumber, but in my world view, tools are tools, and I'm not married to any of them, nor am I averse to using other tools.

I still remember some Perl from my CGI days, but it's been awhile.