Saturday, March 13, 2010

Can You Really Test Anything?

A couple of days ago, I commented on the idea that there are those who see testing as an "easy" profession, and that "anyone can do testing". I disagreed with this statement and said that, actually, it was quite the opposite, that it takes a certain type of temperament and skill set to be good at testing.

There's another piece to this, and it deserves to be mentioned as well... the idea that testers can test anything. That's partially true, but just as I said that to be truly good at testing, not just anyone can do it, likewise, it's very possible to be a superstar tester in one endeavor and struggle in another. I've had these experiences over the years, and I know how it feels to be flush with victory in one sphere, only to be slapped hard with the realization in your next endeavor that "you aren't as good as you think you are".

I came into testing like many people did, from a sideways trajectory. Truth be told, I did not go to school and say to myself "you know what, I'm going to be a tester when I finish school". As a matter of fact, outside of using a tiny old school Macintosh to put together fliers for my band, I barely had any interaction with computers at all. I was working as a housekeeper and a part time student by day and as a musician by night when, in 1991, I decided I needed something more stable to help pay the bills as a musician. I took a temporary job to help pay the rent while waiting for super-stardom. Well, super-stardom never came, but that temp job grew into a ten year long career, and one that introduced me to the world of software testing and the people that did it.That experience took place at what was then a little, barely known company called Cisco Systems.

As I interacted with the testers, I learned about things related to networking interfaces, and how to configure switches and routers, and how to build networks. From there, I learned what it took to get data from point A to point B and what I could do to cause problems in those transactions, but see if the systems would recover. In short, I discovered that testing was fun, but challenging to be effective. I had some great help along the way and worked with some great people who taught me many things.

Several years into this process, I decided I wanted to go into the Network Management side of things and test Network Management applications. I figured all of my experience with routers and switches would make this a breeze... I was very much mistaken. This would be one of the first times I realized that my skills were not entirely transferable.

Lesson #1: Not all testing is created equal, and just because you know how to do some things very well, it's no guarantee you will do well with a similar product or even in a similar area.

Lesson #2: I had worked with a team of people when I was testing routers and switches. When I moved into testing Network Management software, I was a lone gun. I learned quickly that when you stand alone, there's no one else to fill in the gaps, so you better figure out what you don't know quickly, and if you can't, get onto something else where you can be effective.

Later on in my career, I decided to leave Cisco Systems and try some other options. For two years I made a stab at being an Application Engineer at two separate companies, but in both cases, I realized that my testing skills, while definitely helpful, were very low on the totem pole of what I was expected to do (actually, I think the only more nebulous job description beyond being a tester is to be an Application Engineer; it seems that whenever I asked people what I was supposed to do, each person had a different answer). One of my stints was at a company that made specialized interface devices that worked with human touch. As such, the company was focused around hardware development with a software interface to communicate with the system. Hey, I'd worked with routers and switches, these wouldn't be that hard to figure out, right? Ha! I soon learned that not having a strong background in physics put me at a distinct disadvantage in the group I was working in, and because of that, I really struggled to test these products effectively or to even understand what I was testing (though I have to confess that learning about the human body model and doing ESD testing was a lot of fun in a twisted way).

Lesson #3: Hardware is not software, and different hardware skills are not necessarily interchangeable. Each time you find yourself working with a different product, you run the chance that you are starting off at almost ground level.

Lesson #4: Actually, you don't really start at ground level, but you do end up using what Dr Rebeccah Reinstein refers to as the "spiral staircase" phenomenon, in which you come back to the same place, but a little bit higher each time.

While I did learn quite a bit at this job, it became clear that I was not a good long term fit in this position, and so I decided to try something else. I went back to school and while I was in school, I went to work for a game publisher. I figured, hey, I like games, and I have 12 years of testing experience, this will be a piece of cake! It took me about five days to realize that, no this was a paradigm with its own rules, its own methods, and that there was a whole vocabulary to testing that was totally different than anything I'd ever done before. My general testing skills helped quite a bit, but I was definitely outclassed by those who had done this for years, and it showed. I did pick up a lot of great tips and methods and learned to see things in a different way and became a pretty good game tester, but I'll be the first to state that my experience in the game industry was a sobering one; my 12 years of general skill was perhaps 25% helpful. The other 75% was domain specific, and I had to learn that from the ground up just like everyone else.

Lesson #5: General skill is important, and knowing how to tap it is helpful, but it will only carry you so far. Primary domain knowledge is the deciding factor between who is adequate and who is genuinely good. The good news is that, if you have the fundamental knowledge, the domain knowledge can be developed in time, provided you are willing to put in the effort to learn it (and yes, when it comes to games, there is a lot of domain specific knowledge to learn).

Lesson #6: People think it is fun to test games, and it is, to a point. However, people hoping that their enthusiasm to play games will make them great game testers tend to get a bit of a rude awakening. It's work, just like any other, and to be successful, you have to have more than just a desire to play, you have to actually learn the ropes of what it takes to be effective in finding the problem spots, just like any other application.

Today, I work in a unique capacity. Again, I'm a lone gun, and again, I work with a product that has its own special quirks, and it took me some time to learn the ropes of what it takes to be an effective tester in this sphere (and at times, I question just how effective I actually am). Believe me, I have my good days and my bad days, but after almost 20 years, I have had enough turns on the Spiral Staircase to see that, yes, I can possibly test just about anything now, but to be effective, I can't just drop in and expect to hit regular home runs (that's very rare, actually, unless you are doing something very similar to what you have always done). I have learned, however, that there are many skills that I have picked up over the years that I can cobble together to help me test any number of things at a fundamental level, while developing the knowledge to go deeper as I get the domain specific knowledge necessary to be effective, and knowing that that domain specific knowledge will be different just about everywhere a tester goes. It certainly makes work interesting.

No comments: