Friday, September 30, 2011

Sometimes, Design Really Is the Problem

Another fun entry from Aaron Scott's Two Leaf Clover cartoon. Admit it, you've been on projects like this. I think anyone who has ever been involved with testing for longer than a few years has at least one doozy of a story where we go back and ask ourselves "OK, really, what were they thinking?!"

While I don't typically go out of my way to out my companies and the issues over the years, there are a few that were certainly memorable. One of my favorite painful experiences was a 3rd party title that was presented to Konami to publish. I remember this title because it was the very first "game" I ever tested. I will admit the premise seemed interesting. It was very European, a post apocalyptic game with religious overtones and elements, battle of good vs. evil in deep space. What's not to like? After a week, I discovered that there was lots not to like. Clunky levels, really bad AI, and frustrating challenges that, if not played just right, were painfully hard to pass. This was originally a title for PC and Xbox, but it was dropped from the Xbox distribution list and developed just for PC release. That alone should have been fair warning. Judging by how many people we had work this title, and how long it was in testing, I released a sigh as I realized that it was not going to go well in the marketplace, and sure enough, it didn't. This wasn't just an issue with bugs. It wasn't just an issue with story. It was an issue with design flaws throughout the game, to the point that finishing the game was more of an expenditure of tremendous effort than a feeling of any real triumph. More of a feeling of "Glad THAT's over!"

Sometimes design flaws need not be so devastating, and occasionally, they can be found early enough to be corrected before big costs are incurred. When I worked at Synaptics, I got to do more than just test software. I actually got to lok at Environmental factors on physical devices. We're talking real stress tests with weights, fulcrums, freezing temperatures, super hot ovens, and my personal favorite but slightly annoying device, the Electro-Static Discharge (ESD) gun. ESD is a neat black art, and it's actually quite interesting to see what responds to it and how to isolate tests. One of the key early developments at Synaptics was the touch pad that is now ubiquitous on various vendors laptops. Making covers for these capacitance touch pads meant that they had to have surfaces that matched the notebooks they were going to be inserted into. We had a cool bunch of silver pad covers come in, and as I was testing them, we found really strange behavior at around 9,000 volts (that may seem like a lot, until you realize that the human body routinely carries the ability to discharge about 8,500 volts at any given time just through our fingertips). 50,000 volts and strange behavior is understandable. 9,000 volts and strange behavior was definitely odd.

I asked for a chemical makeup list for the sheets of metallic silver covering we were working with, and nothing jumped out at us immediately. It shouldn't be a problem, the metallic silver was being simulated with crushed mica. Mica is a dielectric and an insulator, so why is it behaving strangely when we apply the ESD tests? This prompted some of our engineers to have additional talks with the engineers producing the covering (a 3rd party) and it turns out that there was a higher level of other elements in the mix to make the covering in addition to mica. Not a large amount, but just enough to skew the results of our tests. A few weeks and a reformulated sheeting was ready. We tested it and it didn't have any of the problems. Our Program Managers often joked with me that in that one day I may have justified several times my salary :).

As I posted earlier today in my review of "How to Reduce the Cost of Software Testing", Selena Delesie makes the point in her chapter that we cannot test quality in.  We absolutely cannot test and add quality if there is a fundamental flaw with the design. No matter what we do as testers, we cannot overcome some issues, but we may have a chance to influence a few if we get to them early enough.

1 comment:

Tal said...

great post, the mica example cracked me up :)