One of the key things I look for in talks is not to find things that I am good at (otherwise, really, what's the point of going?) but to visit and examine areas where I could use improvement or, to be frank, may just not be all that savvy about at all. Jim Hazen and I have communicated through Twitter and testing channels for years, and I have often appreciated his insights on writing test automation, specifically because he highlights the pitfalls that often get glossed over.
One of those areas is the Test Automation Pyramid. What's that you say? Well, there are different levels of automation, and foundations that automation is built on. Really, that's the core of the idea of a test automation pyramid. We build up from a lower foundation to get more specific and more focused, just as a pyramid start broad and wide and rises to a point at the top. Different levels and contexts require a different focus, some larger than others, but all structurally important. Commonly, this pyramid is structured in three layers: Unit Tests at the base , Service Tests in the middle, and UI tests at the top. Some benefits here are that by considering these tests in their places and consider them in order of creation, we put test design up front and carry it through the whole process.
Sometimes we look at the pyramid and we equate placement with importance. UI tests are at the top, and the smallest part of the pyramid, so that means they really aren't as important... right? Well, no, that's not what it means at all. What it does mean is that, by focusing on the Unit and Service tests, we are able to put in place logic and fixtures we can use higher up the pyramid. UI tests have value, but yes they can be finicky if the work below it hasn't been done or done well. Also, many think that because the Unit tests are at the bottom of the pyramid that that should be most of your tests. Well, not quite. The idea of unit tests is atomic testing of elements (methods, functions, etc.) and to make sure that we have a strong sense the tests are validating the functionality of the smallest elements of the code. They re numerous by their very nature, and by developing robust unit tests, we can develop a base for Service and UI tests. So does that mean the pyramid diagram is wrong? Potentially, but it still serves as a good way to look at the structuring of tests, timing and precedence. The pyramid is meant to be a guideline, not hard and fast rules.
Think about it this way. The extent of unit testing may result in a test for each line of code, and sometimes more. Positive and negative conditions may be exercised, and the basis of those unit tests can be used to help develop fixtures to help test higher up the pyramid. Additionally at the unit test level, we can look at ways to build testability and instrumentation into the product from the ground up. Where is that insight likely to happen? Probably at the component level. write a lot of unit tests, and it's likely procs and fixtures will come out of it, because good programmers are lazy, and that is a very good thing :).
When someone tells you they are going to do 100% automation, take a step back and realize that it's never going to happen. There's a lot that can be done, but there are still a lot of tests that require eyes, ears, and brains to determine and evaluate. Having said that, the tools are getting us closer to that 100%, but we still have quite a ways to go.
Another helpful rule of thumb is to think of tests getting more granular as you go down the pyramid, and becoming more complex the higher up the pyramid you go. Tools help, but no tools covers all of these areas comprehensively. No one tool will solve all of your problems. Custom code will very likely be required to make sure each of the levels and layers can play well with each other. Someone needs to write that code, and its likely a new tester writing automated tests will not be the ideal candidate for that job (they may well grow into it over time, but unlikely at the early part of their careers).
We also need to have a chat with management and set realistic expectations. See above about one tool to rule them all (hint, there isn't one). You need a staff with proper skills at each level. You may get lucky and find all of them in one person, but it's more likely you will have a bunch of people that have those skills. Ge them together and talk amongst themselves.
Overall interesting model, much of it makes sense, some of it breaks down, but taken in the spirit in which it was originally designed, it's quite useful, even if it should be more of a broad parallelogram, but that's nowhere near as catchy ;).
No comments:
Post a Comment