We're starting out with breakfast at the moment (don't worry, I'll not post what I'm eating, other than to say it's real food, and for that I am appreciative. I'm currently having a chat with Becky Fiedler, and we've shared some horror stories and had a laugh about crises in classes and how we've had to deal with getting things together in less than ideal circumstances. I've been dealing with the fact that two BBST courses are running simultaneously (one of which I am leading, while I am here. Yes, it's been an entertaining few days ;) ). Henke Andersson, Andy Tinkham, Griffin Jones and a few others have joined us and we've been chatting about my frustrations and things that I've learned going through a crash course in accessibility testing (yes, hilarity has ensued).
We're getting ready for the first Keynote, which is going to be a panel discussion between Rex Black, Bob Galen, and Cem Kaner, which will be about "What Will 2013 Mean for Software Testers?" So if you'd like to see what that might be about, I suggest you come back and see.
Rex Black started out the conversation about the challenging environment testers face, many of which are technology specific, and many others are cultural. With the advent of cloud computing and the ability to create unique instances for a period of time is radically changing the framework that testers are used to. The days of being able to say "well, we only have so many spaces to test these things", using a metaphor that he described as the "cubic meter of devices", is no longer an issue, but it opens up a new challenge. Now, every configuration is available, and the old excuses of 'we don't have access to that" now becomes "do we have the time to do that?"
One theme that Rex mentioned was that 2013, if there was a term to consider, it would be "retooling". With the explosion of open source and freeware tools in the space, we no longer can step back and say "oh, the tools re too expensive, therefore I can't be part of that. We also need to stop thinking about "what tool would you buy to solve X problem" and instead get an understanding of what we actually want to do. For me, Selenium is great, and I spent lot of time talking about it. Not because it's an application that is the be all and end al, but because it fits a certain need and allows a lot of different interfaces to interact with it.
Rex quipped that, if someone wants to become insanely rich, if they can figure out how to anonymize large amounts of customer data, from multiple disparate repositories, and still retain its value as test data and provide genuine business value through its use, that might be a good problem to solve.
In a nutshell, 2013 will not provide any silver bullets.
Cem Kaner started out with a compelling thought, specific to the fact that he comes from working with students, and he laments the fact that one of our biggest challenges for students is that we don't seem to provide anything of genuine lasting value for them in (fill in the blank) career. Also, let's stop talking about agile as som hip new thing. It's part of the culture, it's been here for 13 years, and it's another methodology.
One interesting factor, whether some of us want to admit it, is that the U.S.A. is no longer the undisputed center of the universe center of the technological universe. Many of us have seen this coming for decades, and with that, we have had to come to grips with the fact that "the old order" while not completely gone, has been seriously disrupted. With this, new paradigms are now available to us, and new ways of thinking, and organizing, that will change even more as we move forward through 2013 and beyond.
The development of virtualization to the effect that models like EC2 and other options available in the cloud now make the possibility of massively parallel automated testing is something that only the most wealthy companies could perform available to just about everyone. The bigger question, though, is whether or not the testing being performed is doing much more than just running a lot of tests a lot quicker. Quantitatively, yes, we are running more tests faster than ever. What we still don't know is whether or not the tests that we are performing are actually providing a real specific business value.
Education is another avenue that is facing a major disruption as well. People are currently paying a tremendous amount of money to go to school, and many individuals who have gone through this process have found themselves second guessing the value proposition. Is it wise to be educated? Yes. Is it wise to take on a huge amount of debt to get a job that will not allow you to pay back what you have borrowed? That is an individual question. The real question is "do we have people out there who can actually do the work that is needed, and is the educational experience helping them actually do that?" Complex problems are not really covered in schools, and because of that, people are looking elsewhere to learn. Online classes are exploding, and some people are getting great benefit from this new model. Some, however, cannot really learn in this new manner. Ultimately, it is possible that we may see a de-coupling of the educational process and the credentialing process. Could we see a world where someone works through all of the coursework on Coursera.com, and then go to an independent body to validate and prove that they deserved to be credentialed with a degree? Cem thinks that may become a very real possibility.
Bob Galen says that he has about ten hopes and observations he sees not just for 2013 but for the near and not so near future.
Do less testing (or emphasize the testing that is most important). Testing doesn't automatically add value. The information you provide that allows stakeholders to make good decisions is much more important.
Become comfortable with asking for an giving slack time to think and innovate. Testers should take advantage of creative periods and adapt and get creative as well.
Critically think about risk. Bring valuation thinking and focusing on the right amount of risk. testers are stakeholders and customer surrogates, so let's do what we can to internalize this.
Customer experience is critical, deliver and measure what the customer wants, and be able to pivot based on the feedback you receive.
Lone Wolfing it is becoming passe. Testing and the value that it offers is less and less just the domain of one person (as a recent Lone Tester, I can attest that this is finally taking hold in some organizations). It's no longer "why didn't test catch this". Instead, it's now "why didn't our team (the whole team) find this issue?"
It's about people. Let people ask for help, let people admit their ignorance. Allow yourselves to be vulnerable and allow for experimentation.. failure need to be allowed... no really, not just say it's OK, but mean it, as long as people can spring forward and learn from it. Failing shouldn't be a sin. Repeated failure in the same places should be.
Testers need to ask questions and help partner with product owners to really help to understand if we are solving the right problems.
It's not about the tools. Analyse the problems. Don't lead with tools. People and data lead, tools follow. Switching the order causes heartache and frustration.
Courage is going to matter. Courage in all interactions. Challenge your tram to work better. Challenge your product owner to think about what actually solves a customer's problem. Challenge yourself to learn more. Have courage to talk transparently and be honest, no matter how painful that might prove to be (and yes, it certainly can be).
That's a pretty good list, and it's very actionable, not just for Agile teams but for any team.
The Q&A period started with, of course, the question "What do they think about the current meme of "testing is dead?" The three believe that the comment is coming from a "bomb thrower", but they also feel that the bomb thrower is doing a service. Testing is evolving, and if we do not adapt to that evolution, then yes, testing that cannot adapt and will not adapt is dying and deserves to die. To say testing is dead implies that experiments are never run, and that we never learn from them. We can all say that that is ridiculous.
Another questioner asked about what skills were required to be effective in this "brave new world". The big myth is that we have the find the "golden tester", that person who has all of the neat buzzwords at a high enough level. The simple fact is, no jack of al trades who is a rock star in all areas exists. Developing a team that allows for specialization and distributing skills among multiple people is the same reasoning as goes into an investment portfolio. the key in the financial portfolio is diversification. Likewise, to develop a team that is resilient and not headed for the evolutionary scrap heap, evolve the skills for the entire team. Niches are good, as long as there is an understanding of the niches and so that they don't exist in silos.
Another question dealt with testers having to become intimately familiar with the code, as well as the growth of the SDET, the focus being programmer testing. The three agree that the skills are important, but there may be too much emphasis on the code, an not enough about what the users wil experience. To a user, the code itself is almost irrelevant. They don't care what the code is, they care what the tool can do for them and help them solve a problem or perform a task. Bob recommends a simple approach... pair more. Not continuously, not always, but be inquisitive, pair with developers, look at code, and try to understand the perspectives of the programmer and what they are testing.
The last question was focused on Exploratory/Rapid Software Testing and whether or not it's a fad. It's an activity, it's a tool, and it has its place. It has a benefit, it is effective, and it should be used by many teams. Exploratory testing is not a new thing. It's been around for decades, though the phrase is a recent development. The fact it, any time we investigate to learn about something, we are exploring. Does it make sense to drive all projects with Exploratory testing? It's possible, but there is a balance between free form exploration and verifiable accountability and reporting. It's not this or that, all or nothing, it's using the right tool, at the right time, in the right context.
And with that, the first keynote is over. Now you'll just have to follow along with me and see if you like my choices. I hope they prove to be of some value to others :).
The first session I decided was most relevant to me and what I came to get answers for had to cover performance. That and the fact that I've conversed and worked quite a bit with Mark Tomlinson via TWiST podcasts and listening to his own PerfBytes podcast.
Some people struggle with trying to understand the importance of software testing. We know it's important, we've been told it's important. We can at a gut level internalize that it matters... but why? What aspects of it, when you get right down to it, actually matter? The aspects of how are pretty well understood. The point to this presentation was to step away from the how, and focus on the why's. Capitalizing Performance allows us to "capitalize on Performance".
Here's a question... how can we find new ways to amplify the message about the benefits of performance testing? How can we make what we are doing relevant to the disparate stakeholders? How can we make the case for the business value of what we are doing? OK, that's multiple questions, but they are all related.
Performance doesn't happen in just one place, and performance tests have different value aspects in different places. Database performance is different than front end interface performance, and they have different audiences.
To often, we flood people with facts and figures, but we don't really explain what it is they are looking at. Pictures are helpful, but without context of what they are looking at, much can be lost. More to the point, results can be hard to conceptualize if we don't understand what group or stakeholder may actually care about it. Benchmarks are great, but can we really explain what the benchmarks actuallyrepresent?
A key technique that can help make this more clear is to not look at values by themselves. Comparing two or three different data points and overlaying them can help make sure that the message that is important is conveyed. If there is a spike in a graph, will it make any sense by itself, or will it make more sense to see it with another variable? CPU cycles by themselves may be interesting, but we probably care a lot more about how many people are accessing a site at the time that those CPU cycles are being recorded.
Where do things like this matter? Imagine you are selling tickets to a concert. the site can perform very well 99% of the time, but the period that really matters is the first few minutes that an item goes on sale. In a spike, what can happen to a site? It can get clobbered, and if the site isn't set up and ready to take on that immediate spike, the 99% of the time the site ran beautifully doesn't really matter. Put financial information into the data where applicable. Why? People at the top of the company food chain don't really care about the esoteric details of the performance data. They care if they are making money, or losing money. That's a message they can understand. If the performance details show that we can make more money if we can keep our transaction time to less than 6 seconds, and roughly quantify it, now we can show where we need to be and why? Money talks, oh yes it does ;).
Color is important when used correctly. A blue line that steps up at a particular point in the graph may mean something technical, and be statistically interesting, but people outside of the tech details won't really understand what you are looking at. A spectrum graph that shows a lot of red, and a little green, without any additional information, can help communicate a lot:
Without going into details about the graph above, the point is that that steadily climbing region of red communicates volumes. We may not know exactly what we are looking at, but we can see that as the graph progresses, the increased red means "not good" and not good is geting more prominent. Simple psychology, but it's effective.
When at all possible, you should use a copy of prod data, or at least a very large representation of that data. There's a lot of pressure to prevent people from being able to use data in this capacity, which then means that we have to scrub the data and change values to make it so that the data cannot be used nefariously. Sounds good on the surface, but as was mentioned in the keynote, that scrub very often causes the data to be very much less effective, as well as not being able to get a true representation of the data. Instead, consider scrambling the data, which will preserve the cardinality of the data, but randomize the actual characters. That data then remains the same size and with the same parameters as the original data, and that helps us determine if the performance we are seeing is genuinely representative.
Other key things to help make the case for this approach to performance testing, is the use of os simulating genuine, real world tests. Do the pages created persist for any respective amount of time? Are we actually looking at the real data values that matter? Can we partition are data in a way to help determine which data sets get the most demand?
Ultimately, we want to make sure our tests are convincing to our stakeholders. How do we do that? Let's make sure we can answer these questions:
-- is our test credible?
-- is it relevant?
-- is it meaningful?
-- can our test help them?
-- are our results accurate?
Ultimately, we need to make sure that these data points that we provide and present can be communicated in a way that everyday people can understand. Everyday people understand time and volume. The physical details such as cpu, disk, memory, networking, power and cooling are great for us geeks. To help people outside of the data center actually make connections with and internalize what is going on, we need to communicate things in time and volume. In short, the details need to make a compelling story as to "why" something is going on.
I had a great dinner last night with Lee Henson (see my comments yesterday about getting bags of crayfish shipped ;) ), so when I saw that he was speaking today on "Persona Testing and Risk", I decided this would be a fun place to spend the second session.
For those who follow my blog regularly, you're probably saying to yourself "but Michael, isn't persona based testing something you are already intimately familiar with?" Well, yeas, but even areas I know a lot about, I can learn something new and interesting from different perspectives. I teach BBST Foundations regularly, and I learn something new every single time I teach it.
The sad fact is that, while we want to believe that we "care" about quality, and that we want to provide the maximum amount of testing, getting something out into people's hands so that they can use it and so that the company can make money trumps howe much we care about quality. there's no such thing as complete, fully tested or error free. Nothing is. Nowhere. The fact is, we really have to approach anything we do with a sense of what is good enough, complete enough, and fast enough, to be delivered in a way to be effective, and what risks are we willing to take to meet that point.
The earlier we can get together and talk about what the requirements are and how they can be expressed, the more likely we will be able to identify the areas that have the greatest risk. Key to that is understanding the roles that people play. We sometimes use people in the equation, and truthfully, that's the wrong way to look at user interaction. We know that people are going to use the site, but more importantly, we care more about what that person does and how they would interact with the system.
Personas help us define better what someone is doing, not just who they are. It's one thing to say "I am a middle aged woman". It's another to say "I am a middle aged woman who is very anxious about my personal information appearing in an application". That's much more specific, and that kind of a persona will be much more informative when it comes to how we want to interact with that person, and what options we provide to them.
When we say "as a user", we make a very nebulous and squishy entity. There's so much that a "user" can do, and much of it isn't helpful as to how important it is. If we can identify people, not just by role, but by behavior, preferences, and personality. When we have this information, now we can narrow down on who we are actually making our product for. Even more important, we can actually test the aspects that are important to that individual. Not a nebulous amorphous person, but someone we have actually named and may know as well as our next door neighbor (OK, yes, that's a weird analogy, but work with me here ;) ).
There are negligible risks, and then there are serious risks. As a way to drive this home, imagine you fly every week on a particular airline. Would you be happy to know that the pilots of this airline land their planes safely 89% of the time? If you are going in for open heart surgery, would you feel comfortable knowing that your open heart surgeon has had 75% success rate with his operations? Of course not, to us in these circumstances, anything less than 99.99% would probably be too risky. Do we then provide this same metric for a web site that plays videos? Again, of course not. It's easy and fun to joke about the fact that some risks are ridiculously applied, but risks are not ridiculous when looked at in the appropriate context (if you're playing the TESTHEAD live blog drinking game, go ahead and take a drink ;) ) .
When we make changes in an organization, Lee talks about "calling the COPS"? Huh? COPS is an acronym for:
Every change goes through these steps, and everyone in the organization has to be on board with the COPS to make a change actually stick. The expectations need to be clear, but they also have to be reasonable.
No organizations is going to test everything, cover everything, and be everything to everyone. They can provide good test coverage for the people that matter the most. User personals will help you find out who those people are, and what those people actually do.
The lunch time session is being made a little more interesting by the fact that several tables have what is called a "Speed Geeking" session. These are lightning talk style presentations, and they are held at individual tables, and when the session ends, the participants moved to different tables.
After perusing, I've decided that I need to participate in:
The Death Star Was An Inside Job (Matt Heusser)
The Top Ten Performance Testing Types ( Mark Tomlinson, James Pulley)
Why Executives Often See Testing as an #EpicFail (Scott Barber)
I'm trying a new technique in recording, so we shall see how well these work out (I'll post these later tonight if it works as I hope it will).
Jane Fraser is someone I've talked with and been part of TWiST podcasts with in the past. Additionally, a good number of my friends have worked either directly or peripherally with her, so I was very interested in seeing what her presentation, "Becoming and Influential Tester" would be about.
Jane has spent much of her career in games, and has recently moved into a company that focuses on robotics. As such, Jane has had to demonstrate and show her influence over many years, and how to make it possible to get organizations to look at testers and testing in a proper light and with the respect it deserves.
Influence and leadership are somewhat intertwined. You don't have to be in a high profile position to be a leader or have influence. People respond to people based on the level of influence that they have. Those who try to influence by force tend to not get much in the way of buy in from their team mates. They may do the work, but they will not give their all. Intimidation likewise gets confused for leadership, but is not. It rarely makes for people willing to follow your lead. Manipulation is also not very good for long term influence, though it can work in certain circumstances. The position we hold can provide some level of influence, but again, that still is seen on the negative side on the ledger.
So what goes on the positive side? When we Exchange information with others, that works in our favor. Trying to Persuade others to understand our views works on the positive side. Giving Respect often gets Respect back. Timing is important, and early is better than late, always. If we see issues or potential challenges, when we can announce things early enough for organizations to adapt, that will help build credibility and show that we are aiming to be effective and show that we know what we are doing. If we do all of the positive aspects mentioned previously, we can build the most important influencer, which is Trust. Trust requires that we have initiates enough influence over time to demonstrate that we are worthy of and deserve the confidence that has been given to us. Bringing Teamwork into the equation, for many people, being a part of the team can either be for the moment in a company, or a permanent situation between people that span jobs and careers.
Influence requires that we are responsible for the stewardships of our efforts. We are responsible for ourselves and others, and while influence can be hard won and slowly gained, it can be lost very quickly. When we make efforts to provide a positive influence, and when we are working to add value, trust is likely to rise. If you are negative, and if you damage other people's efforts, it's likely taht your own influence will dwindle.
Integrity is critical. If you are transparent, do what you say you will do and delivering, then your strength of integrity will grow. Do you have the courage to tell your CEO that you have messed up on something? Are you willing to take the fall for a mistake, and not assign blame? Chances are, if you do mess up, and you learn from those issues, and don't make the same mistakes, then you are likely to recover well.
Jane uses the acronym LADDER to help us increase our influence. We can increase our influence when we Listen and do the following:
Look at speaker
Don't Change Subject
Jane recommends that we focus on Listening, Learning and Understanding. Being brave and asking questions may be hard, but it's easier than walking away not asking questions, and then having to go back later and re-ask the same questions. Jane also recommends a process of 360 degree leadership, in a process she calls leading up (helping may your boss or manager successful), leading across (working directly with and leading effectively with your peers, and leading down (being a mentor and providing the compas for junior team members so that they can be successful). It's important to understand what our co-workers, team members and direct reports need and value. Setting an example is huge.
Learning, and helping others learn, is hugely important. If your team members see that you are looking to keep learning and keep growing, and use your knowledge to help others learn, that will help you to communicate better with your team, and it shows that you respect their efforts and want to be able to communicate effectively with them. With learning comes growth, and as you grow in knowledge and skills, and you help your team grow and learn, again, you start building trust among your team.
Being an effective navigator helps a lot in the process of influencing others and giving others the confidence to see your leadership and influence grow. If you can help clear road blocks for others, they will look to you in the future. Understanding the history of where things went well, as well as where things went poorly, and demonstrating that you have learned from your experiences helps develop more credibility.
Lots of cool ideas and information. I definitely enjoyed this presentation :).
When I saw that Anna Royzman was teaching on Context, I knew I had to be there. Anna started off the session with James Bach's tester's version of "The Towering Inferno". If you haven't seen it, do a search, it's both funny and educational.
The point that James was making in the clip (and what Anna is making in the course) is the idea that the tester is the Steve McQueen character. He's the one asking the questions, and explaining why he's asking the questions he is asking.
Ann a walked us through an example where we had to try to determine how to test an item where we couldn't get any information about the product. Three of us (yours truly included) tried to see if we could penetrate the wall and find something out about the product that we could learn so that we could do some kind of effective testing.
Ultimately, testing comes down to three questions:
Those who have taken or have read through the Rapid Software Testing materials may recognize this. It may seem flip, but really these three questions are *very* important. Why does something work (or not work)? Is what we understand actually true? Why does it matter?
There's a decision map that the U.S. Army uses called the OODA Loop. OODA stands for:
The fact is, there are lots of reasons and issues that can drive different stakeholders into making the decisions that they do. There can be politics, there can be fear, there can be mistrust or a low level of overall communication. Thus, to be able to move forward and be effective, one of the things we need to do is ask questions that are relevant to the stakeholders and their individual spheres. We will ask different questions, and in different ways, if we are talking to someone in Sales, Administration, or Operations than we will if we are talking to a software developer or an architect. Each has a specific language, and each has their own concerns. We re much more likely to be able to determine how to test effectively if we can articulate our goals and our mission in a way that makes sense to diverse stakeholders. Sometimes we have to build trust in places where there isn't any trust.
To close out this session Anna asked Andy, Griffin and me to come up with a product with a "hidden agenda" and let the rest of the participants ask us questions to try to see if they could determine what it was that we were developing. I will say that, while we did provide a lot of clarity of the project, the "hidden agenda" was not revealed until the very end, and it wasn't ferreted out by the questions asked. An additional item in this process, a book I think I need to get, called "Game Storming" (Dave Gray, Sunni Brown, James Macanufo, published by O'Reilly).
And with that, it's 6:00 p.m. and I'm hungry. Catch you all later!