Am I being funny? Actually, no, I'm being very serious.
A number of us decided to sponsor one of the breaks at CAST this year. Often people would sponsor a happy hour or something. I don't drink, so that's not something I would sponsor, but Ice Cream? You bet :).
Again, think I'm kidding?
http://cast2013.sched.org/event/2883cbeeac41d1e9b609bdd4f8a0ab02#.UZ61vitARQU
OK, it's not just me. JeanAnn Harrison, Matt Heusser, Anna Royzman, Pete Walen and I decided to pool together to bring this little sweet spot of CAST to the attendees.
Oh, and I should also note that, while my bio says Socialtext, they're not covering this. I am. Therefore, "TESTHEAD is bringing you ice cream"!
Don't say I never did anything for you ;).
Thursday, May 23, 2013
Gaming and Skills Acquisition
Over the past couple of years, I've been involved in learning about, as well as developing, ways to teach software testing to others. One of the aspects that is appearing in many social interactions, from the trivial to the fairly high stakes, is an aspect called "gameification". If this term is new to you, it's the idea that, by adding elements of gaming to the skills we are working on, and sharing those details, we can both inspire ourselves and inspire others by our performance. Also, there's something that's just kinda cool about going somewhere, pointing to a screen and saying "see that high score? Yeah, that's MINE!"
The question that constantly comes back to me, though, is just how much gameification, or gaming itself, actually lends to skill acquisition. To experiment with this, I decided to resurrect an old friend and some old memories. Some background; I played some video games in my youth and teen years. We owned an Atari 2600 console, and had several of the 8 bit games available at the time. I did not, however, follow on to the "Golden Age of Gaming" the way my brother and elder younger sister's generation did. My own gaming experience wouldn't be brought out of hibernation again until 2003, when I went to work for Konami. It was a prime time to start looking at video games once again, since my kids were just becoming old enough to be aware of things like Pokemon, Yu-Gi-Oh, etc. ). With that, I decided to buy a couple of GameBoy Advanced handheld consoles and start playing along with them. Shortly afterwards, the PlayStation 2 invaded our house, followed by the Nintendo Gamecube, the Nintendo Wii, the Playstation 3, and subsequent updates to the GBA and Nintendo DS formats (I still have my original cobalt blue DS I bought in 2004; it works perfectly). For grins, I plugged in a favorite game that I spent many hours with a decade ago; "Castlevania: Aria of Sorrow".
One of the things I learned from this game in particular is that there are two types of gamers, when you get right down to it. The first type of gamer is what I call a "finesser". These are the people that learn the special moves, practice their hand/eye coordination, and their overall speed in moving through games. If I had to use a sports analogy, I would call them "boxers", a la the Muhammed Ali type. These are the type that, to carry the boxer metaphor forward, would wear out their opponents, and then through grace and yes, finesse, finish off the fight. The second type of gamer is what I call the "statmaster". These players know what it takes to beat a given level, and they build the stats necessary. When you hear about people who grind through levels to maximize skills, points, equipment, amass money in games to get top notch equipment, it's the "statmasters" that make up the far end of the spectrum. These are the ones that believe in "victory by overwhelming firepower". Level up to 15, and taking on a Level 10 boss is almost boring. These people are, to carry on the boxing metaphor, are what I would call "punchers", a la Mike Tyson in his prime. They don't have a lot of finesse, they don't play elegantly, but they are devastating when they hit, and they hit really hard.
So why does it matter if you are a finesser or a statmaster? Personally, I think it matters a lot. Gameification of objectives needs to consider the fact that people are a spectrum, and that, typically, they fall on either far sides of the finesser or the statmaster, or anywhere in between. Elevate the stats aspects, and people will gravitate towards that will grind to meet the stats. Focus on the finesse aspects and those who are most adept at those aspects will learn how to finesse the game. Give an experience that is too heavy one side or another and you will alienate or underwhelm the participants that fall to either side.
My question is, as I consider activities and goals is to see how do we actually balance these two aspects? Why does Aria of Sorrow speak more to me than, say, Metroid Fusion, even though both games share a lot of the same anatomy and approach? It's because, at least for me, Metroid Fusion rewards the finessers early on; there's little in the way of grinding to get better. Either you figure out how to win the battles or you do not advance. Aria of Sorrow, on the other hand, lets you level up and build strength if you want to. Granted it can take a lot of time, but by doing so, you can both get the stats you want, as well as develop the moves necessary to win (though frankly, at a high enough level, the moves aren't even all that important, just hit and hit until the enemy goes down).
Note, the examples I'm using are a bit old now, but I think the dialog still applies. As we think about the skills we want to acquire, and we say to ourselves "make it like a game", there's more to that statement than meets the eye. Not only if you like to game, but how you like to game, will determine which model and approach works best for your personality and style. As an admitted statmaster, I realize what works for me won't necessarily appeal to the finesser, which means I either have to consider two different methods and models, or try to find a happy medium ground between the two. How about you? Do you consider your gaming style when you engage in these types of things? Does the gaming metaphor work for you when you try to learn new things?
The question that constantly comes back to me, though, is just how much gameification, or gaming itself, actually lends to skill acquisition. To experiment with this, I decided to resurrect an old friend and some old memories. Some background; I played some video games in my youth and teen years. We owned an Atari 2600 console, and had several of the 8 bit games available at the time. I did not, however, follow on to the "Golden Age of Gaming" the way my brother and elder younger sister's generation did. My own gaming experience wouldn't be brought out of hibernation again until 2003, when I went to work for Konami. It was a prime time to start looking at video games once again, since my kids were just becoming old enough to be aware of things like Pokemon, Yu-Gi-Oh, etc. ). With that, I decided to buy a couple of GameBoy Advanced handheld consoles and start playing along with them. Shortly afterwards, the PlayStation 2 invaded our house, followed by the Nintendo Gamecube, the Nintendo Wii, the Playstation 3, and subsequent updates to the GBA and Nintendo DS formats (I still have my original cobalt blue DS I bought in 2004; it works perfectly). For grins, I plugged in a favorite game that I spent many hours with a decade ago; "Castlevania: Aria of Sorrow".
One of the things I learned from this game in particular is that there are two types of gamers, when you get right down to it. The first type of gamer is what I call a "finesser". These are the people that learn the special moves, practice their hand/eye coordination, and their overall speed in moving through games. If I had to use a sports analogy, I would call them "boxers", a la the Muhammed Ali type. These are the type that, to carry the boxer metaphor forward, would wear out their opponents, and then through grace and yes, finesse, finish off the fight. The second type of gamer is what I call the "statmaster". These players know what it takes to beat a given level, and they build the stats necessary. When you hear about people who grind through levels to maximize skills, points, equipment, amass money in games to get top notch equipment, it's the "statmasters" that make up the far end of the spectrum. These are the ones that believe in "victory by overwhelming firepower". Level up to 15, and taking on a Level 10 boss is almost boring. These people are, to carry on the boxing metaphor, are what I would call "punchers", a la Mike Tyson in his prime. They don't have a lot of finesse, they don't play elegantly, but they are devastating when they hit, and they hit really hard.
So why does it matter if you are a finesser or a statmaster? Personally, I think it matters a lot. Gameification of objectives needs to consider the fact that people are a spectrum, and that, typically, they fall on either far sides of the finesser or the statmaster, or anywhere in between. Elevate the stats aspects, and people will gravitate towards that will grind to meet the stats. Focus on the finesse aspects and those who are most adept at those aspects will learn how to finesse the game. Give an experience that is too heavy one side or another and you will alienate or underwhelm the participants that fall to either side.
My question is, as I consider activities and goals is to see how do we actually balance these two aspects? Why does Aria of Sorrow speak more to me than, say, Metroid Fusion, even though both games share a lot of the same anatomy and approach? It's because, at least for me, Metroid Fusion rewards the finessers early on; there's little in the way of grinding to get better. Either you figure out how to win the battles or you do not advance. Aria of Sorrow, on the other hand, lets you level up and build strength if you want to. Granted it can take a lot of time, but by doing so, you can both get the stats you want, as well as develop the moves necessary to win (though frankly, at a high enough level, the moves aren't even all that important, just hit and hit until the enemy goes down).
Note, the examples I'm using are a bit old now, but I think the dialog still applies. As we think about the skills we want to acquire, and we say to ourselves "make it like a game", there's more to that statement than meets the eye. Not only if you like to game, but how you like to game, will determine which model and approach works best for your personality and style. As an admitted statmaster, I realize what works for me won't necessarily appeal to the finesser, which means I either have to consider two different methods and models, or try to find a happy medium ground between the two. How about you? Do you consider your gaming style when you engage in these types of things? Does the gaming metaphor work for you when you try to learn new things?
Seasons Change and Energy Fluctuates
It's been interesting to look at the last few years of my blog's existence. In it I can see certain trends and also I can look back and see what mattered to me, what I chose to expend energy on, and what took me away from posting on the blog or making good on things I said I would do.
Why am I thinking of this right now? Because it's been several weeks since I've done an update to the Selenium Practicum, and I've been averaging about a blog post a week, which many people who follow my blog know is abnormal for me. However, it really isn't that abnormal. If I look back at the past few years of data, January and February are strong months of lots of activity, multiple posts (often twice a day or more), and then as the weather gets better and the days get longer, the posts become less frequent. Typically around May, the post spigot dries up to about one every five days, or perhaps one a week.
Now that I've had a chance to review it a bit closer, I realize why. The past few years, I've been involved in a number of initiatives, and each of them seems to have a life cycle of their own. SummerQAmp has a life cycle, preparing for conferences has a life cycle. Posting for other blogs has a life cycle. Add to that the activities my family participates in and its easy to see that the energy isn't really growing or diminishing, it being redirected. For some reason, it's easier to take on a big learning initiative during the winter months. Preparing to write articles also seems to be something that I am better primed to do in the early spring. Summer is about developing presentations and talks. Fall is when I go into gathering mode and try to put it all together and see if I can condense and explain the avenues that I have done down. This may not be absolute, but if the past three years of blog data is to be believed, it's a pretty accurate description of how and where my mind is going.
I had a chat with a fellow tester some weeks back where she shared with me that she really appreciated my blog, and the fact that "she could relate to me as an individual, not just as a tester". I think in some ways, that's what makes writing TESTHEAD critical for me. In some ways, this blog is more than me pontificating on some cool test idea. Very often, test ideas are few and far between in my posts. Events take up a fair amount of space, but they really are dictated by what I actually participate in. While there's a lot of opportunities to learn and practice out there, events happen when they happen. That leaves one more aspect of the blog, and that's the one that really matters the most. Interaction. I interact with this blog in many ways. Its my accountability partner. It lets me know when I'm doing a great job, and when I'm not doing so great. It lets me know who finds what I say interesting, and who doesn't.
Most importantly, though, it gives me a chance to talk about things that I am, dare I say it, not so good at. This is an idea and a philosophy I picked up from Merlin Mann several years ago, in which on one of his podcasts, he said (paraphrased) "I don't blog about this because I'm good at it. I blog about this because I'm actually pretty lousy at a lot of this stuff!" I find it interesting that, when taken on a whole, I am "endorsed" by more people for test automation than I am any other skill. Personally, I find that ironic, because test automation may be the skill I struggle with the most. I'm not innately good at it. I spend a lot of time trying to figure things out that it might take a seasoned programmer just a few minutes to do. Still, because I do want to get better at it, I write about it. A lot. I write about the challenges I face. I write about the frustrations I have. I write about the fact that a lot of the explanations made just aren't that easy to understand and I share my confusion. Maybe that's why so many have endorsed me for test automation, not because I am good at it, but because I have a clear understanding of just how frustrating it can be, and I can offer lots of proof to that fact :).
In any event, Spring is almost over, Memorial Day weekend is coming up (and I am deliberately going "off the grid" to celebrate it :) ). Lots of interesting stuff is going on, and as soon as I know for a fact that we are good to go on a number of things, I'll be talking about quite a few more things, including getting back on the hobby horse that is my Selenium WebDriver Practicum.
Why am I thinking of this right now? Because it's been several weeks since I've done an update to the Selenium Practicum, and I've been averaging about a blog post a week, which many people who follow my blog know is abnormal for me. However, it really isn't that abnormal. If I look back at the past few years of data, January and February are strong months of lots of activity, multiple posts (often twice a day or more), and then as the weather gets better and the days get longer, the posts become less frequent. Typically around May, the post spigot dries up to about one every five days, or perhaps one a week.
Now that I've had a chance to review it a bit closer, I realize why. The past few years, I've been involved in a number of initiatives, and each of them seems to have a life cycle of their own. SummerQAmp has a life cycle, preparing for conferences has a life cycle. Posting for other blogs has a life cycle. Add to that the activities my family participates in and its easy to see that the energy isn't really growing or diminishing, it being redirected. For some reason, it's easier to take on a big learning initiative during the winter months. Preparing to write articles also seems to be something that I am better primed to do in the early spring. Summer is about developing presentations and talks. Fall is when I go into gathering mode and try to put it all together and see if I can condense and explain the avenues that I have done down. This may not be absolute, but if the past three years of blog data is to be believed, it's a pretty accurate description of how and where my mind is going.
I had a chat with a fellow tester some weeks back where she shared with me that she really appreciated my blog, and the fact that "she could relate to me as an individual, not just as a tester". I think in some ways, that's what makes writing TESTHEAD critical for me. In some ways, this blog is more than me pontificating on some cool test idea. Very often, test ideas are few and far between in my posts. Events take up a fair amount of space, but they really are dictated by what I actually participate in. While there's a lot of opportunities to learn and practice out there, events happen when they happen. That leaves one more aspect of the blog, and that's the one that really matters the most. Interaction. I interact with this blog in many ways. Its my accountability partner. It lets me know when I'm doing a great job, and when I'm not doing so great. It lets me know who finds what I say interesting, and who doesn't.
Most importantly, though, it gives me a chance to talk about things that I am, dare I say it, not so good at. This is an idea and a philosophy I picked up from Merlin Mann several years ago, in which on one of his podcasts, he said (paraphrased) "I don't blog about this because I'm good at it. I blog about this because I'm actually pretty lousy at a lot of this stuff!" I find it interesting that, when taken on a whole, I am "endorsed" by more people for test automation than I am any other skill. Personally, I find that ironic, because test automation may be the skill I struggle with the most. I'm not innately good at it. I spend a lot of time trying to figure things out that it might take a seasoned programmer just a few minutes to do. Still, because I do want to get better at it, I write about it. A lot. I write about the challenges I face. I write about the frustrations I have. I write about the fact that a lot of the explanations made just aren't that easy to understand and I share my confusion. Maybe that's why so many have endorsed me for test automation, not because I am good at it, but because I have a clear understanding of just how frustrating it can be, and I can offer lots of proof to that fact :).
In any event, Spring is almost over, Memorial Day weekend is coming up (and I am deliberately going "off the grid" to celebrate it :) ). Lots of interesting stuff is going on, and as soon as I know for a fact that we are good to go on a number of things, I'll be talking about quite a few more things, including getting back on the hobby horse that is my Selenium WebDriver Practicum.
Wednesday, May 15, 2013
The Sound of "White Noise"
Sometimes the most challenging thing we face is trying to balance out multiple and competing demands. When we have a lot of good things we want to do and accomplish, and on the surface, each of them seem to be, if not easy, at least manageable.
The frustrating thing is when each item seems to come at us simultaneously, and we try to cover all bases at the same time. I've long believed that human being cannot really multi-task. We can do a number of things adequately with distractions, but we cannot really do our best work with competing goals. I've also come to see that there's more to this, and that there's an insidious distractor that we often don't give enough heed to. That distraction can be best called "White Noise".
This may make more sense to people who have done multi-track tape recording in the past... you know, that ancient method that we used before digital recording on hard disks became all the rage? One of the cardinal rules of doing tape recording and gathering tracks together is what we called the "law of the bounce". When you recorded multiple tracks, and you needed to make more room, you had to take your currently recorded tracks and "bounce" them down to one or two tracks. However, this came at a price. Each time we did that, we would add about 3dB of white noise or "hiss" to the recording. Doing it once, probably not noticeable. Doing it multiple times quickly overran the recording with hiss and made the sound quality intolerable. Why? Because each time you add 3dB, you get a multiplication of the effect, and soon, that hiss becomes ever present. You no longer hear the music, instead you find yourself distracted by the hiss. In short, the "White Noise" overtakes the quality of the signal.
When we find ourselves with too many good goals and too many things that are demanding our attention, again, it's the same effect as bouncing multiple individual tracks into one. We add white noise, and soon that white noise becomes a constant rumble, to the point that we often lose track of the things we want to be doing to take care of whatever is demanding our attention at that given moment. In short, the White Noise prevents us from actually focusing our attention on what really matters.
In the recording world, we typically do all we can to isolate unique sounds until the very last step. The mix down process for a song can often be a very labor intensive process, where filtering, muting and un-muting, and external effects help us to keep each sound at its proper level until exactly when we need it. The end result is a two track stereo master that has a minimum of artifacts. In our own general work lives, we would do well to take the same lesson. While it is impossible to isolate everything we do into its own sphere and just deal with them one at a time in complete isolation, the quicker and more directly we can focus on thee areas, the less likely they will be to have to be incorporated with other activities. In short, we need to try our best to minimize the bounces, or have a backlog of things we need to do build up to the point where there is a lot of rumbling. By doing so, we can prevent the White Noise from becoming a monster that requires our full attention, to the detriment of the things that we actually want to accomplish.
The frustrating thing is when each item seems to come at us simultaneously, and we try to cover all bases at the same time. I've long believed that human being cannot really multi-task. We can do a number of things adequately with distractions, but we cannot really do our best work with competing goals. I've also come to see that there's more to this, and that there's an insidious distractor that we often don't give enough heed to. That distraction can be best called "White Noise".
This may make more sense to people who have done multi-track tape recording in the past... you know, that ancient method that we used before digital recording on hard disks became all the rage? One of the cardinal rules of doing tape recording and gathering tracks together is what we called the "law of the bounce". When you recorded multiple tracks, and you needed to make more room, you had to take your currently recorded tracks and "bounce" them down to one or two tracks. However, this came at a price. Each time we did that, we would add about 3dB of white noise or "hiss" to the recording. Doing it once, probably not noticeable. Doing it multiple times quickly overran the recording with hiss and made the sound quality intolerable. Why? Because each time you add 3dB, you get a multiplication of the effect, and soon, that hiss becomes ever present. You no longer hear the music, instead you find yourself distracted by the hiss. In short, the "White Noise" overtakes the quality of the signal.
When we find ourselves with too many good goals and too many things that are demanding our attention, again, it's the same effect as bouncing multiple individual tracks into one. We add white noise, and soon that white noise becomes a constant rumble, to the point that we often lose track of the things we want to be doing to take care of whatever is demanding our attention at that given moment. In short, the White Noise prevents us from actually focusing our attention on what really matters.
In the recording world, we typically do all we can to isolate unique sounds until the very last step. The mix down process for a song can often be a very labor intensive process, where filtering, muting and un-muting, and external effects help us to keep each sound at its proper level until exactly when we need it. The end result is a two track stereo master that has a minimum of artifacts. In our own general work lives, we would do well to take the same lesson. While it is impossible to isolate everything we do into its own sphere and just deal with them one at a time in complete isolation, the quicker and more directly we can focus on thee areas, the less likely they will be to have to be incorporated with other activities. In short, we need to try our best to minimize the bounces, or have a backlog of things we need to do build up to the point where there is a lot of rumbling. By doing so, we can prevent the White Noise from becoming a monster that requires our full attention, to the detriment of the things that we actually want to accomplish.
Thursday, May 9, 2013
Web Performance S.F.: A TESTHEAD Live Blog
I needed to spend some time taking care of some other adventures, one of which, as you all might know by now, was the launching of the Miagi-Do Blog. With much thanks to David Greenlees, we now have our own little place on the web to call home. I wrote the second missive to appear there. If you haven't read it, please do.
Also, I was immensely surprised and given a big reason reason to smile yesterday when I saw that Cem Kaner wrote about "credentialing in software testing", in which he said about Miagi-do:
"I think this approach is probably reasonably attainable, and to me, it is definitely credible. Whether it scalable and commercially viable has yet to be seen. I think this is a clear and important alternative [...] I hope it is successful".
We hope it is, too :).
Tonight, however is not about Miagi-do. It's about Meetup Culture in San Francisco and elsewhere in this area that I call home. This continues my thought experiment that you cannot swing an umbrella in San Francisco without hitting a meet up someplace. Tonight brings us to Yelp, and a conversation that's a little different than what I normally come up for. Since STP-CON, however, I have felt that performance is an area I definitely want to beef up my knowledge about, and thus, here I am.
 |
| Some early arrivals/Yelp employees playing pool before the talk starts. |
The Top 5 Performance Shenanigans of CSS Preprocessors
with Nicole Sullivan
Thursday, May 9, 2013
7:00 PM
Yelp
706 Mission Street,
San Francisco, CA
Tonight's topic is focused on a specific aspect of performance, the front end display of web pages and the use of CSS. More to the point, CSS preprocessors and their overall effect on site performance, usability and maintainability, and what we can do to help tip the odds of keeping them on their good sides in our favor.
The sessions started with a conversation about "Living Style Guides". A style guide used to be all about a definitive guide for every item (logo size, spacing, colors, etc.)... do we really care about how the logo for AOL should look on a hat? Structured style guides don't really work. They take months to produce, they are often out of date before they are finished, and they are design centric. Living style guides focus on existing site styles, it lives in code, so that it evolves as the app evolves. Plus, engineering and design teams can work together with it. Design and Engineering have a cap between them, and that gape is often where performance gets optimized or falls all over itself.
An example using Trulia was presented. They demonstrated a process that used a number of steps:
Visual Inventory: Finding the right data to make decisions.
Trulia chose ten pages to analyze, which was meant to get an idea of the site as a whole in the smallest footprint. Often styles are duplicated over and over again. Analyzing duplication and finding variations are the most important first steps. grep is a good tool to start this process. grep for various selectors, properties and values. The main point is to determine how big a mess you have, not a granular count.
Examples shown from Trulia's audit showed lots of duplication (font size appears over 2,000 times, for example. Padding appears almost 4500 times... and this is actually pretty typical). Many of these values can be put into classes to greatly decrease the amount of repetition. Unused CSS and background images can be a huge hit to performance. Remember, all that stuff needs to be loaded, even if it isn't actually having a visible effect for the page. Positioning shows up a lot, and may be a good indication that an abstracted way of laying out the page would be hugely helpful.
Design inconsistency can also be a killer. Profiling and looking to see how often items appear, or don't appear, can help make systems consistent (imagine having dozens of almost the same but not quite the same color of blue. Each one is being used individually, instead of one declaration.
Going through and creating a reference library of the items used, the images, the layout boxes, and any number of element types (images, headers, light boxes, font sizes, font styles, buttons, text boxes, etc.) can help to create a reference for each of the items so that they act as a living style guilde, one that can be seen in real time and more easily determined which aspect and element is preferred to be used, or which items are used frequently vs. one offs.
The gosl is to do this for all component types, and try to identify ans many variations that are there.
Rationalized Styles
When we see the cumulative effect of a million tiny choices, we can see how styles proliferate, see how many styles don't do anything. It can show all present the tangled web of options that are listed and how many of them are "just there', not doing anything.
Get people involved early in the process. This can help make sure that everyone has the time and the focus to see which items are most important. Decide on a single aspect or item so that they can be iterated later. Try to match current styles where possible and appropriate.
Component Library
Once all this is done, we can create a legitimate style guide, and in this process, the discusion shifted to SASS. Some examples shown were the idea of nesting CSS so that you don't have to repeat the class names for every mention. Another option described was the use of mix-ins. when you have a small number of property value pairs or groups. Using @include and @extend allows the designer to create and design a variety of element groups in one wrapper.
Additionally, if you want to see what your output for SASS will be, you can check in the config.rb file. You may be surprised how much complexity is created when you are hoping to simplify CSS calls with extends and includes.
Several examples show how we think that we are simplifying our code and layouts, when in reality, we are very often adding way more than we think we are.
In the process of describing this style guide, we went well beyond five Performance Hits, we're up to number none now, which is "semantic grids are a terrible idea". It causes massive duplication, unreadable HTML and CSS that solves too many problems.
Build simple components first, as they often get used in complex components. An example, they create variables for colors rather than using the direct color value. Create basic typography before more advanced and specialized styles. Creating base variables make it much easier to make changes where needed, and it's easy to see what the changes are and what they represent. Anyone can make changes (well, almost anyone ;) ).
The last part was a live demo of the style guide, which would be way too difficult to explain here, but suffice it to say it was pretty cool to see. It helps me make sense of a lot of the stuff I've seem my friend Rich do for the past couple of years :).
Deployment
Choose the most trafficked pages, most important flows, go for 80% of Page Views. Only convert other pages as new features are added. For new features, add additional functionality to the library first. Use real user measurements (what matters to the users?). The example of the changes made caused a 40% reduction to the overal HTML code transmitted, and they increased page load speed by about 60%.
The examples shown helped me see where a number of areas could be improved in ways I hadn't considered. There's a lot of duplication in sites, and there are some relatively simple tools to find out where. Using OOCSS and SASS together can help people make performant pages. Also, it's important to remember, good performance is a feature :).
About the presenter:
Nicole Sullivan (@stubbornella):
Nicole is a front-end performance consultant, CSS aficionado, and author. She started the Object-Oriented CSS open source project, which answers the question: how do you scale CSS for millions of visitors or thousands of pages? She also consulted with many companies including Facebook, Salesforce, the W3C, Adobe, Paypal, Trulia and Box. She is the co-creator of Smush.it, an image optimization service in the cloud, and CSS Lint (http://csslint.net), a tool that helps correct common CSS errors before they are pushed to production. She is passionate about CSS, web standards, and scalable front-end architecture for large commercial websites. She co-authored The Web Performance Daybook, Vol 2 and Even Faster Websites and blogs at http://stubbornella.org.
Friday, May 3, 2013
So, what is the "Miagi-do School of Software Testing?"
We've been a bit cagey with this over the past few years, but we've decided that we want to do more, be more, and actually draw a line in the sand and say "this is what we stand for, this is who we are, and this is why we are here".
Miagi-do officially placed itself in a place where it could be found today. If you would like to see our opening salvo, Markus Gaertner has written about who we are and why we do what we do. This will be a place where many of us (including me) will be discussing what we do, how we do it, and how others can get involved. With a lot of the conversation going on right now on Twitter regarding certifications, being anti-certification, questioning the validity of certification, questioning the validity of those questioning the validity of certification, and a Twitter account that, at least on the surface, looks to call into question the motives of all in question (pause, long deep breath)... I wanted to make the point that I feel is missing from all of this.
I distrust any certification or course of study that doesn't, in some way, actually have a tester demonstrate their skills, or have a chance to defend their reasoning or rationale behind those skills.
I dislike metrics. I find them easy to exploit and easy to politicize. I find it much more valuable to give someone a challenge and see how they actually do. I value seeing why someone feels the way they do about a topic. I want to hear them explain their position and persuade me, or defend their position if I disagree. I want to watch them sweat through real world problems. If there's any one certification I actually see of real value that commercially available, it's the Cisco Certified Internetworking Expert, because they have had to prove multiple times in actual labs that they can solve a problem and fix issues in real time.
That's why I gravitate towards BBST, Weekend Testing and Miagi-do. I want to see an emphasis on real skill transfer. To borrow my martial arts metaphor from earlier this week (and really, from three years ago), you may or may not have the skills needed to progress and be effective if you pass a MCT. Just like in Aikido, you may know the ins and outs of dozens of Kata moves. You may even be able to explain them real well, but if three guys with knives rush you simultaneously, one of two things is going to happen. You will either fend them off, or you will be taken down. I'm all for you being able to explain what to do. That's great. More important, I want to see that you can also fend off the knife wielders. I may be impressed with your intellect by the former. I'll put my trust in you if you can show me the latter. My goal, and the goal of Miagi-do, is to help as many testers as possible be firmly in the second option.
Miagi-do officially placed itself in a place where it could be found today. If you would like to see our opening salvo, Markus Gaertner has written about who we are and why we do what we do. This will be a place where many of us (including me) will be discussing what we do, how we do it, and how others can get involved. With a lot of the conversation going on right now on Twitter regarding certifications, being anti-certification, questioning the validity of certification, questioning the validity of those questioning the validity of certification, and a Twitter account that, at least on the surface, looks to call into question the motives of all in question (pause, long deep breath)... I wanted to make the point that I feel is missing from all of this.
I distrust any certification or course of study that doesn't, in some way, actually have a tester demonstrate their skills, or have a chance to defend their reasoning or rationale behind those skills.
I dislike metrics. I find them easy to exploit and easy to politicize. I find it much more valuable to give someone a challenge and see how they actually do. I value seeing why someone feels the way they do about a topic. I want to hear them explain their position and persuade me, or defend their position if I disagree. I want to watch them sweat through real world problems. If there's any one certification I actually see of real value that commercially available, it's the Cisco Certified Internetworking Expert, because they have had to prove multiple times in actual labs that they can solve a problem and fix issues in real time.
That's why I gravitate towards BBST, Weekend Testing and Miagi-do. I want to see an emphasis on real skill transfer. To borrow my martial arts metaphor from earlier this week (and really, from three years ago), you may or may not have the skills needed to progress and be effective if you pass a MCT. Just like in Aikido, you may know the ins and outs of dozens of Kata moves. You may even be able to explain them real well, but if three guys with knives rush you simultaneously, one of two things is going to happen. You will either fend them off, or you will be taken down. I'm all for you being able to explain what to do. That's great. More important, I want to see that you can also fend off the knife wielders. I may be impressed with your intellect by the former. I'll put my trust in you if you can show me the latter. My goal, and the goal of Miagi-do, is to help as many testers as possible be firmly in the second option.
Tuesday, April 30, 2013
Becoming a "Certified" BBST Instructor
Disclaimer: this message is mainly being posted for the benefit of members of the Association for Software Testing. So why am I posting it on my blog, and not on AST's site? Well, because I want to do a double appeal, and this has to work on two levels.
First, I want to encourage those out there with a passion to teach about software testing to join AST.
Second, I want to encourage those people out there to become instructors in their own right and help us teach what I personally consider the "colossus of all software testing classes", the Black Box Software Testing courses, aka BBST.
I'm going to come right out and say it. BBST is huge. It's demanding. It's a massive time commitment. It has a high attrition rate of those who take it, mainly because the work load is tremendous. There's a lot to learn, and a lot to do. I also think it's one of the most powerful groups of software testing education classes currently available. Hearing the responses of participants that complete these courses solidifies my belief in that statement after every run.
Can anyone teach these classes? Truthfully, I'll have to say "no". Instructors have to be patient, they have to be willing to take a back seat and let other participants learn from each other as well as instruct. They need to be coaches and mentors. In fact, I think that is the primary role that they really perform. The participants tend to do best when they teach themselves. A BBST Instructor needs to know when to help, and when not to help. When to give a hand, and when to get out of the way. When to give someone encouragement, and sometimes, to break the news to someone that they have been caught cheating, and cannot continue in the courses. Yes, I've had to do that. No, I won't mention names.
So, if I haven't scared you off yet, you may just be wondering "so, Michael, how can I create an exciting, thrilling and lucrative career as a BBST Instructor?" Well, first, if you think that third adjective applies, you're probably not a good candidate. All kidding aside, though it can be fun, and it can be extremely educational, it's a lot of work and as of now, we do not pay our instructors. They are all volunteers.
"Um, yeah, OK, so why would I want to do this again ?"
You can be rewarded for your efforts in a variety of ways. First, you can be rewarded in the fact that, even as instructors, you learn something new every time you teach. Seriously, every single time I teach a class I learn something new based on the insights of the participants and what they bring to the table. Second, it's really cool to be in a position to see people let go of antiquated, so called "best practices" and see them embrace "good practices for the appropriate context". For some, just that is huge!
All right, so now that I've mentioned all of this, what does it take? Glad you asked.
If you want to become a full fledged Lead Instructor (as defined by AST for teaching BBST classes):
1. Each Instructor has to have successfully completed the class they are interested in being an Instructor for (Foundations, Bug Advocacy, Test Design). If you've completed a course, and feel you would like to be an Instructor for that course, let us know. We will do what we can to have you assist in an upcoming course. We have many spots for Assistant Instructors, and we often have classes with two to four Assistant Instructors participating.
2. Take the online Instructor's Course, so that we can show you the avenues that we use and the methods to help instruct and coach participants. This class is currently only offered once a year, but if we were to get enough interested Instructors, we would be happy to double that amount to help more Instructors get the training they need and not have to wait. Also, while it is strongly encouraged to have Assistant Instructors take the Instructor's Course before teaching, it's not mandatory to do so. If you want to assist but haven't taken the Instructor's Course, it's left up to each individual Lead Instructor to decide if they want to take you on for that particular class and mentor you. Successfully completing the Instructor's Course clears that hurdle somewhat. If you want to be a Lead Instructor, the Instructors Course is mandatory.
3. Assist in at least two courses for the class that you would like to be a Lead Instructor for, and have good evaluations from your Lead and other Assistant Instructors who follow and "shadow" the courses. If the reviews are good, and we feel you have the temperament to be a Lead Instructor, then we will set up the opportunity for you to lead a course (note: you can only lead courses that you have already participated as an Assistant Instructor at least twice).
4. Each Lead Instructor gets a "shadow" lead for the first time they lead a course. That shadow lead evaluates their performance, and makes recommendations if they should lead future courses. If the reviews are favorable, then they get a chance to "fly solo" with a class. If they need a little more practice, we can offer to shadow them a second time. For most, two shadow runs is usually sufficient.
5. After an Instructor has been the Lead for two classes, then they can be "certified" for that course. This certification is a collection of reviews, both from fellow instructors as well as participants. The certification process is where AST states that we feel this person has the skills and the temperament to deliver a quality instruction experience. Also, the BBST certification allows that instructor to take the BBST materials and teach them anywhere they want to, with AST's approval. While that prospect may only appeal to a handful of people, the abilities that are reflected in being able to teach and mentor participants through such a dense and multi-layered class series shows a solid commitment to being a good mentor and instructor, and should impress just about anyone who takes training seriously.
For those who have been keeping track, that means every "certified" Lead Instructor for any of the courses offered have, at bare minimum, been through the material a minimum of five times (once as a participant, at least twice as an Assistant Instructor, and at least twice as a Lead Instructor).
So, if I haven''t scared you off yet, there it is. In a nutshell, that's the BBST approach to developing and "certifying" Lead Instructors. My goal is to see a way to teach solid software testing skills to as many people as possible, without diluting or dumbing down the material. That's going to require a dedicated group of instructors, and those instructors are, quite frankly, those of you that care enough to want to see that level of instruction grow and thrive.
In other words, those of you actually reading this right now :).
First, I want to encourage those out there with a passion to teach about software testing to join AST.
Second, I want to encourage those people out there to become instructors in their own right and help us teach what I personally consider the "colossus of all software testing classes", the Black Box Software Testing courses, aka BBST.
I'm going to come right out and say it. BBST is huge. It's demanding. It's a massive time commitment. It has a high attrition rate of those who take it, mainly because the work load is tremendous. There's a lot to learn, and a lot to do. I also think it's one of the most powerful groups of software testing education classes currently available. Hearing the responses of participants that complete these courses solidifies my belief in that statement after every run.
Can anyone teach these classes? Truthfully, I'll have to say "no". Instructors have to be patient, they have to be willing to take a back seat and let other participants learn from each other as well as instruct. They need to be coaches and mentors. In fact, I think that is the primary role that they really perform. The participants tend to do best when they teach themselves. A BBST Instructor needs to know when to help, and when not to help. When to give a hand, and when to get out of the way. When to give someone encouragement, and sometimes, to break the news to someone that they have been caught cheating, and cannot continue in the courses. Yes, I've had to do that. No, I won't mention names.
So, if I haven't scared you off yet, you may just be wondering "so, Michael, how can I create an exciting, thrilling and lucrative career as a BBST Instructor?" Well, first, if you think that third adjective applies, you're probably not a good candidate. All kidding aside, though it can be fun, and it can be extremely educational, it's a lot of work and as of now, we do not pay our instructors. They are all volunteers.
"Um, yeah, OK, so why would I want to do this again ?"
You can be rewarded for your efforts in a variety of ways. First, you can be rewarded in the fact that, even as instructors, you learn something new every time you teach. Seriously, every single time I teach a class I learn something new based on the insights of the participants and what they bring to the table. Second, it's really cool to be in a position to see people let go of antiquated, so called "best practices" and see them embrace "good practices for the appropriate context". For some, just that is huge!
All right, so now that I've mentioned all of this, what does it take? Glad you asked.
If you want to become a full fledged Lead Instructor (as defined by AST for teaching BBST classes):
1. Each Instructor has to have successfully completed the class they are interested in being an Instructor for (Foundations, Bug Advocacy, Test Design). If you've completed a course, and feel you would like to be an Instructor for that course, let us know. We will do what we can to have you assist in an upcoming course. We have many spots for Assistant Instructors, and we often have classes with two to four Assistant Instructors participating.
2. Take the online Instructor's Course, so that we can show you the avenues that we use and the methods to help instruct and coach participants. This class is currently only offered once a year, but if we were to get enough interested Instructors, we would be happy to double that amount to help more Instructors get the training they need and not have to wait. Also, while it is strongly encouraged to have Assistant Instructors take the Instructor's Course before teaching, it's not mandatory to do so. If you want to assist but haven't taken the Instructor's Course, it's left up to each individual Lead Instructor to decide if they want to take you on for that particular class and mentor you. Successfully completing the Instructor's Course clears that hurdle somewhat. If you want to be a Lead Instructor, the Instructors Course is mandatory.
3. Assist in at least two courses for the class that you would like to be a Lead Instructor for, and have good evaluations from your Lead and other Assistant Instructors who follow and "shadow" the courses. If the reviews are good, and we feel you have the temperament to be a Lead Instructor, then we will set up the opportunity for you to lead a course (note: you can only lead courses that you have already participated as an Assistant Instructor at least twice).
4. Each Lead Instructor gets a "shadow" lead for the first time they lead a course. That shadow lead evaluates their performance, and makes recommendations if they should lead future courses. If the reviews are favorable, then they get a chance to "fly solo" with a class. If they need a little more practice, we can offer to shadow them a second time. For most, two shadow runs is usually sufficient.
5. After an Instructor has been the Lead for two classes, then they can be "certified" for that course. This certification is a collection of reviews, both from fellow instructors as well as participants. The certification process is where AST states that we feel this person has the skills and the temperament to deliver a quality instruction experience. Also, the BBST certification allows that instructor to take the BBST materials and teach them anywhere they want to, with AST's approval. While that prospect may only appeal to a handful of people, the abilities that are reflected in being able to teach and mentor participants through such a dense and multi-layered class series shows a solid commitment to being a good mentor and instructor, and should impress just about anyone who takes training seriously.
For those who have been keeping track, that means every "certified" Lead Instructor for any of the courses offered have, at bare minimum, been through the material a minimum of five times (once as a participant, at least twice as an Assistant Instructor, and at least twice as a Lead Instructor).
So, if I haven''t scared you off yet, there it is. In a nutshell, that's the BBST approach to developing and "certifying" Lead Instructors. My goal is to see a way to teach solid software testing skills to as many people as possible, without diluting or dumbing down the material. That's going to require a dedicated group of instructors, and those instructors are, quite frankly, those of you that care enough to want to see that level of instruction grow and thrive.
In other words, those of you actually reading this right now :).
Monday, April 29, 2013
TESTHEAD REDUX: Aikido and the Role of Certification
When I wrote the original post for this back in 2010, I think it was the first time I was willing to break out and say "I don't even know what I'm hoping to find with this". It was prompted by my looking at a variety of "certification" options out in the testing market at the time. Most of them I had just started to hear about, many of them were somewhat nebulous, all of them made me feel somewhat uneasy. At the time I said the following (note, I used my experiences with the martial art Aikido as an analog to my understanding of the certification landscape at the time):
In my mind, this is the big thing that is missing from most of the certification paths that I have seen to date. There is a lot of emphasis on passing a multiple choice test, but little emphasis on solving real world problems or proving that you are actually able to do the work listed on the exam, or that you genuinely possess the skills required to test effectively.
The other issue that I have with this is that, just like in an actual real world confrontation, some of the best practitioners of Aikido may not be the best at articulating each and every step, but my goodness they are whirlwinds on the mat and on the street! This is because they are instinctive, and their training has been less on the intellectual explanation and more on the raw “doing”!
The reason I mention these details is that I still, all these years later, have yet to find a true certification that actually leads to the goals I am after and desire, but I also have found several examples of exactly what I want to see certification become. In short, I want to see a certification that really lives up to the principles of Aikido. I want to see testing as a martial art in its own right (with perhaps a de-emphasizing of the "martial" aspect. Perhaps a better phrase would be a "philosophical art").
In my mind, this is the big thing that is missing from most of the certification paths that I have seen to date. There is a lot of emphasis on passing a multiple choice test, but little emphasis on solving real world problems or proving that you are actually able to do the work listed on the exam, or that you genuinely possess the skills required to test effectively.
The other issue that I have with this is that, just like in an actual real world confrontation, some of the best practitioners of Aikido may not be the best at articulating each and every step, but my goodness they are whirlwinds on the mat and on the street! This is because they are instinctive, and their training has been less on the intellectual explanation and more on the raw “doing”!
The reason I mention these details is that I still, all these years later, have yet to find a true certification that actually leads to the goals I am after and desire, but I also have found several examples of exactly what I want to see certification become. In short, I want to see a certification that really lives up to the principles of Aikido. I want to see testing as a martial art in its own right (with perhaps a de-emphasizing of the "martial" aspect. Perhaps a better phrase would be a "philosophical art").
Before I get too far into this, I will say already that the three things I am going to suggest need to be taken with a very large grain of salt. Why? Because I have a vested interest in all of them, but not for the reasons you may be thinking. A disclaimer... I make no money from any of these endeavors. In fact, in some ways, I forego earning money in other ways so that I can champion them. If I wanted to make myself the equivalent of the impoverished warrior monk, or a Zatoichi, I may have found the perfect recipe in these three examples ;). Nevertheless, I do them because of the value that I believe they provide, and from the anecdotal value that others come back to me and say that they offer.
BBST - BBST is the Black Box Software Testing courses offered by the Association for Software Testing and others. Note, you can get very close to 100% of the benefits of BBST without ever taking a class. All of the materials (the lectures, the course notes, the readings, etc) are available online. What's not readily available online is the course quizzes and exams, and the ability to be coached by other testers who help instruct the course. There is a cost associated with it ($125 for the Foundations course, $200 for the Bug Advocacy and Test Design courses), but the costs are used to pay for the hosting of the instances of the class, the servers, and administrative overhead. At this point in time, every Instructor for BBST is a volunteer, i.e. we don't get paid to do what we do.
Weekend Testing - while BBST is one of the best direct trainings out there, Weekend Testing is, IMO, one of the best organized skills workshops held on a regular basis for testers to sharpen their swords on a variety of topics. Weekend Testing works on a variety of levels. It has much to offer the beginner who wants to learn how to test. It has much to offer the intermediate tester who wants to mentor newer testers, and likewise learn more themselves. It has much to offer advanced testers who can work to develop their skills as leaders by facilitating sessions and designing interesting and unique content to talk about, learn from, and make for a positive influence in the broader testing community. Also, unique to Weekend Testing is the fact that every session is archived. If you would like to show someone just how much you contributed, it's there in black and white, for all the world to see.
Weekend Testing has several chapters that are in various stages of operation. Chapters have been formed in India, Australia/New Zealand, Europe and the Americas. Currently, the India and the Americas chapters are the most active, but all it takes is willing folk to facilitate and have a strong desire to help testers improve their craft for more chapters to open up where they are needed (hint, I would have no problem seeing a South America Chapter develop, and we have yet to see a chapter develop in Africa. So there's plenty of room to grow, as far as I can see :) ).
Weekend Testing has several chapters that are in various stages of operation. Chapters have been formed in India, Australia/New Zealand, Europe and the Americas. Currently, the India and the Americas chapters are the most active, but all it takes is willing folk to facilitate and have a strong desire to help testers improve their craft for more chapters to open up where they are needed (hint, I would have no problem seeing a South America Chapter develop, and we have yet to see a chapter develop in Africa. So there's plenty of room to grow, as far as I can see :) ).
The Miagi-do School of Software Testing - this is the one that probably means the most to me, and yet it's the one that will, guaranteed, never make me rich. Well, none of them will, but unlike BBST and Weekend Testing, which could be used as a marketable option or product, Miagi-do cannot. Actually, I should say it will not, as long as the founders have anything to say about it. It's not a not-for-profit. It's a ZERO-profit. It's also a ZERO-income enterprise. No money ever changes hands, and likely, no money ever will. People have to seek out the school, have to show they are willing, have to face multiple testing challenges, and actually put in a lot of work that leads to the betterment at large of the software testing community.
Everyone's path is different, but my path was through signing up with AST and learning about the BBST classes, taking them as a participant, and then offering to teach them over the past three years. It included my producing a podcast dedicated to software testing topics, and frequently researching and presenting my own findings in episodes where I was features as a guest or a panelist. It involved my getting into Weekend Testing as then offered in Europe, and making enough of a commitment to it to be considered knowledgeable enough to facilitate, and then bring Weekend Testing to the Americas, where I have fostered its growth and development (along with several others, to be fair) for the past two and a half years. It involved writing in many areas, including the book "How to Reduce the Cost of Software Testing", as well as multiple articles for other distribution channels (ST&QA, Techwell, Tea Time for Testers, The Testing Planet, plus guest blogs for numerous companies and outlets).
In between all of this, I have sat for and failed, and then later passed, several testing challenges, each one with the idea that I would demonstrate to my testing peers what I knew how to do, and what I didn't know. The Black Belt/Instructor level that I hold in Miagi-do may be laughed at by some. What does it mean, really? On paper, and in the eyes of HR departments, probably less than nothing. If, however, you and others out there feel that the talks I have given, the sessions I have facilitated, the courses I have taught, the articles I have written, and the podcasts I have recorded have helped, in some small way, to the improvement and betterment of the software testing community, then my black belt speaks volumes. In fact, it's my hope that everything else I have done, and will continue to do, makes the mention of a black belt completely irrelevant.
In short, I am not my "certification". I am the ideas and the experiences that went into it. The fact that my certification is one that I have made for myself, surrounded with like minded people that I respect and admire, means more to me than any certification that can be given to me by any "officially sanctioning body" and yes, I'll include my Bachelors Degree in that list of "inferior certifications".
While a "certification" may carry someone into a second round interview, I will much frankly prefer to see my fellow Miag-do ka, BBST instructors and Weekend Testing facilitators on any project I would hope to lead and own. Why? Because I already know what they can do. I've seen it multiple times. In a dark alley situation, I already know they can fight, and I also know they won't run away :)!
Thursday, April 25, 2013
Final Day of #STPCON (Live Blog)
Today is bittersweet. It's the last day of STP-CON, and the last day of what has proved to be an intense, interesting, fascinating, and thoroughly enjoyable few days. I love the opportunity to get together with people from different industries, experiences, and world views and learning about what works and doesn't work for them, as well as getting advice about what is and isn't working for me. Sleep tends to become a very limited resource, but that's because the time spent after talks and workshops conferring with others is what makes these events so valuable.
If I can make any one piece of advice for anyone who attends a conference, make sure that your emphasis is on the "conferring", and understand that most of the real learning, breakthrough moments and epiphanies will not be in Q&A periods in the closing minutes of a presentation, they will be in laughter and discussion over Red Sauced Pork Barbecoa at a wonderful Mexican restaurant, or watching your conference mates look at you with both exasperation and understanding as they fall prey to another round of "The Pen Test".
Yep, this morning will be the closer, and more to the point, I will be the closing speaker. Well, OK, not exactly the closing speaker, but I'm in the last track session slot along with four other topics. Part of the dynamic today is going to be that the back of the room will be filled with suitcases and duffel bags and people who may be asking me subtly with their eyes "ummm, are you gonna' wrap up a little early by chance, because I've got a flight to catch!" My answer is "I'll do my best, but I hope that I can offer something worthwhile enough to make you want to stick around at least until the end of the session. For obvious reasons, I will not be able to live blog my own session (maybe someone else will help me do that via Twitter and I'll pull in their comments, but I will post a synopsis after I'm done.
With that, I need to go and check out, as well as get set up for the closing three speakers I will be attending. Lynn McKee and Matt Heusser, y'all best get ready :).
Oh, a shout out to Mike Himelstein, a new friend from Atlanta. He's been drawing little sketches of the attendees and speakers, and he shared his "vision of me" while I've been here...
I think I have a new avatar :).
-----
And we're back. Breakfast is over and we are now getting underway with Lynn McKee. For those who don't know Lynn, I'm happy to count her as both a terrific colleague and a great griend. Lynn invited me up to facilitate the POST 2012 peer conference in Calgary, and additionally, she helped me complete a bucket list item by taking me snowboarding at Banff. Oh, and she's also an excellent Agile Coach and team mentor, which is part of why she's bale to speak here and now. I'm excited that Lynn is getting a Keynote, though I must say I'm slightly sad that her "wonder twin" isn't here to see it (Nancy Kelln, we miss you!).
In agile teams the whole team is responsible for quality. TDD during design to help identify the right design, acceptance tests are built, automation testing is done throughout the entire cycle. This can be a good and bad thing for the Lone Tester because it can dramatically help you cover the system but also requires a great amount of time. Always something to do.
Theres always a need for testers on an Agile team. Testing is always happening at all levels, which is a wonderful change (especially when used correctly), but there is no 'quality police' - especially not for the testers. This has changed the role for the testers, it requires a variety of skills like domain knowledge and technical competency to interact with the development team. According to Bret Pettichord: Agile Testing is/are the: "Headlights of the project - where are you now? Where are you headed?" It can "[p]rovide information to the team - allowing the team to make informed decisions." With the information we provide managers they can make the decisions they need to about the product / project.
The testing we do on projects is a hedge on the risk of the product. Sure testing is a cost center but its a hedge against loosing customers because companies don't know anything about the products they are shipping. In order to help with this hedge, Lone Testers in an Agile Development Team need to be agile themselves. There's plenty of room for "testing" but we need to broaden our toolkit - we (testers) need to adapt to different expectations. A good example of this is when you move from one software shop to another.
Lone Testers should automate where you can. If you don't have the skills that's ok. You can start small while you learn. You don't need to create huge amounts of automation, just something alone the lines of 'Sunshine tests' (or smoke tests) - a small set of tests that can help you look at the broad picture of the software. This is the perfect thing for a Lone Wolf to attack first. Michael's a fan of the 'when in rome strategy' which means you look at whatever languages, tools, your development team uses and you use the same. If they are using Junit, you can use Junit for your tests. Then you can share the results or problems with them and they'll be more likely to help you because they see the common connection.
The disadvantage to automation is it's passive "checking" as opposed to active testing. When you spend time building automation it means you aren't spending time exploring and/or testing. Tests can quickly become stale and you have to consistently upkeep those tests.
[Michael: Clarifying. I meant that having the machine just running the automated tests and accepting the passes at face value is passive checking. The process of developing the automation goers through lots of iterations, with its own debugging and learning, so to be clear, the creation process of automation involves a lot of direct active testing.].
Testers are more than people looking through tests, trying to break things. Testers are like anthropologists:
If I can make any one piece of advice for anyone who attends a conference, make sure that your emphasis is on the "conferring", and understand that most of the real learning, breakthrough moments and epiphanies will not be in Q&A periods in the closing minutes of a presentation, they will be in laughter and discussion over Red Sauced Pork Barbecoa at a wonderful Mexican restaurant, or watching your conference mates look at you with both exasperation and understanding as they fall prey to another round of "The Pen Test".
Yep, this morning will be the closer, and more to the point, I will be the closing speaker. Well, OK, not exactly the closing speaker, but I'm in the last track session slot along with four other topics. Part of the dynamic today is going to be that the back of the room will be filled with suitcases and duffel bags and people who may be asking me subtly with their eyes "ummm, are you gonna' wrap up a little early by chance, because I've got a flight to catch!" My answer is "I'll do my best, but I hope that I can offer something worthwhile enough to make you want to stick around at least until the end of the session. For obvious reasons, I will not be able to live blog my own session (maybe someone else will help me do that via Twitter and I'll pull in their comments, but I will post a synopsis after I'm done.
With that, I need to go and check out, as well as get set up for the closing three speakers I will be attending. Lynn McKee and Matt Heusser, y'all best get ready :).
Oh, a shout out to Mike Himelstein, a new friend from Atlanta. He's been drawing little sketches of the attendees and speakers, and he shared his "vision of me" while I've been here...
I think I have a new avatar :).
-----
And we're back. Breakfast is over and we are now getting underway with Lynn McKee. For those who don't know Lynn, I'm happy to count her as both a terrific colleague and a great griend. Lynn invited me up to facilitate the POST 2012 peer conference in Calgary, and additionally, she helped me complete a bucket list item by taking me snowboarding at Banff. Oh, and she's also an excellent Agile Coach and team mentor, which is part of why she's bale to speak here and now. I'm excited that Lynn is getting a Keynote, though I must say I'm slightly sad that her "wonder twin" isn't here to see it (Nancy Kelln, we miss you!).
Change is easy, right? Sure it is. Oh, you mean effective change, one that is internalized by your team and organization. Yeah, sorry, that's hard. And frequently unsuccessful. Why? Often, it's because there are conflicting goals and visions. Often it's the blind leading the blind. "We don't know where we're going, but we're making great progress!
Some people have an appetite for change. People who attend conferences, talk at them, stay up late talking about testing and gadgets, these are all likely early adopters or at least have an understanding or appreciation for change. Others have a different appetite or desire for change, ranging from totally willing to completely unwilling.
For change to happen, we have to create a sense of urgency. For those of us who are testers, what are we willing to do to create that sense of urgency? Ultimately, we need to show the business value to the organization as to why testing not only matters, but is vital. We also need to show what we can bring to the change and what our value truly is.
Additionally, there needs to be a coalition of the willing, and perhaps even the fool-hearty. There needs to be expertise in this group so that the change can happen. there also needs to be credibility, and someone willing to run up San Juan Hill or be one of the Light Brigade (hopefully with the results closer to the former rather than the latter).
Testers can be transformative by daring to place themselves in the cross hairs. We may need to prove we are willing and able to do what it takes to gain trust, or if nothing else, scare everyone else so bad that the treat us like Oda Nobunaga's solders who walk through the forest with their match locks lit (Pete, that one's for you ;) ).
Key influencers are important. They don't have to necessarily be the visible ones, but if you can get those in the organization to believe in your view, and if they believe that what you are striving to offer will bring real value to the organization, that could be the catalyst to make it all happen.
Making testing great is a wonderful goal, it's a terrific vision, but what doe it ultimately offer to the organization? If we cannot answer that question, then we are not likely to make headway, even if they can agree that "better testing" is a wonderful goal. Better testing for what? If it's just for the sake of kudos, or for team cohesion, neat goal, but maybe not compelling enough. We're a cost center. Tester's don't make money for an organization. Sorry, but unless you are selling test services as your product, no testers at a company actually make cash for the company. What we do provide, however, is a hedge. We can safeguard revenue earned, and we can prevent revenue erosion. Make no mistake, that's huge! What is the value of a thoroughly tested product? It's hard to quantify, but there's no question what the value of a 1 or a 2 rating in a app store is. THAT is something that the business can understand!
So what are the obstacles that can get in the way of change? People can get in the way. Sometimes WE are those people. Processes can get in the way. Sometimes they are well intentioned but pointless. Sometimes other people's perceptions of us get in the way. Sometimes the technology stack we use can be an impediment (Rails is a great framework, but if your entire infrastructure and product is developed for Win32, that may be a real problem to change.
Buzzwords abound, and often the buzzwords start flying with little understanding as to what they mean. Avoid this. If you use a buzzword, make sure that you a clear on what it is, what it means, and what it's really going to provide. More to the point, what does the buzz word add to the bottom line? Also, if you are going to use mnemonics, make sure that you have spent time to show what they are, what they mean and how they are used. Heuristics are valuable, but if we cannot communicate what they help us do, no one will care or invest the time to make it work. Slogans, jargon, call it what you may, make sure that you are clear as to what they are, what they mean, and what they do.
One of the beautiful tools that Lynn mentions, and I believe strongly in them as well, is the "quick win". Erasing Technical Debt is a lot like erasing Financial Debt. It's a big challenge, and it's a long hard slog. To make things happen, and to build heart and morale, there needs to be early wins and quick wins. Pick of a small problem and solve it before taking on the Colossus. those of us who play video games understand we battle small fry first to level up so we can take on the level bosses later. Short term wins give us strength, flex out muscles, and give us confidence to take on bigger problems.
Trim the fat where you can. Don't focus on processes that hinder you, work on the activities that get you results. Learn where "good enough" really is. Snazzy looking docs are nice, but if you are spending all your time making snazzy looking docs, that is time you could be doing real and valuable testing. Learn what really matters for reporting, and provide just that. I dare you! See what would happen if you gave a trim, slim, solid summary of what you have done an what you have found. See what happens. Will you get yelled at because you didn't attach the cover page to your TPS report? maybe the first time, but if you do it and show that your testing is happening and you are finding really great insights for the team, I'll bet you that the process will change (and yes, I am willing to take that bet, at least for most organizations).
There is risk in every project, and there is risk in change. How much and to what level varies in organizations, but nothing is fool-proof. We all need to focus on and show that w understand the risks. We have to give the information that will either tell people that we are looking good or we are in significant trouble, or something in between. It's possible we may stop the train. It's possible the train will keep going. If we have made an impact and allow our executives to sleep well at night, then good going. It certainly beats the alternative.
Change and transformations are not easy. Not if it's going to stick. Not if it's going to really change hearts and minds. Not if it's going to clobber the bottom line. Not if the group doesn't have a long view. Some people will not get on board. Some people might actually leave over the choices made to change. The phrase "you can change your organization, or you can change your organization" really rings true. Sometimes, the change that is needed is that YOU may need to go elsewhere. Are you brave enough to do that? Do you have the will to do that? Are you willing to "fall on your own sword"? Some people are, but many are not. There is the process in forging steel for swords called "the refiners fire". To purify steel, you have to heat it, beat it, and then plunge it into water to cool and harden it. The steel goes through a lot, but the end result is a hard and strong metal ready to cut through anything.
Lynn, thanks so much, you rocked this :).
-----
Now we are talking about "Where Do Bugs Come From" and Matt Heusser is going to be our ringleader. He's promised to make this different than all other presentations we've seen this week. Instead of a lecture, we're going to have a discussion that everyone can get into.
Often, when we go to conferences, we come back with lots of ideas and enthusiasm, but when we present our ideas, we often get blank stares, crossed arms, and reasons why we can't do that. that happens, so what can w do so that we don't get into that position.
The first thing we need to do is stop asking permission. Just do what you plan to do. Make an experiment out of it. Decide to implement whatever key item you want to do, and figure out how you can put in into play on the first day you get back to work. Don't ask permission. Just go for it. That's at least my plan ;).
It's easy to say we try to find bugs. Every program has bugs. Even "Hello World" has bugs, if you dig deep enough. So yea, finding bugs is important, but beyond the trivial, where do bugs really come from? In truth, they all come from us, i.e. people. They come from programmers, but that's really only a small part of the story. Hardware can actually cause problems, voltages can flip bits, actual insects can fly into relays (Grace Hopper's legendary "bug" was exactly that, a moth that got caught in the circuitry and short circuited something).
Bugs are not just glitches in code. They can be glitches in requirements, glitches in behavior, glitches in emotions, and glitches in markets. Seems a bit over-reaching? Maybe, but really, what is a bug? It's something that doesn't work the way we want it to, or someone else wants it to. Can a product work exactly the way it's "intended" and still be a bug? Absolutely! If the CEO decides that the paragraphs in a legal disclaimer need to be reversed, even though it is written as intended, and they decide it needs to be changed, now, then yes. what was working as written can become a P1 bug by "royal decree". Actually, that's not the best characterization. The real reason it was a bug was because there was a hidden stakeholder that wasn't considered.
There are differences in the way that desktop, web and mobile display and process events. We have issues with usability and also with intuitiveness. There's also conditions that bring to light issues that you just won't find unless they are met. How does your app work with a nearly dad battery on a mobile device? What happens when you plug in the power cord to start recharging? how about while you are walking around between cell towers? Different environments can bring to light many interesting anomalies. If we are aware of the possibilities, we can consider them, and perchance find them.
Cultural assumptions often come into play. When localization becomes part of a product, then there's a real "lost in translation" issue that can come into play. Not lost in translation of requirements, but genuinely an incorrect translation. One of the most interesting things I've seen was when one of my blog posts was summarized on a site in Japan. When I translated the site from Japanese to English, were I not the one to have written the original blog post, I would not have been able to make much sense of what the original article was actually saying. The reason? Not that the Japanese article was wrong, but that the literal translation that was provided genuinely didn't make sense to me as an English speaker. The grammar was still Japanese rules of speech, which frankly just don't make sense in a direct word for word translation. Real localization efforts go deeper than that, of course, but it helps emphasize just how that can become an issue if we are not really doing our due diligence.
In addition to understanding where bugs come from, it's also important that we understand the risks that those bugs represent, and we then have to decide if we genuinely care about them. Not all bugs are created equal, and what bothers one group of people may not bother another group at all. It may be so esoteric that the odds of it ever being expressed is .0001%. Do I care with my facebook page? Of course not. Would I care with the computer that controls the landing gear on the airplane that's taking me home tonight? Absolutely!!!
OK, so we have risks. What can we do to mitigate those risks? There's lots of things. We can use prototypes to test ideas. We can manage expectations. We can iterate and examine stories in smaller increments. We can pair testers with devs. We can start early and test requirements. This leads into the "three amigos meeting" model (one that Socialtext uses actively with our kick-offs, I might add; Google "Three Amigos Meeting" and you'll find lots of stuff to look at :) ). The main takeaway for that is "bring the stakeholders together and make sure everyone agrees on the work to be done".
So some takeaways...
- Pick latest critical bugs in production
- Map them to techniques to mitigate risk
- Discover what you are not doing enough of.
Oh, and just go and do this. Don't ask permission. You don't have to. Just delight them that you are doing it ;).
-----
And now it falls to me. The closer, the last man standing. I'm ready now to address the Lone Wolf, the Armys of One, the Lone Ranger... and hey, if you work with a team, this may still prove to be relevant, because all of us Lone Wolf it some of the time. What craziness will come out of my mouth over the next hour and fifteen minutes? I guess we will just have to see...
[Editors note: Michael is talking right now, but the written words coming your way from here are courtesy Chris Kenst].
Michael's talking about his background, about how he made the switch to an agile environment in January of 2011 and not so long ago moved away from being a Lone Wolf and now works with 5 other testers (for some reason he seems quite excited)!
He's been through the traditional waterfall approach, as a lone tester, and now as a part of a "wolf pack" he's working his way through an agile process. It's all about learning, exploring in small chunks and iterating. This involves Test Driven Development (TDD) to help design the code. The class is asked, does anyone think TDD is testing? No one raised their hands. It's not testing. Its a design tool. In fact testing (not TDD) starts at testing the requirements often during the kickoff meeting. During these 3 amigos meetings the testers ask questions about the requirements in order to understand how the developer interpreted it.
In agile teams the whole team is responsible for quality. TDD during design to help identify the right design, acceptance tests are built, automation testing is done throughout the entire cycle. This can be a good and bad thing for the Lone Tester because it can dramatically help you cover the system but also requires a great amount of time. Always something to do.
Theres always a need for testers on an Agile team. Testing is always happening at all levels, which is a wonderful change (especially when used correctly), but there is no 'quality police' - especially not for the testers. This has changed the role for the testers, it requires a variety of skills like domain knowledge and technical competency to interact with the development team. According to Bret Pettichord: Agile Testing is/are the: "Headlights of the project - where are you now? Where are you headed?" It can "[p]rovide information to the team - allowing the team to make informed decisions." With the information we provide managers they can make the decisions they need to about the product / project.
The testing we do on projects is a hedge on the risk of the product. Sure testing is a cost center but its a hedge against loosing customers because companies don't know anything about the products they are shipping. In order to help with this hedge, Lone Testers in an Agile Development Team need to be agile themselves. There's plenty of room for "testing" but we need to broaden our toolkit - we (testers) need to adapt to different expectations. A good example of this is when you move from one software shop to another.
Lone Testers should automate where you can. If you don't have the skills that's ok. You can start small while you learn. You don't need to create huge amounts of automation, just something alone the lines of 'Sunshine tests' (or smoke tests) - a small set of tests that can help you look at the broad picture of the software. This is the perfect thing for a Lone Wolf to attack first. Michael's a fan of the 'when in rome strategy' which means you look at whatever languages, tools, your development team uses and you use the same. If they are using Junit, you can use Junit for your tests. Then you can share the results or problems with them and they'll be more likely to help you because they see the common connection.
The disadvantage to automation is it's passive "checking" as opposed to active testing. When you spend time building automation it means you aren't spending time exploring and/or testing. Tests can quickly become stale and you have to consistently upkeep those tests.
[Michael: Clarifying. I meant that having the machine just running the automated tests and accepting the passes at face value is passive checking. The process of developing the automation goers through lots of iterations, with its own debugging and learning, so to be clear, the creation process of automation involves a lot of direct active testing.].
Testers are more than people looking through tests, trying to break things. Testers are like anthropologists:
- Observe the development team
- Look for feedback form actual users
- Work with content developers
- Discover the underlying and unspoken culture
Being a lone tester requires that you are a good communicator. You've got to be able to build bridges in the company, talk with developers and product owners and this includes all the different languages that each speak. Its about playing well with others (think of pairing). Paring can be tester/developers, tester/support person, tester/customer, tester designer. Lone testers should participate with planning and standups with the development team and they should become a domain expert about the customer needs.
Lone testers need to develop your craft. This is hard to do when you are the lone person, you'll have to reach outside your organization. Mentors will come from other sources, sure they can be a manager or developer if they've got the time to work with you. Otherwise you can reach out to other Lone Testers in other organizations that you can learn from. Michael is reciting some of the values of Context Driven Testing school of testing. Good software testing is a challenging intellectual process. This requires judgement and skill exercised throughout the project.
Lone Wolves, you are not alone. (For some reason I hear the Michael Jackson song in my head now. Weird) You have many allies you just aren't aware of them. Take advantage of the meetups in your area. Beer, pizza and some similar challenges are the basis for many meet ups. In-company people like developers, support people and designers can also provide feedback and support. Remember everyone in the Agile team does testing - its not all on you.
... and with that, I'm back. My thanks to Chris Kenst for being "virtual me" for a bit. I'm done. Deep breath... and now the room is being taken apart. This is the oficial end of the "formal conference" but as we all know, the conference may be over, but the conferring can still happen... and that's where I'm going now. I have a lot of new friends to talk to and learn from.
See y'all later :).
Lone testers need to develop your craft. This is hard to do when you are the lone person, you'll have to reach outside your organization. Mentors will come from other sources, sure they can be a manager or developer if they've got the time to work with you. Otherwise you can reach out to other Lone Testers in other organizations that you can learn from. Michael is reciting some of the values of Context Driven Testing school of testing. Good software testing is a challenging intellectual process. This requires judgement and skill exercised throughout the project.
Lone Wolves, you are not alone. (For some reason I hear the Michael Jackson song in my head now. Weird) You have many allies you just aren't aware of them. Take advantage of the meetups in your area. Beer, pizza and some similar challenges are the basis for many meet ups. In-company people like developers, support people and designers can also provide feedback and support. Remember everyone in the Agile team does testing - its not all on you.
... and with that, I'm back. My thanks to Chris Kenst for being "virtual me" for a bit. I'm done. Deep breath... and now the room is being taken apart. This is the oficial end of the "formal conference" but as we all know, the conference may be over, but the conferring can still happen... and that's where I'm going now. I have a lot of new friends to talk to and learn from.
See y'all later :).
Wednesday, April 24, 2013
Day 2 at STP-CON (Live Blog)
Good morning everyone, STP-CON is once again underway. Rich is onstage now and getting today's program in order. There's a full and action packed day in store, plus something interesting later today with Matt Heusser called Werewolf. It's limited to twenty people, and I'm going to help support the event, not specifically participate. For those at the conference that want to play Werewolf, come on out and participate at 5:30 p.m.
I want to give a shout out to the folks at the Sheraton that have been fantastic dealing with logistics, food, drink breaks, etc. Seriously, they have been fantastic. Also, I want to thank everyone who has hung around for the amazing conversations that go on each night afterwards. The plus side, I learn so many interesting things from various perspectives. The down side is that I am going to bed way too late this week ;).
We're about to get ready for the second keynote, so I'll be back shortly.
Ever had a conversation with someone who says "I don't care where we go to eat" but when you actually make a suggestion, they always say "no"? If that annoys you, then the keynote speaker might actually ring a chord with you. In "Get What You Want With What You’ve Got", humorist Christine Cashen is in the middle of describing those people that always complain about the world around them, and how nothing works for them. How will they deal with situations that require doing more with less?
Christine is describing the personalities of people based on single words. The who people, the what people, the why people, the how people. If you want to get what you want, you have to realize that each of these people has their own language and their own way of dealing with things. If you want to get what you want from these people, you have to know what they need to hear and how that works for them.
One of the things I will already say about Christine is that she is directly engaging with individuals in the audience, and engaging them. I had to laugh when she did the partner exercise with the fist, and my immediate reaction was "oh, this is so Wood Badge!" However, it was fun to see the various reactions of the people in the audience.
One of the great tools that I use often, and I've found it to be greatly helpful comes from a phrase that James Bach said in an interview a couple of years back... "I have strong convictions, but they are lightly held." What does he mean by that? It means that he genuinely believes or has come to certain conclusions, and he will battle and fight for them, but if new information comes to light, we can modify our understanding and see things differently. That's an extremely valuable tool.
With humor, a bit of silliness, and a lot of heart, this was honestly a way better talk than I was expecting. By the way, for those who want a taste of what Christine is like, check this out:
-----
I have wanted to participate in Henrik Andersson's talk "Now, what's Your Plan?" several times, but I have either been speaking or otherwise engaged each time he's given it. When I saw he was doing it as a two hour block, I knew I had to get in on it. Thus, much of today will be focused on Henrik's presentation (and thanks to Leonidas Hepas for helping facilitate the session). I think this will be fun :).
First thing we all did was write down a working definition of "context". Simple right?
Hmmm... maybe not ;). Context is deceptively easy to define, but it's not as easy to come to an agreement on what it actually means. Henrik of course was not content to just have people consider a definition, we needed to internalize and understand it. When he pulled out the spider robots, I smiled and laughed... and then told my group that I would need to recuse myself from the exercise, since this exercise is the contents of the "What is Context?" module that is being used in the SummerQAmp curriculum. Still, it's cool to see how the groups are addressing the various challenges.
Without spoiling the exercise (some of you may want to do it later if Henrik offers this again, and I recommend you go through it if you get the chance), it's interesting to see how many different scenarios and contexts can be created for what is essentially the same device.
As each team has gone through each round, changes in the requirements and the mission are introduced. Each change requires a rethinking and a re-evaluation of what is needed, and what is appropriate. This is where "context" begins to be internalized, an the ability to pivot and make changes to our testing approach based on new information. It's stressful, it's maddening, and it really shows that not only is context a consideration for different projects, but it is also appropriate to consider there can be different contexts for the project you are actually working on, and the ability to change one's mind, ideas and goals mid-stream is a valuable skill to have.
What was interesting was to come back and see, based on this experience, whether or not the team's ideas of context had changed. We can look at context as to the way we test. We can look at context as to the use of the product. We can look at context base on the people that will use it. Several of the teams had come back to their initial definitions and decided to modify them. I could be a smart aleck right now and say that this is the moment that everyone comes out and says "It depends" ;).
So... what did our instructors/facilitators use to define context? See for yourself:
------
Lunch was good, and and we are now into our afternoon Keynote. Matt Johnston from uTest is talking now about "the New Lifecycle for Successful Mobile Apps". We talk a lot about tools, processes and other details about work and what we do. Matt started the talk with discussing about Companies vs. users. Back in the day, companies provided product to users. Today, because of the open and wide availability of applications in the app store, users now drive the conversation more than ever. A key think to realize is that testing is a means to an end. It's there to "provide information to our stakeholders so that they can make good decisions" (drink!).
Mobile is just getting started. We are seeing a transition away from desktop and laptops to mobile (all sources; tablets, phones, etc.). Mobile is poised to eclipse the number of desktop and laptop machines in the next three to five years. Mobile apps are also judged muh more harshly than their desktop or web equivalents were judged at their point in the product lifecycle. The court of public opinion is what really matters. App store ratings and social media will make or break an app, and it will do so in record time today.
Much of the testing approach we have used over the years has come from an outside in perspective. Mobile is requiring that our testing priorities invert, and that we focus on the inside-out approach, especially with mobile. What the user sees and feels trumps what the product actually does, fair or not.
The tools available to mobile developers and mobile testers is expanding, and the former paucity of tools is being addressed. More and more opportunities are available to check and automate mobile apps. Analytics is growing to show us what the users of mobile devices are actually doing and see how and there they are voting with their feet (or their finger swipes, in this case ;) ).
A case study presented was for USA today, a company that supports Printed Paper, a website and 14 native mobile apps. While it's a very interesting model and great benefit to its users, it's a serious challenge to test. They can honestly say that they have more uniques and more pageviews on mobile than on the web. that means that their mobile testing strategy really matters, and they have to test not just domestically, but worldwide. The ability to adapt their initiative and efforts is critical. Even with this, they are a company that has earned regularly a 4.5 star app store rating for all of their apps.
If your head is spinning from some of that, you are not alone. Mobile isn't just a nice to have for many companies, it's now an essential component to their primary revenue streams.
-----
One of the unfortunate things that can happen with conferences is when a presenter has to drop out of a conference at the last minute. It happened to me for PNSQC 2011 because of my broken leg, and it happened to one of the presenters scheduled today. In his place, Mark Tomlinson stepped in to discuss Performance Measurements and Metrics. The first thing that he demonstrate was the fact that we can measure a lot of stuff, and we can chew through a lot of data, but what that data actually represents, and where they fit in with other values, is the real art form and the place that we really want to place our efforts.
Part of the challenge we face when we measure performance is "what do we actually think we are measuring?" When a CPU is "pegged", i.e. showing 100% utilization, can we say for sure what that represents? In previous decades, we were more sure about what that 100% meant. Today, we're not so sure. Part of the challenge is to get clear the question "What is a processor?" We don't really deal with a single CPU any longer. we have multiple cores an ach core can create child virtualization instantiations. Where does one CPU reality end and where does another begin? See, not so easy, but not impossible to get a handle on.
Disk space is another beloved source of performance metric data. parking the data that you need in the place you need it in the optimal alignment is a big deal for certain apps. the speed of access and the feel of the system response to present data can be heavily influenced by how the bits are placed in the parking lot. Breaking up the data to find spot can be tremendously expensive (this is why defragmenting drives regularly can provide such a tremendous performance boost). Different types of servers have a different way they handle I/O (Apps, DB, Cacheing, etc.).
RAM (Memory) is another much treasured and frequently coveted performance metric. Sometimes it gets little thought, but when you find yourself using a lot of it, it can really mess up your performance if you run out of it. Like disk, if you reach 100% on RAM, that's it (well, there's page file, but really, you don't want to consider that as being any real benefit. This is called a swapping condition, and yeah, it sucks).
The area that I remember doing the most significant metric gathering on would be in the Network sphere. Networking is probably the most variable performance aspect, because now we're not just dealing with items inside of one machine. What another machine on the network does can greatly affect wat my own machine's network performance is. Being able to monitor and keep track of what is happening on the network, including re-transmission, loss, throttling, etc. can be very important.
Some new metrics we are getting to be more interested in are:
What does it mean to push, or "jam" a story upstream? Jerry Welch is introducing an idea that, frankly, I never knew had a word before. His talk is dedicated to "anadromous testing", or more colloquially "upstream testing". How can we migrate testing upstream? We talk about the idea that "we should get into testing earlier". Nice to say, but how do you actually do that?!
Christine is describing the personalities of people based on single words. The who people, the what people, the why people, the how people. If you want to get what you want, you have to realize that each of these people has their own language and their own way of dealing with things. If you want to get what you want from these people, you have to know what they need to hear and how that works for them.
One of the things I will already say about Christine is that she is directly engaging with individuals in the audience, and engaging them. I had to laugh when she did the partner exercise with the fist, and my immediate reaction was "oh, this is so Wood Badge!" However, it was fun to see the various reactions of the people in the audience.
One of the great tools that I use often, and I've found it to be greatly helpful comes from a phrase that James Bach said in an interview a couple of years back... "I have strong convictions, but they are lightly held." What does he mean by that? It means that he genuinely believes or has come to certain conclusions, and he will battle and fight for them, but if new information comes to light, we can modify our understanding and see things differently. That's an extremely valuable tool.
With humor, a bit of silliness, and a lot of heart, this was honestly a way better talk than I was expecting. By the way, for those who want a taste of what Christine is like, check this out:
-----
I have wanted to participate in Henrik Andersson's talk "Now, what's Your Plan?" several times, but I have either been speaking or otherwise engaged each time he's given it. When I saw he was doing it as a two hour block, I knew I had to get in on it. Thus, much of today will be focused on Henrik's presentation (and thanks to Leonidas Hepas for helping facilitate the session). I think this will be fun :).
First thing we all did was write down a working definition of "context". Simple right?
Hmmm... maybe not ;). Context is deceptively easy to define, but it's not as easy to come to an agreement on what it actually means. Henrik of course was not content to just have people consider a definition, we needed to internalize and understand it. When he pulled out the spider robots, I smiled and laughed... and then told my group that I would need to recuse myself from the exercise, since this exercise is the contents of the "What is Context?" module that is being used in the SummerQAmp curriculum. Still, it's cool to see how the groups are addressing the various challenges.
Without spoiling the exercise (some of you may want to do it later if Henrik offers this again, and I recommend you go through it if you get the chance), it's interesting to see how many different scenarios and contexts can be created for what is essentially the same device.
As each team has gone through each round, changes in the requirements and the mission are introduced. Each change requires a rethinking and a re-evaluation of what is needed, and what is appropriate. This is where "context" begins to be internalized, an the ability to pivot and make changes to our testing approach based on new information. It's stressful, it's maddening, and it really shows that not only is context a consideration for different projects, but it is also appropriate to consider there can be different contexts for the project you are actually working on, and the ability to change one's mind, ideas and goals mid-stream is a valuable skill to have.
What was interesting was to come back and see, based on this experience, whether or not the team's ideas of context had changed. We can look at context as to the way we test. We can look at context as to the use of the product. We can look at context base on the people that will use it. Several of the teams had come back to their initial definitions and decided to modify them. I could be a smart aleck right now and say that this is the moment that everyone comes out and says "It depends" ;).
So... what did our instructors/facilitators use to define context? See for yourself:
The interesting thing is that, all of the definitions differ, but none of them contradict each each other. Here is why "best practices" are not helpful, because the context changes, not just between projects but often within a project. Hearing the different team's discuss their own experiences as to how they match up with the exercise, many teams see that context is a genuine struggle, even in groups that profess that they understand context. there is a tremendous variety in markets, needs, timelines and issues. Learning how to undertand those elements, and how to address them as they come up, will go a long way in helping to drive what you test, how you test, when you test, and what prioritization you place on what you test.
Again, a deceptively simple, yet complex issue, and a seriously fun session. Thanks to Henrik and Leo for their time and energy to make it possible.
------
Lunch was good, and and we are now into our afternoon Keynote. Matt Johnston from uTest is talking now about "the New Lifecycle for Successful Mobile Apps". We talk a lot about tools, processes and other details about work and what we do. Matt started the talk with discussing about Companies vs. users. Back in the day, companies provided product to users. Today, because of the open and wide availability of applications in the app store, users now drive the conversation more than ever. A key think to realize is that testing is a means to an end. It's there to "provide information to our stakeholders so that they can make good decisions" (drink!).
Mobile is just getting started. We are seeing a transition away from desktop and laptops to mobile (all sources; tablets, phones, etc.). Mobile is poised to eclipse the number of desktop and laptop machines in the next three to five years. Mobile apps are also judged muh more harshly than their desktop or web equivalents were judged at their point in the product lifecycle. The court of public opinion is what really matters. App store ratings and social media will make or break an app, and it will do so in record time today.
Much of the testing approach we have used over the years has come from an outside in perspective. Mobile is requiring that our testing priorities invert, and that we focus on the inside-out approach, especially with mobile. What the user sees and feels trumps what the product actually does, fair or not.
The tools available to mobile developers and mobile testers is expanding, and the former paucity of tools is being addressed. More and more opportunities are available to check and automate mobile apps. Analytics is growing to show us what the users of mobile devices are actually doing and see how and there they are voting with their feet (or their finger swipes, in this case ;) ).
A case study presented was for USA today, a company that supports Printed Paper, a website and 14 native mobile apps. While it's a very interesting model and great benefit to its users, it's a serious challenge to test. They can honestly say that they have more uniques and more pageviews on mobile than on the web. that means that their mobile testing strategy really matters, and they have to test not just domestically, but worldwide. The ability to adapt their initiative and efforts is critical. Even with this, they are a company that has earned regularly a 4.5 star app store rating for all of their apps.
If your head is spinning from some of that, you are not alone. Mobile isn't just a nice to have for many companies, it's now an essential component to their primary revenue streams.
-----
One of the unfortunate things that can happen with conferences is when a presenter has to drop out of a conference at the last minute. It happened to me for PNSQC 2011 because of my broken leg, and it happened to one of the presenters scheduled today. In his place, Mark Tomlinson stepped in to discuss Performance Measurements and Metrics. The first thing that he demonstrate was the fact that we can measure a lot of stuff, and we can chew through a lot of data, but what that data actually represents, and where they fit in with other values, is the real art form and the place that we really want to place our efforts.
Part of the challenge we face when we measure performance is "what do we actually think we are measuring?" When a CPU is "pegged", i.e. showing 100% utilization, can we say for sure what that represents? In previous decades, we were more sure about what that 100% meant. Today, we're not so sure. Part of the challenge is to get clear the question "What is a processor?" We don't really deal with a single CPU any longer. we have multiple cores an ach core can create child virtualization instantiations. Where does one CPU reality end and where does another begin? See, not so easy, but not impossible to get a handle on.
Disk space is another beloved source of performance metric data. parking the data that you need in the place you need it in the optimal alignment is a big deal for certain apps. the speed of access and the feel of the system response to present data can be heavily influenced by how the bits are placed in the parking lot. Breaking up the data to find spot can be tremendously expensive (this is why defragmenting drives regularly can provide such a tremendous performance boost). Different types of servers have a different way they handle I/O (Apps, DB, Cacheing, etc.).
RAM (Memory) is another much treasured and frequently coveted performance metric. Sometimes it gets little thought, but when you find yourself using a lot of it, it can really mess up your performance if you run out of it. Like disk, if you reach 100% on RAM, that's it (well, there's page file, but really, you don't want to consider that as being any real benefit. This is called a swapping condition, and yeah, it sucks).
The area that I remember doing the most significant metric gathering on would be in the Network sphere. Networking is probably the most variable performance aspect, because now we're not just dealing with items inside of one machine. What another machine on the network does can greatly affect wat my own machine's network performance is. Being able to monitor and keep track of what is happening on the network, including re-transmission, loss, throttling, etc. can be very important.
Some new metrics we are getting to be more interested in are:
- Battery Power (for mobile)
- Watts/hr (efficiency of power consumption in a data center, i.e. "green power")
- Cooling in a data center
- Cloud metrics (spun up computer unit costs / hour)
- Cloud storage bytes (Dropbox, Cloud Drive, etc.)
Other measures that are being seen as interesting to a performance evaluation of systems are:
Correlative graphing is also used to help us see what is going on with one or more measurements. A knee in the curve may be interesting, but wouldn't it be more interesting to see what other values might be contributing to it?
This fits into the first talk that Mark gave yesterday, and here's where the value of the first talk becomes very apparent. Much of the data we collect, if we just look at the values by themselves, don't really tell us much. Combining values and measuring them together gives us a clearer story of what is happening. Numbers are cool, but again, testers need to provide information that can drive decisions (drink!). Putting our graphs together in a meaningful way will greatly help with that process.
-----
- Time (end user response time, service response time, transaction response time)
- Usage/Load (number of connections, number of active threads, number of users, etc.)
- Multi-threading (number of threads, maximum threads, thread state, roles, time to get threads)
- Queuing (logic, number of requests, processing time)
- Asynchronous Transfer (disparate start/end, total events, latency)
At some point with all of this, you end up getting graphical representations of what you are seeing, and the most bandied about graph is the "knee in the curve". Everyone wants to know where the knee in the curve happens. Regardless of the value, the knee in the curve is the one hat people care about (second most important is where things go completely haywire and we max out, but the knee is the real interesting value.. by some definition of interesting ;) ).
Correlative graphing is also used to help us see what is going on with one or more measurements. A knee in the curve may be interesting, but wouldn't it be more interesting to see what other values might be contributing to it?
This fits into the first talk that Mark gave yesterday, and here's where the value of the first talk becomes very apparent. Much of the data we collect, if we just look at the values by themselves, don't really tell us much. Combining values and measuring them together gives us a clearer story of what is happening. Numbers are cool, but again, testers need to provide information that can drive decisions (drink!). Putting our graphs together in a meaningful way will greatly help with that process.
-----
What does it mean to push, or "jam" a story upstream? Jerry Welch is introducing an idea that, frankly, I never knew had a word before. His talk is dedicated to "anadromous testing", or more colloquially "upstream testing". How can we migrate testing upstream? We talk about the idea that "we should get into testing earlier". Nice to say, but how do you actually do that?!
Let's think about the SDLC .We're all familiar with that, right? Sure. Have you heard of the STLC? The Software Testing Life Cycle? The idea is that just as there is a software development lifecycle, there is also a test lifecycle that works similar to, and in many ways synchronously with the SDLC.
One of the key ways that a test team can make movement upstream is to make sure that their team understands what they are responsible for and what they deliver. Make an emphasis on training, and plan to have your development team interact with your test team, and do so in a fun way (make play dates with developers and testers. Sounds weird, but I like the spirit behind the idea a lot :) ).
Testers have to make the commitment to transition over to being an extension of the development process. testers need to learn more about what is necessary to get a product covered. If a company hires full stack developers, likewise, full stack testers might be a valuable goal. While that doesn't mean that the testers need to become full stack developers themselves, they need to understand all of the moving parts that actually come into play. Without that knowledge, testing is much less effective. It's not going to happen overnight, but geting that skill level up for the testers will help them be able to get farther upstream with the testing process.
Along with learning about what to test, make a commitment to focus on estimating what you can really get done in a given sprint or cycle. Stop saying that you cannot get your testing done in a given sprint. Instead, get a real handle on what you can get done in a given test sprint.
Every organization's STLC is going to be a bit different. It will be based on the talent the team has and on the talent that they can develop. Just like a salmon swimming upstream, your team has to develop strength. More to the point, they need to be able to show what they know. Effective Bragging is a point that is emphasized, and if you have a wiki, use it to show what you have learned, what milestones you have met, etc. Another aspect is Battle Elegance, which addresses such areas as people vs. projects, customers or team members, and developing goals to keep the teting team focused and moving forward (or swimming upstream).
I'm not sure I'm totally on board with this idea, but I admire its goals, and I think it's one of the few ways I've seen articulated to actually get thinking about this process. We all want to move upstream, or more to the point, we wish we were involved earlier. The metaphor of "swimming upstream" works for me. It's muscular, demanding, and exhausting, but you will get stronger if you keep swimming. Of course, I'm not so fond of where the metaphor ends. Think about it what happens to salmon when they finally get to the spawning grounds. They reproduce, then they become bird food. I like the reproduce idea. The bird food idea, not so much ;). I guess our real challenge is finding out how we can sync up so that we don't die at the end of the journey.
-----
The last "official" activity for Wednesday (and I use that term loosely ;) ) is a group of testers getting together to play a game called "Werewolf". Think of this as "networking with a twist. The participants are all at a large perimeter of tables, and the goal is to determine who are villagers and who is the werewolf. The point of this game is to observe others around the table, and based on conversations, clues and details, see how quickly the werewolf can be identified, and also not make false identifications. this has been fun in the sense that everyone is both laughing and trying to see i they can get it right the first time without false identification. After the round of sessions today, a little levity is going a long way.
Subscribe to:
Posts (Atom)