Showing posts with label Test Retreat. Show all posts
Showing posts with label Test Retreat. Show all posts

Saturday, August 1, 2015

Testing the IoT - Live from #TestRetreat

Ah yes, The Internet of Things! that weird cross section of devices and services that are super specific or are focused on areas that are still being defined. We hear about thermostats that learn, or devices that report blood sugar levels for people with diabetes.

The Internet of things are going to introduce all sorts of devices and needs for testing that come in all different shapes or sizes. Scott Allman brought in a bunch of interesting devices that are tiny and can be deployed for simple purposes. Imagine a device the size of a nine volt battery that can act as a full stack web server.

One of the fun aspects about these tiny devices is that they often have very primitive interfaces, or raw Linux style interfaces. This encourages people to play around with the devices and interact with the device. Additionally, with these tiny devices, it's possible to create multiple systems that can be deployed very cheaply, so we can set up little networks and sandboxes for very little cost.

Some of the challenges we face with these devices are in the realm of security, as well as the way that the system is powered (battery vs. dedicated power, using protocols like low energy bluetooth, etc.). there are also devices that use accelerometers, so part of testing includes subjecting the device to motion. Additional tests we might care about would be related to power, heat, durability, responsiveness, performance, usability, and accessibility. Many of the test approaches and themes that we use for mobile testing will be able to be applied to the Internet of Things as well.

What I find interesting is that these could be very low priced devices to implement and test with performance and security tools. I could see being able to create a small network of devices and implement jMeter and Kali Linux to poke around with. I joked that it would be fun to be able to have a belt where each section is a separate server,  and HSA and load balancing applications could be experimented with.

I noticed that in this talk, I kept thinking about using these devices for servers and applications I'm already familiar with. There's lots of applications I haven't even remotely considered, and I know that with time as I get more familiar with uses, I can start applying ideas for these devices and see what other avenues of exploration I can discover.

Teaching Testing to Programmers - Live from #TestRetreat

We hear a lot about teaching software testers how to program, but do we hear the converse? Why don't we ever hear about software testers teaching programmers how to test? Jesse Alford thinks this deserves to be a two way street and that there's a lot of opportunity to have engineers pick up exploratory testing skills to support the team in growing and learning.

One thing that should be mentioned is that Jessie works with Pivotal Labs, and Pivotal is a pretty awesome team with a lot of high functioning people on their team, and also have a fairly good number of engineers who are well versed in software testing methodologies.

There are a number of games that we use as testers that are well known. We use The Dice Game, we use The Pyramid Game, we use James Lyndsey's Black Box Puzzles, and we use simple games like "The Pen Test" that can be infuriating, but can teach us how we model problems and apply testing principles to those games.

A phrase that Scott Allman shared, and I like a great deal, is that  "our job as testers is to discover laws that have been smuggled into the model universe that the programmers didn't know about". Having the opportunity to share some of these exercises allows us to share this mindset with programmers in an area that is separate from their core experience.

One suggestion that Jessie encourages is to have two people be part of the game, especially when the game in question has a "find the rule" element to it (Dice Game, Zendo, Pen Test, etc.). One person runs the game, i.e. they know the rule and the aspect that makes the game relevant. Another participant is there to help coach the person playing the game, but they don't know what the game rule is. they are familiar with the game itself, but they don't specifically know what the particular rule for that game is. that way, the coach is also operating from a level of not knowing the specific answer, but still being able to coach the participant in how they might go about solving the problem. One point that Jesse made that makes the games themselves at times less effective is the "all knowing" game master. The coaching the game master provides turns into maddening hints, coming from the person who knows what the game is. By having a third person who doesn't know the rule, the feedback on trying to solve the problem is less biased and less of that "knowing glance".

Another way that I personally would encourage having a programmer get involved with testing is to have them participate in a few Weekend Testing sessions. These are great training grounds for session based testing, as well as to explore some interesting avenues for software testing via individual topics. The cool thing about doing these sessions is that a lot of them are focused on black box skills, and they allow everyone to discuss what they are working on and learning in the process.

It's fun to take the chance to learn from each other, and testing games are an easy way to get programmers to see what it is we do and how we do it.


Bringing Energy Back to Testing - Live from #TestRetreat

Picture the scenario. A tester has been in the game for a number of years. They know the details, they know the product, they've done countless stories, and at some point, the testing becomes rote and paint by numbers. The thrill is gone. the excitement level has left the building. Do you see yourself in this scenario? If not, fabulous, this talk is not for you ;).

If, however, you have been in this situation, or currently see yourself in this situation, then Phil McNeely's session on "Bringing Energy Back to Testing" is for you. I know how this feels. I've been there and done that. At a certain point, I lost the joy and fun that testing used to provide, and at some point, I was just going through the motions. It wasn't intentional on my part, I didn't intend to go on autopilot, but I did find that there have been times where I just did what I had to do, and often what I really didn't want to do. Often the reasons came down to doing the same thing over and over. Sometimes people just burn out. For many, their  passion is somewhere else, like snowboarding, or knitting, or writing a novel. Fort those people, it helps a lot to accept that their true passions lie elsewhere and to encourage them to invest in those areas to the point they feel they are getting satisfaction there, and the follow on effect is that they can focus on what they do at work.

In my case, I often found myself overcommitted to too many good things. It's not that I was necessarily burned out on my work, but that I was committed to my day job, and to writing, and to teaching, and to community engagement. In short, I found I was spreading myself too thin in too many areas. At those times, it became apparent that I need to "give myself a haircut" across the board. When a tester who was at one time productive seems to be less focused or engaged, it might be worth making a lunch date and just getting to see what is going on in that tester's life at that moment, both inside and outside of work. It's possible you might discover that they have recently become the PTA president at their school, or they have taken on an important but time consuming position at their church, or there may be a family situation due to an illness or situation in the family that is drawing upon their energy. I know when I have too many things happening in my life, every area suffers, and yes, that includes work. By realizing that energy commitments have changed, we can help that person (or ourselves) consider what options we have to make modifications.

I wrote about this a few days ago over at the ITKE Uncharted Waters blog, but something I am using to help me stave off this challenge is using an Objective Journal. By having me consider what I am working on, and questioning it on a regular basis, I can keep myself engaged with the problem, rather than waiting for something to complete and going of and working on something else that's not on target or, sometimes, not even remotely productive. The trick for me with the Objective Journal is that it allows me to see small, everyday wins. By seeing those wins, I stay motivated and excited.

Ultimately, regardless of how engaged or not engaged we are, at the end of the day we need to realize that WE are the ones that need to develop our motivation. We can encourage and offer to help others achieve motivation, but externally motivated people tend to not stay motivated for too long. Internally motivated people can stoke that fire indefinitely, so work to encourage that spark in others, but more importantly, help them develop and maintain that spark in themselves.

What is the Career Path for a Tester? - Live from #TestRetreat

Carl Shaulis asked a simple question, or what seemed to be a simple question... what are some of the skills needed at various points of a career for a software tester?

There are many variations of software testing and approaches to software testing, but for many of us, it seems that there is a specific path. The first round is what we as a group called "the muscle", i.e. classic manual software testing. The starting point for this is, to borrow from Jon Bach, a need for curiosity. Jon has said that he can teach people technical stuff as needed, but he can't teach people how to be curious (I will come back and get the quote and post where he says this, but for now, forgive the live blogger and lack of immediate attribution ;) ).

Another aspect we discussed is that the skill levels are not specifically better at higher job levels. Many of the skills of a Level 1 tester are still applied at higher levels. they don't necessarily have specialized knowledge, but they do have experience using it over several years. What is expected for any level is the ability to evaluate a product, to look at the product with an eye to look at a workflow and determine if something is out of place or not working in the way that people expect. Critical thinking skills are valuable at any level of the job. The differentiators tend to not be the skills, but the level of influence within the organization the individual has. Junior testers and senior testers are often differentiated not so much by their skill level, but with experience and leadership, as well as overall influence.

One obvious question to ask is "does a tester need to learn how to code and make coding part of their job if they want to advance?" My answer is that, if you want to be a toolsmith and work on test tooling, then yes, programming is essential. If you don't aim to be a toolsmith, or if you are not interested in focusing on automation or tooling, then programming may not be essential. Having said that, I think that many more questions are capable of being asked and evaluated when a tester can look at the underlying code and understand what is happening, even if just in a general sense.

Much of these discussions come into play so that they can have a shorthand when it comes to job titles, compensation, and ability to allocate people into an organization. I've primarily worked in smaller companies the past fifteen years, but during the first ten years of my career, I worked with Cisco Systems, which went from a smallish 300 person company when I joined it in 1991 to a 50,000 plus person company when I left in 2001. Early in the life of Cisco, job titles were less important than they were later. When dealing with a company that is much larger, titles and the shapes of the "cogs" starts to matter. Within smaller teams, generalists and people able to cover lots of different areas are much more important, and there's a fluidity in the work that we do. Career path is less relevant in a smaller company, and the reward aspects are different. In many ways, in a smaller company, you are not rewarded with titles or advancement, you are rewarded with influence (and in some cases, with money or equity).

As a closer, it was suggested that we check out the Kitchen Soap article "On Being a Senior Engineer".  There is a lot of meat in this one so I might do a follow-on post just on this article :). ETA: Coming back to this two and a half years later, I was asked if I would consider adding "7 Reasons You Can't Get a Junior Web Developer Job" as a follow-on read. Granted, a little outside of the software testing space, but many of the same issues also fit this discussion, so yes, considered and added :).

How Can We Interview Testers Better? - Live from #TestRetreat

One of the challenges that anyone who has been involved with hiring software testers can tell you is that it can be maddening to interview testers. We can find people, but getting the right people is often a struggle. We have all had the experience of reading a resume and seeing what looks to be very promising history and experience, only to have them in an interview and have to ferret out what they actually understand or do not understand. is there a way that we can do this better? Dwayne Green led a group of us to discuss how we approach these interviews and how we can improve the process.

Many of us have worked through resumes and had to make phone screens or initial interviews, and a common phrase that came up was to "audition" the candidate. There are several approaches to auditions that we can use. Some people will use a sample program and walk through how to test it. Some people like to use their own company's product as part of the audition, and to see how the candidate tests the product, or if they can find problems we already know about.

A discussion I recently had with our VP of Engineering at my company was the way that we expect people to work. Most of the programmers that write code have certain things open on their desk at all times: their IDE or programming environment, a browser with a google tab open and Stack Overflow or some other reference sites. the point being made is that, when we work, we tend to have these tools open to help us get to the things we need. When we interview, we deprive these candidates of that ability, and we expect them to "code" on a whiteboard. One of the ways that was suggested that we could change this would be to ask in advance what they like to use to work, and to feel free to bring that in with them. When we put the audition challenge in front of them, we also say "use the tools you would use". This has tended to give a more representative view as to how those programmers actually code. We discussed the idea that people should do the same thing when it comes to software testing. Let a candidate come in and use the tools they already understand.

One of the questions that we asked was "how could we make these interview approaches more real, but avoid a situation where we are making people "work for free" as part of the interview". There is always a danger that putting people into a simulation or scenario with the company's product runs the risk of replicating real work without compensation. One approach could be to use a virtual machine with a version of a release and a story that has already been worked, and have a list of issues that have already been discovered. Do we want to have them run an abbreviated test session and see what they discover from our list? Do we want to have them sit with our team members and pair for a particular period of time? In my view, I think we need to make clear that we are going over material that has already been covered, and that we are using it as an evaluation criteria and not as a way to get "free testing" out of someone.

Some people like to use games or use sample programs to do these experiments. The issue there is that some people are not good at games, or are not good at particular applications. Does that mean they are a bad tester? Probably not, but it does mean we need to take into consideration multiple avenues. They might not be good at one area, but be great in others. If we find that they don't do well in multiple areas, that might well give insights as to how well they will perform or not perform in our work environment.

We all discussed some of the worst experiences we had with interviews, and I shared that would often be asked some strange questions related to math problems or other specific questions that seemed to have no bearing on what we would be testing. Over time, I came to realize that these questions served one purpose: does this person think like I do? To be fair, it's been years since I've had to deal with those types of questions, but I know that some people still do, and if I can make one plea, it's to say "knock it off!" ;).


On Being The "Least Talented" Person in the Room - Live from #TestRetreat

Good morning from Grand Rapids, Michigan. I'm still functioning on West Coast time, but everyone else around me thinks it's the middle/late morning, so I need to kick in and get with the groove of TestRetreat. For those not familiar, TestRetreat has been an ongoing event the past few years hosted by Matt Heusser and is an un-conference event with a broad list of ideas and talks being presented by people who have interests in given areas.

A piece of housekeeping this time around. In the past, I have Live Blogged events in a day long format and have updated each page during the day. Due to the way that CAST is going to be run this year, and with the way that I will be more facilitating rather than just attending sessions, I won't be able to do the live blog the same way. Instead of doing a running log, I'll be making individual posts for each session or event. Downside here is that there will be multiple posts in a given day. Let me know which way you find most useful :).

Out first session is being hosted by Ajay Balamurugadas, and he's asking about the interesting dichotomy in that he at times feels like he is the least talented person in the room, yet he is the one that gets invited to speak at and participate in conferences. Why is that?

I've personally struggled with this attitude myself, and in part, I think the title is a little misleading, but the overall perception is an important one. By starting with the idea that we are the least talented person in the room, we are stating that we feel that we have a lot we can learn, and we are willing to put in the time to learn it. I would be hard pressed to consider Ajay the least talented person in any room, but I would say that that sense of humility and desire to learn as much as possible drives him and gives him the motivation to keep learning and growing. Adding to this is the fact that when you do create a reputation as a person who is actively engaged in the broader community, you can be held to a standard that is unfair to where you actually are. As I've talked to people over the years, I've likewise seen that they consider me to be a "thought leader" or "testing guru", and at times, I've considered their expectations to be unfounded. It's not that I have any super level of expertise, but the fact that I've written about or talked about the problem in any capacity tends to make people think I know or understand more than I do.

I'm not sure there is an easy way to deal with this other than to acknowledge that there is a reputation that being actively engaged gives someone. When we say "just because I write and talk about these topics, don't think that I am somehow superior to those who don't write or speak on topics" (fact is, some people who are mega performers do not engage in the broader community, and that's totally OK).  I personally talk to my team with the idea that I am allowed one completely stupid question of each member  of my team per day, and that they should accept it. To date, I've only had a couple of instances where people actually said "wow, you should know this". Many times, the answer I get is "no, that's not a dumb question. We don't really describe or define that area very well." the one thing I think is important is that the onus is on us to communicate both what we understand and what we don't understand. Also, it's important to get a gauge as to the expectations of what people want to see from us, and that we are meeting those expectations. If we are not, we should determine what their expectation actually are.

Tuesday, February 25, 2014

Test Retreat 2014: Why I'm Going, and Why You Should, Too :)

First, I need to paraphrase this with something you will be seeing a lot from me in the coming weeks:


CAST 2014 will be held in New York City August 11-13, 2014.


I will be giving a talk along with Harrison Lovell. Details on this will follow, but not until it gets officially posted.


I want to see as many of you as possible come attend CAST 2014, because I feel it is one of the best, if not THE best, software testing conferences a software testing practitioner can attend and come away with real value for their time and investment. As I said previously, you will see me doing a lot more commentary and promotion for CAST going forward.


Having said all that, I want to talk about something that is a preamble to CAST, and looks to be turning into an annual event that I support and want to see thrive. That event is called "Test Retreat".


Test Retreat is rapidly becoming one of my favorite events to attend each year. This year it will be held Saturday, August 9, 2014 from 8:30 AM to 4:30 PM (EDT). I attended the inaugural event in 2012, when it was introduced at CAST 2012 in San Jose, CA and then in Madison, WI during CAST 2013. 

What makes Test Retreat worthwhile? 

Many other events allow a select few to present on ideas that have to be highly structured. The Open Conference option that Test Retreat offers allows me, and others, to present ideas that might be very preliminary and embryonic. Through the event, these preliminary ideas often develop into calls for action, with input from many other participants, that are significantly better than anything I would have proposed on my own. It's this rich level of interaction, conferring with peers, and all willing to work together to develop "better ideas" that make this format a success. Several of my better talks (Let's Stop Faking It, Balancing ATDD, GUI Automation and Exploratory Testing, and others) that I have given at Meet-Ups, conferences and have written up as published articles and papers have had their genesis in Test Retreat.


For those wondering if it makes sense, or if it's worth it to attend a Saturday event (yes, I know Saturdays  are precious), I say "yes"! So far, it has proven to be every bit worth it these past two years. I've already signed up for year three. 

Will I see you there? Will we, perhaps, come up with better ideas together than we would separately? I hope you will come attend and find out!