Walking in Your Footsteps: a #30DaysOfTesting Testability Entry

OK, this gets me to a third of the way through the challenge. Fortunately, this is a relatively easy entry as I get to talk about other people doing great stuff and I'm glad to oblige. With that, let's keep this 30 Days of Testability Party going, shall we?

Follow and share three people on LinkedIn/ Twitter who regularly talk about testability.

Sweet, I get to brag up some people. Always fun.

1. Alan Richardson (Alan's LinkedIn, Twitter: @EvilTester)

If you read my previous entry about sharing a video, this name should be familiar. He also has a pretty extensive blog and a lot of learning resources via his own books and via YouTube.

2. Maria Kedemo  (Maria's LinkedIn,   Twitter: @mariakedemo)

Maria covers a lot of ground and has excellent content that she frequently shares. As to the topic at hand, check out her "Testability Awakens" article she wrote for Testing Trapeze.

3 Jim Hazen (Jim's LinkedIn,  Twitter: @JimHazen4u)

Jim has a book called "Before The Code: First Steps to Automation in Testing" that's worth a look. I highly suggest you give it a read.

That Voodoo That You Do: a #30DaysOfTestingTestability Entry

Hopefully, you are having some fun with me as I keep this going. This past week was a bit much leading into my daughter performing High School Musical (her last high school drama production) so there was a lot of activity around that so I'm going to be a bit aggressive with the remaining days of March so I can rightly finish the 30 Days of Testability challenge (or at least get as close to finishing on time as I possibly can).

With that...

In your next story kick off session ask, ‘how are we going to test this?’ Share the test ideas and techniques that are suggested.

This was a fun experiment in that I did get to do this for a specific feature that was getting a Responsive update. There were a variety of features that hadn't been looked at as thoroughly in recent efforts (even before the responsive updates) so I got to ask this question quite a bit. The good news is my Engineering team is totally cool with me asking these kinds of questions. We did some brainstorming and I made some suggestions of ways we could better examine how the application is working.

- some ideas revolve around adding calls to the API so that I can query and set values without having to fire up a browser and look at a bunch of things on the screen.

- some ideas revolve around actual configuration enhancerments of an application so that I can tweak various things while I'm setting up a test environment.

- some ideas revolver around making sure that elements are named well (whenever possible) and that duplication is minimal. Additionally with that, do anything we can to avoid dynamically generating elements. If that's not possible, let's at least make sure that we have a mechanism in place to easily discover those values and utilize them.

The biggest benefit I find with these conversations is that the development team and I can get on the same page much more quickly. In addition, it also allows me to see a bit of writing under the board and better understand the app a layer at a time. That's a good outcome for everyone, if you ask me :).

A Touch of Evil: a #30DaysOfTesting Testability Entry

Do any of you have any idea how long I have waited to be able to use Judas Priest's "Painkiller" ballad in a blog title? Way too long, that's how long ;). With much thanks to Alan Richardson (i.e @EvilTester) for so very often having just what I need right when I need it.


Share a video about testability with your testing peers at your company.

What is Testability vs Automatability? How to improve your Software Testing.
there are some very cool concepts in this video and hey, there's a supporting blog post to go with it. Yay Alan!!!

So what are the takeaways here?

Just because you can automate something doesn't necessarily means it's testable. Automatability means that something else can interact with it. Just because that is capable to be done doesn't necessarily mean that it's going to be a good experience for a user.

Alan's TL;DR drives this point home very effectively: "Testability is for humans. Automatability (Automatizability) is for applications."

Testability can add a number of options to a program and can help to make a system more enjoyable for a user. It may help with making an application more automatizable (great word, Alan, I'm adding that one to my lexicon) but it doesn't guarantee that it will.

In any event, the video and accompanying blog post are excellent. Check them out :).

Is it in the Playbook?: a #30DaysOfTesting Testability Entry

Here's another entry for the "30 Days of Testing" Testability challenge. Let's keep this ball rolling.

Find out if your application has an operational manual or runbook? How can you test this?

Put simply, yes, it does. More to the point, I've been one of a few people that has actually updated this documentation with new feature information as well as release notes. To that end, I absolutely have familiarity with what goes in it, how to put content in there and how to look for details about how our software works and what options are available to me.

Having said that, our software has a fairly high level of complexity and thus our documentation needs to reflect that. While I will not say that I look at every single help page, I do look at it from a macro level as I'm the one who builds those help files.

What I do try to do is look at the top level details and traverse them the way I imagine that an everyday user of our product might. What could this tell me?

First and foremost, it tells me what we are telling our customers as to how we should be doing something. That's an important avenue to look at and consider while I am testing because that's what we intend our customers to be focusing on doing to complete their workflows. How often do I do this and determine that I have a different or better way of performing a task? Surprisingly often, as I have come to see. Of course, part of that is the fact that I know the administrator interface and many times using that is *way* more effective than performing the in-app methods. However, since our customers do not have access to that level of interaction (well, most don't) I need to remind myself to either go with the published methods or make a case that, maybe, our backdoor methods probably need to be considered as a way to let our customers do things, too.

To borrow from Red and Overly Sarcastic Productions... "So... yeah!"

Frame By Frame: A #30DaysOfTesting Testability Entry

I'm going to have to get aggressive if I'm going to make the end of month deadline. therefore, I'm going to try to be more specific with these posts and discuss them in a way that will allow me to be more targeted and to post more entries. It's becoming a matter of pride now! Anyway, more "Thirty Days of Testability".

Explore the output of your applications analytics, how can they help guide your testing?

I'll take a two-pronged approach with this. I'll talk a bit aboiut my company's product and a bit about what I use on my own TESTHEAD blog.

With my company, we have two applications we use and monitor for Analytics. One is more for overall HTTP traffic and demographics (such as what pages get hit the most, what browsers are used the most and what time of days do people most use the application, as well as how many parallel connections are we maintaining at any given time. that helps me define what I should prioritize in my testing as well as what level of load and other parameters I might want to consider.

The second tool we use is based on feature analytics, as in "what features in our product are our customers actually using?" We've had a few times in the past where we had a request to get something implemented and that implementation was everything to getting the deal. From there, it often meant we would maintain and keep a feature running which was important for a small set of users but meant a big part of our business. Sometimes though, we would discover that an organization would demand something, later for us to discover that their adoption rate of that feature was low to non-existent. That often meant there would be follow-up discussions. Those discussions would help us to decide if the issue was worth us keeping the feature. Perhaps we weren't making a compelling enough story for why it would be important for the organization wanting to use it. Alternately, we also often decided that the feature had such low adoption that turning it off would have little overall effect on the use of the product. We would then gracefully deprecate that feature.

In short, getting familiar with your product's analytics can tell you a lot about what is being used but also what isn't.

I'm Gonna' Get Close To You - #30DaysOfTesting Testability

Since I don't want to put too many posts in one place, I'll use my catch-up time to mentions some other things going on in the TESTHEAD world while I do this. I don't want to be too gratuitous and self-promoting. Also, thanks to Queensryche to have a song all about monitoring to save my song title titles (say that ten times fast ;) ). OK, their version is about stalking but why quibble ;).

First off, there's a new episode of The Testing Show up over at Qualitest and on Apple Podcasts. This month Matt and I are talking with Lisa Crispin of mabl and Jessica Ingrassellino of SalesForce.org about Advanced Agile and DevOps and how those things are both similar and different. I think we did a good interview this month. Well, I almost always think we do good interviews but I especially enjoyed doing this one so I hope you will go and give it a listen.

Anyway, on with the "Thirty Days of Testability". As you might notice, today is the fifteenth. This is entry five.  Catchup in full effect. You have been warned ;).

What monitoring system is used for your application? Do the alerts it has configured reflect what you test?

Again, we are going to be considering Socialtext as it is currently implemented as a standalone product because that's all I can really talk about. Before this week, I had a partial understanding of what we actually do for this and some holes in that knowledge. I'm fortunate in that I have a couple of great Ops people who are associated with our team. I should also mention that Socialtext can be deployed in two manners. First is a hosted SAAS option and second is a local installation. We leave the monitoring of the local installations to the small percentage of our customers who prefer to do that. The majority of our customers utilize our SAAS option and therefore we host their servers. To that end, we use the following (I'm pretty sure I'm not spilling any trade secrets here, so this should be OK. If this post changes dramatically between when I post it and tomorrow, well, I'll have learned differently ;) ). Anyway:

for Monitoring the systems  (CPU, Disk space, https, http) we use a tool called Nagios.

For monitoring site uptime, https lookup, and time to respond, we use an external app called Alertra.

In addition to those two tools, we also have a variety of hand-rolled scripts that allow us to scan all of our servers looking for specific aspects of Socialtext instances and services to see if there are any issues that need attention. Examples here are things like the IP address, the hostname as viewed by the public and that it is accessible, that we are running certain key services (search, replication, cron, mail, ntp, our scheduler, our search implementation, what version that particular server is running, etc.).

The second part of the question deserves a legitimate answer and that is "Yes" and  "No". Yes, in that some of the alerts map to what we test but "no" in that there's a lot of areas we don't actively test as consistently as we should. The chat with our ops team was definitely enlightening and has given me some ideas of what I can do to improve on that front. What are those things? Well, other than verifying that we are actively doing things that affect and trigger those alerts, I will have to ask that you respect the fact that I am now veering into trade secret territory and I kinda' like the idea of keeping my job ;). 

Who You Gonna' Call? #30DaysOfTesting Testability

Wow, time flies when you are doing actual work and you are trying to get your talk and workshop done and rehearsed for STPCon (seriously if you are coming up to STPCon in Northern California first week of April, come join my talk and workshop or if I'm not your cup of tea at least say "hi" ;) ).

Anyway, on with the "Thirty Days of Testability".  As you might notice, today is the fifteenth. This is entry four. Yeah, I'm going to be playing catchup. You have been warned ;).


Do you know what your top three customer impacting issues are? How could you find out?

This is surprisingly easy for my immediate team. It's less so for my extended team. I have no idea who is coming in at what point here, so I'll say this for those who are new.


Currently, I work for Socialtext. Socialtext used to be a standalone product and for a number of customers, it still is. However, back in 2012, Socialtext was acquired by a company called PeopleFluent. PeopleFluent specializes in HR Tools as their name might indicate. PeopleFluent is also a much larger company by comparison (Socialtext as a standalone group is all of ten people). Last year, PeopleFluent was acquired by Learning Technology Group (LTG) located in the UK and with offices all over the world. Thus, when I have to think about my top three customer impacts, I have to ask clarifying questions. Are we talking about Socialtext? PeopleFluent? LTG? Interestingly enough, since Socialtext is the base platform that many of the PeopleFluent and LTG products run on and interact with, it's entirely possible that Socialtext will not even enter into the top issues of anyone outside of Socialtext and at other times a Socialtext issue can be the number 1 issue for everyone. Still with me? Excellent :).

So to keep things simple, I'll focus on Socialtext standalone and how I would determine what the biggest issues are for us. The simple answer is I can reach out to our secret agent in the field... ok, it's nowhere near that cool. We don't really have a secret agent but we do have a great customer engagement engineer and frankly, a lot of the time that's just as good. I can count on one hand the number of times when I have announced an update on our staging server (read: our personal production server) and not heard a reply from this person of "hey, what's new on staging?" They make it their responsibility to be clear and up to date with every single feature and every option available in the product. They also make a lot of sample accounts and customizations to our product to push the edges of what the product can actually do. If there is any question as to what is a pain point or an issue with a customer, any customer, they are my first point of contact. Sure, we have a CRM and a bug database but the majority of the time, if I really want to see what is happening and what's really important, I know who I am going to call... or bring up in a chat message. I mean come on, this is 2019 after all ;).

Sponsoring a "New" Book? #30DaysOfTesting Testability

Observations for when you start looking at various book titles related to niche topics in software testing. First is that they can be hard to find. Second, they can be mighty pricy. What is the cost when it comes to learning? True, but there are a variety of factors one has to consider, such as, "OK, I really don't have close to $100 to drop for a specialty book at this exact moment."

Having said that, occasionally one finds interesting things in interesting places and sometimes those things are in the process of percolating. It is in this guide that I have chosen to make my mark for "Day Three" of this challenge... yes, I'm writing about it on "Day Seven", let's not quibble here.


Begin reading a book related to testability and share your learnings by Day 30.

To this end, I've decided to help fund a book in the process of being written (or at least to be written for Leanpub distribution). Said book?

Team Guide to Software Testability

by Ash Winter and Rob Meaney

Learn practical insights on how testability can help bring team members together to observe, understand and control customer needs, ensuring fitness and predictability of deliveries.

Now I will have to say up front that I will probably not be able to provide a 100% complete read of this book because the book is literally being written as I've purchased it. However, I will be more than happy to review what has been written and post my findings and actions of it by the end of March. Who knows, perhaps more of the book will be delivered by that time and I'll be able to offer some more details when that happens.

This is what the book says it will ultimately include:

Table of Contents

• Foreword
• Introduction
• Why is testability important
• What does hard to test feel like
• What does testable feel like
• What leads to testability being neglected
• What is covered in this book
• How to use this book
• Feedback and suggestions
• 1. Use a testability inception deck to visualize current team and system state and create an environment for improvement
• 2. Adopt testability mapping to expose common smells of hard to test architectures
• 2.1 Gathering data on poor architectural testability to detect systemic problems
• 2.2 Low testability architectures contribute to slow feedback and deficient decision making
• 2.3 Identify the symptoms of poor architectural testability
• 2.4 Exercise: Measure the impact of testing smells on your architectural testability
• 2.5 Understand how testable architecture can impact your team’s testing efforts
• 2.6 Summary
• 3. Use risk and incident data to remedy architectural design problems which inhibit feedback from testing
• 4. Adopt ephemeral development environments to diversify testing techniques early and create shorter feedback loops
• 5. Utilize events and metrics to model risks in production for continuous improvement of your test strategy
• 6. Adopt incident postmortems to maintain a testability focus as part of your team’s continuous improvement strategy
• Terminology
• References
• About the authors
• Notes

It seems like a good place to start and I for one like to know I'm helping to fund progress on books I'd like to see be written. Win-Win!!!

Big Log - #30DaysOfTesting Testability Challenge

Anyone who has read my blog for a little while knows that I tend to fit song titles into my blog posts because it's just silly fun I like to do. Do you know how hard it is to find blog titles related to logs ;)? Robert Plant to the rescue (LOL!).

OK, seriously, I'm getting caught up with the 30 Days of Testability Challenge and here's the second checkpoint.

Perform some testing on your application, then open your applications log files. Can you find the actions you performed in the logs?

I hate to tattle on my application but it's not so much that I can't find what I need to in the log files, it's that there are so many log files and situational log files that it's a process to figure out exactly what is being looked at. I'm mentioning this because we need to keep a clear understanding of what we mean when we say "your application". Do we mean the actual application I work with? Do we mean the extended suite of applications that plug into ours? I mention this because, for each of the components that make up our application, there is a log file or, in some instances, several log files to examine.

We have a large log file that is meant to cover most of our interactions but even then, there are so many things that fly past that it can be a challenge to figure out exactly what is being represented. Additionally, there are logs for a number of aspects of our application and they are kept in separate files, such as:

- installation and upgrades
- authentication
- component operations
- third-party plug-ins
- mail daemons
- web server logs
- search engine logs

and so on.

To this end, I have found that using screen, tmux or byobu (take your pick) and splitting one of my windows up into multiple fragments allows me to have a clear look at a variety of log files at the same time so that I can see what is actually happening at any given time. Some logs fly by so fast that I have to look at individual timestamps to see dozens of entries corresponding to a single second, while other logs get updated very infrequently, usually when an error has occurred.

To that end, I'm a little torn as to my preference. Having monster log files to parse through can be a real pain. However, having to keep track of a dozen log files to make sense of the big picture is also challenging. Putting together an aggregator function so that I can query all of the files at the same time and look for what is happening can be a plus but only if they use a similar format (which, unfortunately, isn't always the case).

Based on just this cursory look, what could I suggest to my team about log files and testability?


If we have multiple log files, it would be a plus to have them all be formatted in a similar way:

 - log_name: timestamp: alert_level: module: message

repeated for each log file.

Having an option to gather all log files into an archive and have them archived each day 9or whatever time option makes the most sense).

Make it possible to bring these elements together into the same file and be parsed so as to determine what is happening and if we are generating errors, warnings or providing info messages that can help us to determine what is going on.

Finally, if at all possible, try to make the messages put into the log files as human-readable as we can.

What Does Testability Mean To Me?: #30daysoftesting

I've decided that I need to get back into the habit of writing stuff here just for me and anyone else who might happen to want to read it. It's true that any habit that one starts is easy to keep rolling. It's just as easy to stop a habit and literally not pick it up again because of vague reasons. I haven't posted anything since New Year's Eve and that's just wrong. What better way to get back into the swing of things than a 30 Days of Testing Challenge?

This time around the challenge is "Thirty Days of Testability".  As you might notice, today is the fifth. Therefore I'm going to be doing some catchup over the next couple of days. Apologies ahead of time if you are seeing a lot of these coming across in a short time frame :).

So let's get started, shall we?

Day 1: Define what you believe testability is. Share your definitions on The Club.

First, let's go with a, perhaps, more standard definition and let's see if I agree or if I can actually add to it. Wikipedia has an entry that determines overall testability as:

The logical property that is variously described as contingency, defeasibility, or falsifiability, which means that counterexamples to the hypothesis are logically possible.
The practical feasibility of observing a reproducible series of such counterexamples if they do exist. 

In short, a hypothesis is testable if there is some real hope of deciding whether it is true or false of real experience. Upon this property of its constituent hypotheses rests the ability to decide whether a theory can be supported or falsified by the data of actual experience. If hypotheses are tested, initial results may also be labeled inconclusive.

OK, so that's looking at a hypothesis and determining if it can be tested. Seems a little overboard for software, huh? Well, not necessarily. In fact, I think it's a great place to start. What is the goal of any test that we want to perform? We want to determine if something can be proven correct or refuted. Thus, we need to create conditions where a hypothesis can either be proven or refuted. If we cannot do either, then either our hypothesis is wrong or the method in which to examine that hypothesis isn't going to work for us. For me, software testability falls into this category.

One of the aspects that I think is important is to look at ways that we can determine if something can be performed or verified. At times that may simply be our interactions and our observations. Let's take something like the color contrast on a page. I can subjectively say that a light gray text over dark gray background doesn't provide a significant amount of color contrast. Is that hypothesis testable? Sure. I can look at it and say "it doesn't have a high enough contrast." That is a subjective declaration based on observation and opinion. Is it testable? As I've stated it, no, not really. What I have done is made a personal observation and declared an opinion. It may sway other opinions but it's not really a test in the classic sense.

What's missing?

Data.

What kind of data?

A way to determine the actual contrast level of the background versus the text.

Can we do that?

Yes, we can, if we are using a web page example and we have a way to reference the values of the specific colors. Since colors can be represented by hexadecimal values or RGB numerical values, we can make an objective observation as to the differences in various colors. By comparing the values of the dark gray background and the light gray text, we can determine what level of contrast exists between the two colors.

Whether we are using a program that can create a comparison or an application that can print out a color contrast comparison, what we have is an objective value that we can share and compare with others.

"These two colors look too close together"... not a testable hypothesis.

"These two colors have a contrast ratio of 2.5:1 and we are looking for a contrast ratio of 4.5:1 at the minimum" ... that's testable.

In short, for something to be testable, we need to be able to objectively examine an aspect of our software (or our hypothesis), be able to perform a legitimate experiment that can gather actual data, and then allow us to present that data and confirm or refute our hypothesis.

So what do you think? Too simplistic? Am I overreaching? Do you have a different way of looking at it? If so, leave a comment below :).