From Test Cases to Context-Driven Testing

Gary Miller speaking at CAST conference.
Gary Miller presenting “From Test Cases to Context-Driven Testing: A Start-up Story” at CAST 2015

On Tuesday August 4, 2015, I presented my experience report on moving from test cases to context-driven testing at the 10th annual Conference for the Association for Software Testing 2015 in Grand Rapids Michigan.  This was my inaugural speaking engagement and my second software conference.

Thank you to all of those who attended and watched online. My talk and the open season questions were recorded and are available online from the AST thanks to Ben Yaroch.

Thank you to Katrina Clokie who encouraged me to speak, as well as Michael Larsen, both who helped me write the proposal.  Your help was invaluable.

My presentation slides are available here  and on the CAST 2015 website.

Thank you to all who participated by asking questions.  The conversation doesn’t stop just because the conference is over.  I welcome you to comment here on my blog or contact me on Twitter @1030omlette for further discussion.

2 thoughts on “From Test Cases to Context-Driven Testing

  1. erikhun

    Hey! Thanks again for sharing your story! I’m watching your recording again as I’m looking for practical examples on how to apply RCRCRC. I just arrived to the point when you say you A/B tested your existing suite against the new one applying the heuristic. (https://www.youtube.com/watch?v=ENJQGYoE2jU&t=18m50s)

    You also say that while the old suite has passed you started to find problems when you were exploring away from the script.

    What did you do here exactly? Did you write a test/check suite applying the heuristic and indeed run both of them against the same build? Or you started to think along the RCRCRC dimensions and started testing? If so, if it was indeed testing, could you replicate that kind of thinking into a new suite? How did you go about maintaining it, making it stay relevant?

    1. Thanks for the question. When we ran them side by side, we examined each dimension of RCRCRC and came up with ways of gathering information about each of them. For example, when considering the recent changes, we looked in the sprint backlog to see what type of work the team did this sprint and asked questions about how we might want to explore those areas for possible problems. Generated tests were dynamic, and related to what recent work was performed. Same thing when we looked at the other dimensions – for Chronic, we looked at a dashboard of recent bug reports, using features such as word clouds to visualize which components or modules had the most bugs recorded against them, or had recent bugs reported against them. When it came to Core stuff, we used our old test cases as a jumping off point to help us target the most centric parts of our platform, regardless if they had changed this sprint or not.

      To keep the tests relevant, we evaluate the RCRCRC each sprint and find that we don’t have the same Chronic issues every sprint. We don’t have the same Changes every sprint. We don’t have the same Risks every sprint. But we sorta do have the same Core every sprint. If we didn’t re-ask questions each sprint, we would likely have fallen back into a process where we repeated the same tests over and over. We might as well go back to test cases if we did that.

      Although we hadn’t done it yet, one could encode the Core tests into a suite, and perhaps even the other dimensions. However, if you haven’t encoded a test for that Risk, that Core thing, that Chronic thing, then the tests might not help you find certain types of problems. So you still have to keep the tests fresh and evaluate them periodically to make sure they are valuable. I consider a valuable test one that helps me find problems. If I never find problems through my tests, I probably need different tests, varied inputs, or different conditions to test for. If you are writing test cases or coming up with them on the fly for this heuristic, a human is still a valuable component of the process to help ask and answer questions.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s