Do we need test automation? If so, where do we start? How often should we be running the tests? Even the most experienced among us are using these questions to define what we test and how we approach it.
On the internet there’s a wealth of information available to us, ranging from framework selection to BDD using declarative vs imperative test scenarios to everything in between.
And while these have proven themselves to be invaluable, there’s a real absence of entry level discussions for people yet to start their automation journey, or those who have started but need some reassurance that they’re doing the right thing.
With that in mind, I’ve put together this guide to starting your journey into automation, including some simple ways to begin tackling the bogeyman that is test automation.
If you are starting to move to automation, start with your regression pack, which may be a collection of manual test cases that you run each time you do a release. If you start by automating just that, over time, you’ll save a lot of expended effort.
I’ve heard people ask: “What is regression testing and at what point does a new test case become a regression test?” There are lots of different views on this, but the rule I follow is: “Regression tests are a collection of scenarios that ensure, regardless of the functional change, that previous code implementations are not broken by new changes”.
For example, let’s say I write a test case which checks a button has the text “Hello world” and then, a few weeks later, someone adds a new button that says “Hello world 2” next to the original button. My first test is run as a regression test to make sure the second button didn’t break the first button.
If you’ve already been looking into how best to approach test automation, you may have stumbled across a huge number of frameworks, all purporting to be the “world’s best automation tool”. In my opinion, there’s no such thing. It’s more that you need to find the most appropriate tool for the task at hand. You can pick the best libraries, frameworks or tools out there and it won’t suddenly mean you have a good test pack. In fact, if you’re just checking against a list of things, are you really testing?
There’s good news here – computers are really, really good at doing boring checking, whereas humans are great at testing. Let’s make the computer do the boring checks, and we can get on with the interesting testing!
To get a good set of tests, you need to be using good principles in your framework that make it:
Basically, your test code is still just code. Like writing any other code, test code should go through the same rigorous processes development code goes through. This also means test cases are code, should therefore live inside code repositories, and to update them you should follow branching strategies and pull requests to ensure they are protected in the same way dev code is.
At the beginning of your test automation journey, you will come across BDD and TDD, among many other acronyms, and have to decide which one to use. It’s a decision made harder by the fact there are numerous articles about BDD split evenly between “BDD saves the world” and “BDD is ruining test frameworks and doesn’t work”.
The distinction between TDD and BDD is:
BDD and TDD are not mutually exclusive, meaning they can both be used within a project. For example, developers can use TDD to write their unit tests before they start writing the development code, whilst you still follow BDD as a team to ensure all the desired behaviour is met. My view is that BDD is good if you use it the way it was designed, which means that the actual test framework is only part of the process. BDD is a good way for organisations with more of a Waterfall model approach to testing (by which I mean development and test teams as stand-alone teams that pass work ‘over the fence’ to each other) to move to an approach that breaks down the walls between development and testing teams. One of the best ways BDD achieves this is by getting Developers, Testers, BAs, Product Owners, Designers, and anyone else involved, into a Scrum team, to sit in sessions called “Refinement” or “Three-Amigos”, break down pieces of work into Stories and define the Acceptance Criteria, commonly written in Gherkin format. In short, this lets you explain the expected behaviour of a piece of work in simple English so that anybody, regardless of how technical they are, can understand what you’re aiming to achieve. The Acceptance Criteria are then put into a BDD-style automation framework, which is broken down into actual code under the hood but still produces automation test reports that show each of those tests in simple English. This process helps teams design effective test cases and come up with good scenarios to be used as a starting ground for tests.
Some of the common questions I see coming from manual testers moving to automation is “do you still define high priority/severity test cases and have certain ones that are more important than others in the pack?”
The reason it’s a yes and a no is that this amazing automation framework you are writing will become your gateway to saying, “Yes! That bit of code is good enough to make it to the next stage of the route to live”. Here, the entire pack is run in a CI/CD fashion, where every single time a developer pushes their code, the automated regression pack is run, ensuring that nothing is broken (or regressed). This includes testing of the code changes as well as the deployment of that code into environments (regression testing). That same test pack is then used when deploying into each subsequent environment but can normally be tailored (for example, by applying a tag to specific tests, identifying them as something like “Smoke Tests”). A lot of the time, this is a subset of the test pack that the team has defined as tests that cover high “priority/severity” or risky integrations to speed up the execution time of the test pack.
The justification for no is: if you’ve taken the time to write automation tests, then they were important enough to be written. Therefore, they are important enough to be run and run as often as possible. However, the caveat to this is you need to ensure you keep refining/refactoring and ensuring the tests inside your framework stay up to date. Tests need to be regularly reviewed to make sure they are still useful for the evolving nature of the system under build.
You will encounter many more challenges relating to test automation. This article alone won’t get you from a manual test pack to comprehensive CI/CD pipelines allowing you to release code to production daily. When starting your automation journey, you can’t expect that you will get there in a few days, weeks or months. So take the wins and recognise the benefits when you achieve them. If an automation pack that used to take you weeks to run manually now runs overnight, that’s still a huge time saving and a great step forward.