How to decide what and when to automate tests?

We all know that repetitive manual testing can be and is at times boring.... but unfortunately it's a necessity for some aspects of testing.

One thing that I love, and sure enough this reduces the load of manual testing, is automated testing, be it from the Service level, through an API and especially WebUI testing. Whenever any testing comes along, there is a question that is regularly asked by QA, do we want to automate this?

When deciding what tests we should automate I feel that it's important to answer some questions:

• Will this test form a part of the regression pack for the application?
• Will this test be run multiple times during the development process?
• Can the same level of testing be achieved by automating this test?

I'll tackle the first question, as it's the most basic and the easiest to answer. If a test is to form a part of a regression pack, then yes it should be automated. The reason being that it will save time in the future, and will offer more assurance in releasing future software releases.

As for the second question, if a test is to be run multiple times then surely it makes sense to automate it and reduce the amount of effort it takes to run. This is especially interesting when it comes to a test to check if a bug has been fixed, in order to verify that subsequent builds do not reintroduce this bug into the wild, then by all means automate it, and run it as part of the build process (if at all possible) to try and catch these issues.

Finally, there are some aspects of manual testing that cannot be automated, for instance, checking the location of an element on a web page (for Web UI testing), whilst the naked eye can easily notice if an element is misplaced, or if something is appearing incorrectly (i.e. text overlapping other text). Because of this, I tend to shy away from automated testing for cross browser testing at the moment... However...

Google have an interesting piece of software that monitors the top 1000 pages in search results when testing new versions of chrome, they will detect if there are any variations between the version under test or previous versions, and email developers to let them know. It is even so clever to include knowledge of where there may be dynamic content, like on bbc.co.uk for instance where news articles are ever changing and dynamically creating the front page. Whilst I understand that something like that is extremely complex and possibly overkill for some applications, it is an extremely impressive piece of software that I would love to one day see in action!

So whilst automation is an extremely effective toolset to have, there will always have to be some element of manual testing to go along with the automation. Now this manual testing, doesn't have to be scripted, far from it, it can be in the form of Exploratory Testing (more about in future posts). As time goes forward, I am sure there will be more effective ways of performing cross browser testing and ensuring elements are displayed on the front end.  This doesn't hinder the effectiveness of automating tests at a service level or even an API level, as the response and requests for these are structured in a way that isn't going to change over time, and so I find that you can achieve 100% coverage on Service and API level tests in an automation test suite.

There is also the benefit of the time saved by automating a test can be spent tackling more important and complex test cases during the development of an application. So not only does it help with reducing testing effort on regression, it increases the effectiveness of the testing effort going forward. 

It also lends itself to application ownership, which is often only seen during the development of the application, but in reality should be whilst the application is being used, as these tests will live for the lifecycle of the product.

For this post we have only really talked about Acceptance Tests, in future we will discuss the importance of Unit Tests and Integration Tests.