Skip to main content

Considerations when creating automated tests

We recently released to a number of teams our automated regression pack that has been worked on over the past few months. This regression pack tests legacy code, but contains a large number of tests. 

As a bit of background, a number of teams are working on new solutions whilst some are still working on legacy code. With this in mind we constructed an email with a list of guidelines when creating new tests that need to be added to this regression pack.  I figured that these can be quite broad so should apply for any organisation, so thought it would make an interesting blog post... 

So here goes,  when creating automated tests, it's important to consider and adhere to the following:

- Think about data. The tests need to retrieve or set the data they need without any manual intervention - This should help them be more robust and easier to run without manual intervention.
- The tests need to be idempotent - By making it so that each test is standalone and does not affect other tests you will not get random failures if you run the tests in certain orders etc.
- The tests should execute and pass when run in isolation but also in sequence.
- There should be no dependency between tests and it should not matter which order the tests are run.
- The tests should also run on International sites. If they don't apply to one or more sites, use the appropriate tagging - They shouldn't be concerned with language etc. if the element is the same then the test should still be able to run and find the element.
- The tests shouldn't be flaky. Random failures shouldn't happen.
- The tests should be able to be run on any Development environment - There should be no third party dependencies that are not available in any development environment, this way teams will get the most value out of the tests
- The tests should follow some form of existing structure (if one is in place) - We used Page Object model and as such we had a structure that suited this framework.
- The tests shouldn’t take long to run. This can be subjective, but if it takes longer than doing a manual test, then try and come up with a better solution
- Try to reuse code wherever possible, do not duplicate
- Existing Page Objects should also be used where possible
- Changes to shared packages should be run by others first
- Avoid testing too many things at one time in one test, or tests that are “too long”. Ideally, when a scenario fails, you want to immediately know what went wrong.
- Try to keep scenarios within system boundaries. For example, if your test needs some products in the bag, don’t let your test do that through the UI. Do it via the DB.

So that's pretty much it :)  

If you can think of anything else that I might have missed off, that you think needs to be considered when creating an effective and reliable set of automated tests, please let me know!
Labels:

Comments

Post a Comment

Popular posts from this blog

Testers: Be more like a Super-Villain!

Who doesn't love a Super Hero? Talk to my son, and he'll tell you how much he loves them, talk to many adults and they'll say the same! Deep down, we all love to be the Super Hero, we all want to save the day! However, I want to talk about the flip side of Super Heroes, the Super Villains... I often play Imaginext with my son, and I (unfortunately?) am nearly always the Super Villain! Be it Lex Luthor, Joker, Two Face, Mr Freeze or The Riddler! These are all great characters and great Super Villains, but why would I want to write about Super Villains? A while ago where I worked, we had a few Super Heroes, people who would be able to come in and "fix" things that had broken and help deliver projects on time. We then shifted, we decided to do away with the Super Hero culture and try and prevent from being in that position in the first place, whilst we didn't go as far as wanting to hire Super Villains, it's definitely a story that has stuck with me and t...
2 comments
Read more

Value in QA Courses/Qualifications?

I have in the past questioned the value in getting certifications/going on courses for the sake of getting a certificate in testing. Whilst I do still question the worth of such an issue, I have recently read some articles which has shown me there is more value in these courses/certificates than I previously gave them credit for. The main positive that I can think of, upon completing a course like an ISEB Foundation, is that it ensures that testers are on the same page when it comes to communicating. A bug is a bug, or if I'm speaking to someone about Integration testing, they know exactly what I am talking about and won't get confused. I think in ensuring that everybody is on the same page when it comes to discussing testing issues/testing activities, it helps in gaining respect and confidence from other teams and other team members, as we are all singing from the same hymn sheet.  It isn't just about communication in the term of words however, it is impor...
5 comments
Read more

Measuring QA Key Skills and Competencies

I have been thinking about how I can help encourage self improvement within my team, as I understand it, everyone wants to improve, it's just that often there are a number of things that hold people back. I believe one of these things that hold people back are around identifying skills that they are perhaps weak in or that they could/should improve on. So I thought about how I can help tackle that problem. One solution that I want to try with people is to identify the key skills for a QA, what key skills should every QA have, or at least what key skills make up a good QA? If I can identify these then I can start helping people identify if they are lacking in an area. Sure there is a competency matrix that we have, but it has things like "An excellent understanding of XXX", it's often very difficult to quantify what an excellent understanding actually is. So I sat down and came up with the following key skills: OOP Test Documentation Manual Testing Automated...
3 comments
Read more