Skip to main content

Test Iceberg of Automation

I'm sure you've all heard of the automated testing pyramid, I'll describe it briefly here, but you can read all about it here.



It's essentially a strategy that shows good practice ratio of Acceptance Tests (generally UI) to Integration Tests to Unit Tests, and here it is here in a simple form.

It states that it is a good ratio to have your testing covered with 10% of acceptance tests, 20 % integration tests and 70% unit tests. Why is that you may ask? The primary focus of this is on Return on Investment, by finding bugs/breakages at Unit test level you are finding cheap bugs, as Unit Tests are quick and easy to maintain, whereas acceptance tests, whilst having value, are harder to maintain and take longer to run.

Obviously, it's not a strict ratio, but I think it's a good practice to try and live by.

However, I digress, the main point of this post is to put another spin on the automation triangle, and is possibly more QA centric than the automation pyramid, as I don't often see (rightly or wrongly) QA getting involved in creation of Unit Tests/Integration tests, I feel you can visualise the ratio in an easier way, and that is in the form of an Iceberg.



Interesting Fact: only around 11% of an iceberg is actually visible...

Now if we apply that interesting fact to automated testing, (I guess you can see where I am going with this) we can say that the 11% that we can see as QA is the acceptance tests, these are tests that in general have been driven by the QA and are what will often fall into our domain to maintain and create.

The remaining 89%, are more dev focused, in that unit tests and integration tests are generally maintained by the developers, and not the QA department (at least in my experience). So this allows the QA team to work on acceptance tests and get them working effectively, and leaving the Unit and Integration tests to the developers (although it is definitely wise to get involved as much as you can).

Please don't think I'm saying that unit tests aren't important (quite the opposite as my previous blog posts will attest to), as QA need to be aware of what Unit Tests and Integration tests are to be run, as this will influence what tests are run as part of the acceptance tests, you only have to look at the Titanic to see what happened there when the rest of the iceberg was ignored.
Labels:

Comments

  1. Unknown24 January 2014 at 06:25

    Nice post Gareth. I've been working the same iceberg metaphor around in my mind for a while now.

    I hadn't limited the metaphor to just automation though - in my opinion, the iceberg is a metaphor for the test strategy.

    Every member on the development team should be aware of just how much iceberg there is, both above & below the water - i.e. what is the entire test strategy of the development team.

    I really should get my thoughts onto paper for some open criticism.

    Thanks for the post Gareth,

    Duncs

    ReplyDelete
  2. Calkelpdiver23 February 2015 at 08:27

    One thing to note is that Mike Cohn has stated that because of the types of tests and where they are at in the pyramid (which causes it to be a pyramid shape) structure you will get this distribution.

    He doesn't necessarily say this is the way it should be as common practice, he states that it comes out this way because at the lower levels you have more 'atomic' tests (Unit/Code level tests/asserts) in order to get coverage of the code.

    The benefit is finding issues/defects sooner and reducing the amount of churn/rework (lower cost ratio in comparison to later on down the road to release).

    The whole purpose IMO of the pyramid is to get testing going earlier and to leverage the tools at hand (xUnit Harnesses and automation). Acceptance Test level tests are valuable, don't disregard them because of the smaller percentage. These tests via proper framework architecture and planning will be easier to build and maintain. They have value as they represent the final integration of the system with the user.

    By following the suggestions of the pyramid you start at the micro level and work towards the macro. This provides greater coverage and exercising of the software under test. By using automation you leverage a tool/machine to aid in the execution of those tests and get efficiency gains as a benefit. That is what Mike is really pushing.

    Jim Hazen

    ReplyDelete

Post a Comment

Popular posts from this blog

Testers: Be more like a Super-Villain!

Who doesn't love a Super Hero? Talk to my son, and he'll tell you how much he loves them, talk to many adults and they'll say the same! Deep down, we all love to be the Super Hero, we all want to save the day! However, I want to talk about the flip side of Super Heroes, the Super Villains... I often play Imaginext with my son, and I (unfortunately?) am nearly always the Super Villain! Be it Lex Luthor, Joker, Two Face, Mr Freeze or The Riddler! These are all great characters and great Super Villains, but why would I want to write about Super Villains? A while ago where I worked, we had a few Super Heroes, people who would be able to come in and "fix" things that had broken and help deliver projects on time. We then shifted, we decided to do away with the Super Hero culture and try and prevent from being in that position in the first place, whilst we didn't go as far as wanting to hire Super Villains, it's definitely a story that has stuck with me and t...
2 comments
Read more

Start with the End in Mind - My first presentation at a tech meetup!

I was at a football coaching session the other night, and some other coaches put on a training session for us, so that we could learn and critique it. This is not an easy thing to do, to put something on for your peers and open yourself up to criticism is a difficult thing to do. One of the comments from the president of the club was that in order to develop yourself you need to push yourself and step outside of your comfort zone which it was evident that these coaches were doing. I took this to heart in many ways, a few weeks ago I signed up to do a presentation at a meetup that was only a couple of meetups old, The QE Roundabout . I was in contact with Zoe Canning (the event organiser) and I knew it was something I wanted to do, but it's like anything, saying you want to do something and then putting yourself in a position to do it are sometimes two very different things. Anyway, I volunteered to do one, the theme was Automation & Architecture, and we were free to ta...
1 comment
Read more

Delusions of Testing

So I've got in touch with my old QA friend, Richard Lee and we spoke about guest blogging on each others blogs... Richard is an IT Professional for a FinTech based company in London. His activities vary from Release Manager, Build Manager, Database Administrator. Working in a Microsoft workshop, his expertise lies in MSBuild/Workflow/Powershell/SSAS/SSIS/SSRS/SQL, basically whatever isn’t anyone elses’ problem is Richard's problem! When not solving other peoples problems he can be found blogging at redphoenix.me , and jogging to and from home, where he lives with his heavily pregnant wife. Hello, my name is Richard, and I am a former tester. Like most people and their careers, I fell into testing; I first got into testing about 6 years ago, after I had graduated. I went to a university where the attitude was that you should try to get on a graduate scheme with one of the big companies. If you weren’t interested in that, well, good luck with getting any support from ...
3 comments
Read more