Skip to main content

Should a failed test on CI automatically raise a bug?

I got into work this morning, and was looking at some CI Builds, and had a thought that it might be useful to have a bug raised automatically (through an API) when a test that is running on CI failed.... I tweeted about it to get peoples opinion(s)...



I got a few responses on twitter, and one which made me rethink why this might be a good idea and that was:


It could easily lead to and incredible amount of noise if we are raising bugs against every failed test, and also 1 failed test case != 1 bug, in that if 3 automated tests on CI fail then we don't necessarily need 3 bugs. It could also potentially (as my reply states) devalue what a bug actually is, meaning that if a tester raises a bug then it might get ignored/lost in all the noise. 

I also asked our internal QA Slack channel, responses were informative, and again helped lead me away from this potentially noisy and crazy idea.



Both of these points are extremely valid, and if you make it the number 1 priority to fix a broken test then why bother to create a bug? Surely just the visibility of a red cross next to a test run is enough to get people to work on it? Which is very true. 

Another factor to consider is the type of test, if a Unit Test fails, do we need a bug? Most definitely not.... There is definitely more of an argument if an Acceptance Test fails though (though after this discussion today I don't think there is!).

So, what started out as a blog post about CI and bugs, has provided me with 3 insightful thoughts:

1 - The focus of this blog post being that a failed test on CI most definitely does not mean a new bug (whether the process is manual or automated in creating said bug) needs to be raised. It ultimately would devalue a bug, it would mean that people wouldn't necessarily talk about said issues/bug, and 1 failed test != 1 new bug.

2 - Asking people for feedback is invaluable, it can help shape your opinion and give you quick feedback before you go off down a rabbit hole!

3 - Discussions like this are the exact reason why I wanted to have an internal QA community, so we can discuss things and get feedback.


Comments

  1. Unknown31 March 2017 at 22:13

    Quite an insightful post. This has cleared so many of my doubts in this subject & has thrown light on many aspects that I didn’t know before.

    ReplyDelete
  2. Kerry Schultz23 November 2017 at 07:45

    there is always first time for everythign in life and it only takes one small step to conquer the greatest fears although this small step could take alot of courage but at the end its always worth it.

    ReplyDelete

Post a Comment

Popular posts from this blog

Testers: Be more like a Super-Villain!

Who doesn't love a Super Hero? Talk to my son, and he'll tell you how much he loves them, talk to many adults and they'll say the same! Deep down, we all love to be the Super Hero, we all want to save the day! However, I want to talk about the flip side of Super Heroes, the Super Villains... I often play Imaginext with my son, and I (unfortunately?) am nearly always the Super Villain! Be it Lex Luthor, Joker, Two Face, Mr Freeze or The Riddler! These are all great characters and great Super Villains, but why would I want to write about Super Villains? A while ago where I worked, we had a few Super Heroes, people who would be able to come in and "fix" things that had broken and help deliver projects on time. We then shifted, we decided to do away with the Super Hero culture and try and prevent from being in that position in the first place, whilst we didn't go as far as wanting to hire Super Villains, it's definitely a story that has stuck with me and t...
2 comments
Read more

Start with the End in Mind - My first presentation at a tech meetup!

I was at a football coaching session the other night, and some other coaches put on a training session for us, so that we could learn and critique it. This is not an easy thing to do, to put something on for your peers and open yourself up to criticism is a difficult thing to do. One of the comments from the president of the club was that in order to develop yourself you need to push yourself and step outside of your comfort zone which it was evident that these coaches were doing. I took this to heart in many ways, a few weeks ago I signed up to do a presentation at a meetup that was only a couple of meetups old, The QE Roundabout . I was in contact with Zoe Canning (the event organiser) and I knew it was something I wanted to do, but it's like anything, saying you want to do something and then putting yourself in a position to do it are sometimes two very different things. Anyway, I volunteered to do one, the theme was Automation & Architecture, and we were free to ta...
1 comment
Read more

Delusions of Testing

So I've got in touch with my old QA friend, Richard Lee and we spoke about guest blogging on each others blogs... Richard is an IT Professional for a FinTech based company in London. His activities vary from Release Manager, Build Manager, Database Administrator. Working in a Microsoft workshop, his expertise lies in MSBuild/Workflow/Powershell/SSAS/SSIS/SSRS/SQL, basically whatever isn’t anyone elses’ problem is Richard's problem! When not solving other peoples problems he can be found blogging at redphoenix.me , and jogging to and from home, where he lives with his heavily pregnant wife. Hello, my name is Richard, and I am a former tester. Like most people and their careers, I fell into testing; I first got into testing about 6 years ago, after I had graduated. I went to a university where the attitude was that you should try to get on a graduate scheme with one of the big companies. If you weren’t interested in that, well, good luck with getting any support from ...
3 comments
Read more