Crowdsourcing as a viable alternative?

I was recently at a conference that discussed the use of Crowdsourcing as a form of effective testing for your company.

Firstly let me go over what Crowdsourcing is. Wikipedia describes Crowdsourcing as:

"the practice of obtaining needed services, ideas, or content by soliciting contributions from a large group of people, and especially from an online community, rather than from traditional employees or suppliers"

So in testing terms, it's getting an online community of "testers" to test your software.

So why would you use crowdsourcing to test your software? What are the positives?

The big positive for me is that you're letting the people who would potentially use your application the chance to use it and report any bugs. These testers might use the application differently than you intended though? This to me however is a good thing, if they are using it in a certain way, you can bet that the general public or whoever might use your app would use it the same way, so this is definitely a positive. (That is to say that it shouldn't replace true beta testing).

It also allows you to have access to a large number of devices, for instance you can specify what devices you want tests to run on, or want the website to be tested on. Chances are unless you have an enormous budget for buying devices there are going to be plenty of devices that you don't have and can't afford to have, that could possibly do with testing on.

Finally, you can have a big turn around on the testing. I've heard stories of insane numbers of tests being executed in an even insane amount of time! Something that I dont think a QA team can achieve, unless they have massive numbers of people, but then you have the issue of managing this. If you crowdsource it, then they will manage all of that.

What are the disadvantages? Why wouldn't you use crowdsourcing?

There are in my eyes many disadvantages to crowdsourcing, which I will go over below.

The biggest one for me, is that the testers have no business knowledge of the application. They will raise bugs that aren't bugs, they will not fully understand what the application should be doing, and with this in mind bugs may even be missed because of this.

Secondly, you have no idea who is running the test and if they are any good, there is of course an argument that the people running the tests could ultimately be the people using the software, so they don't have to be good testers, but they do need to have a good eye.

Related to the fact that you have no idea who is testing the software, in having no idea who is testing the software, you are blindly trusting someone to execute a test, and what is to say they have executed the test? You could argue you have a similar problem in that you have to trust your team to run the tests, but your team is something you have put together, and people that you know and hopefully trust.

Then there is the issue that the app/system must be publically available, which might not be a problem to some, but it is if you have a large number of people hitting your application on a small test server somewhere.

It also in my eyes, plays down the importance of testing, testing to me, isn't just about executing test cases, it's about identifying areas of risk, identifying what testing needs to be performed, understanding the application. I fail to see how crowdsourcing can offer you this.

Finally, a lot of crowdsourcing is paid per bug, this is extremely dirty to me. I don't agree that people should be paid per bug, as I don't see that motivating people to actually run the tests, as they will blindly be looking to find bugs and not execute tests. Also, the money paid for a bug would go further to motivate someone where the exchange rate is advantageous, to someone in the UK. Finally, someone could spend all day testing, and doing a good job, but not find any bugs, what is the motivation for that person to come back and perform more testing?

Some might argue that it's a form of beta testing, but again I disagree, you can't guarantee that the people testing would be people that would regularly use the application or system, beta testing should be performed by people who will use your application, not just anyone who wants to earn some money finding bugs in your software.

When would you use crowdsourcing? 


So with all those you might think that I am totally against crowdsourcing, which I am... to a degree. However, I can see when it might be useful, and that is as a complimentary effort to the testing that is already performed by a company. A company shouldn't solely rely on crowdsourcing as in my eyes there are too many unknowns over who is executing the tests.

Also, it might be useful if someone is developing a website/simple app and they need it testing for a small company, and they don't have a QA function, then they might use Crowdsourcing as a quick and dirty way of ensuring their software is ready to be released.

So, in conclusion, whilst it does have it's advantages, and can be useful for small scale applications, I don't think it's suitable for a large application or an application that is evolving that doesn't have many test cases around it. If you need to get through a ton of regression test cases on a multitude of devices, then again it might be useful, but again I would say only as a supplement to any testing that is being performed by a QA team dedicated to the application under test.