Skip to content →

What is crowdsourced testing?

Definition and the basic theory of crowdtesting

“Testing”, “crowd” “sourced” – based on the fact that all the words are known, could you guess what it’s all about?

Could you?

Well, Applause! 


Many IT folks never heard about the concept, which isn’t surprising.

Wiki may refer to it as “an emerging trend in software testing”, but let’s face the truth – most people ignore such thing exists.

Given I’ve spent a while exploring this niche I am eager to share some inside information.

Interested? Follow me.

Let’s start with the boring part – definition and basic theory.

Crowdsourced testing is a process which delegates the manual software testing procedure to numerous testers distributed all over the world (the crowd).

Despite the fact, that crowdsourced platforms position themselves as “QA testing as a Service solution” in the strict meaning of the word, it is not the case.

Anyone who ever had to deal with QA field is almost certainly familiar with this classical scheme. Quality assurance (QA) is the broadest category, testing is a segment of QA.

Work at the crowdsourced platforms is all about “Testing” which is mostly an evaluation of the interfaces from the end user perspective.

What is the general workflow for the crowdsourced testing platforms?

The simplified scheme would look something like this.

Clients > Crowd testing platform >  Testers

Clients are mostly well-established companies that are interested in external testing of their software on a wide range of devices and platforms.

Crowd testing platforms – are the intermediate between clients and testers. They take care of all organizational aspects (internal bug tracking systems, bug review, support). Popular Consumer Web and Mobile apps are their main target clients.

Testers – are professional QAs or technologies geeks who are interested in breaking the software. There are some trials and evaluations (depends on the platform), but the point is that anyone can register there and give it a try. This particular aspect seems quite inspiring. It all depends on the level of your practical skills in discovering software defects and not on how fancy your CV is.

How does the crowd testing process look like on the tester side?

You register on the crowd testing platform, pass the initial trial and start receiving the email invitations for the test cycles. You may review the specification, the timeframe and the expected payout. At this point, you either:

а) Decline the test offer

b) Accept the offer and proceed with the project

It’s always up to you if you accept the offer or decline. No one is there to force you to work. If you feel like laziness might be an issue, you better not to venture out there. The crowd requires self-discipline more than other types of freelance. It may hurt your statistics a bit if you join the cycle and don’t do anything. Other than this it’s you who decide how much time you are going to allocate for testing and what areas to test. The payout at crowdsourced platforms is strictly linked to the number of defects found by each tester and approved by the customer (you may discover more about test cycles in the crowd here) 

Perks and downsides of working in the crowd

Let’s start with the advantages:

  • The extreme flexibility comparing with other forms of freelance. With regular freelance contracts, you are involved in relatively long-term commitment. In the crowd, you take a decision to commit or not on a daily basis. The test cycles have very short intervals (up to 3 days), so after you’ve finished one, you may agree for a next cycle for the same customer or may decline without consequences.
  • Ability to test a huge variety of Web and Mobile products. I can’t think of any equivalent QA format that would allow testers to explore such a wide range of software products in different fields: e-commerce, education, healthcare, real estate, mass media, entertainment, social media, transportation, tourism etc.
  • Automatic project invitations by email. Crowd freelancers don’t have to look for clients by themselves. The platform takes care of this and distributes the available projects among testers.
  • Most cycles require exploratory testing which allows more freedom and creativity in the terms of scenarios exploited.

Naturally, everything comes at a price. Here are the major disadvantages testing in the crowd:

  • Time and efforts applied aren’t always proportional to the payout you receive. The defects may be rejected for a whole set of reasons (out of scope, not fitting the priority limit or issue type), in which case you aren’t compensated for the time spent. You receive nothing.
  • Lack of assistance. Some platforms do offer chat and support function, but most of the time the issues you encounter can’t be resolved as quickly as the crowd dynamics requires. You are on your own and have to get by.
  • The unpredictability of future projects. You never know when the next test invitation arrives, which basically shatters all your planning and time management efforts.
  • Slots in the projects are usually very quickly filled. You may miss an interesting project simply being unable to confirm your participation right away after receiving an email invitation. The count sometimes goes for minutes or in some cases for seconds. Quite a stressful experience (You may check out my fail list with crowdtesting here)

You didn’t think it was all black or white, did you?

The crowd is not an error-tolerant place. Moreover, it’s the most highly competitive environment I’ve ever had to deal with.

You may hate it or you may like it, nevertheless, the experience it gives you is priceless.

So in case, you’d like to know more, let me be your guide.

Published in Crowdtesting


Leave a Reply

Your email address will not be published. Required fields are marked *