Crowdsourced Testing – An Overview

Crowdsourced testing plays a pivotal role in testing of mobile applications. As we are aware, some of the reasons why most application testing companies find it difficult to test a particular mobile is because of the large number of targeted devices having updated or basic OS, multiple carrier providers located in different parts of the world and multiple platforms based on devices. Furthermore, this method of testing is considered when the application is open to the world and is inclined towards user-centric feedback such as, social networking sites, and mobile apps which are available to anyone for testing.

I remember one of the use cases where we had to create an online reader for a global organization specializing in developing cloud based learning platforms. The online reader was targeted for virtual classrooms across the globe and the content was deployed on CDN servers. We wanted to rollout the beta build worldwide at the earliest to match the launch schedule. We thought of choosing crowdsourced testing for beta testing. During the testing phase, our CDN monitoring tool showed how the particular site is being accessed from various regions. After crowdsourced testing rollout, we were able to fix errors pertaining to devices and security. Once these issues were sorted, we ensured to test the application one last time at our own lab and we were happy to release it globally on the same day.

Where is crowdsourced testing useful:

To perform load testing in a distributed applications crowdsourced testing plays vital role. Testers will hit the application from different places and will analyze how the application behaves. Also, crowdsourced testing highlights changes in weather condition, browser specific issues, network/carrier issues, along with testing of rare embedded devices. Using crowdsourced testing, we can perform end-user perspective realistic usability testing as well.

Let us now look at the widely used Crowdsourced Testing process:

• Usually in crowdsourced testing, the customer has the business requirement plans and desirable goals which he wants to achieve over a period of time
• As per the business plan, the project coordinator prepares the test cases and recruits crowdsourced testers and shares the environment details and test cases with the customer
• Crowdsourced testers across the globe perform testing on the test cases. In certain scenarios, they are required to perform performance and security testing as well. The results and outcomes are shared with the project coordinator.
• The project coordinator then validates the testing outcome and prepares the final set of deliverables and its results
• The final outcome is then shared with the customer where the statistics play a major role denoting the outsourced project being a success

Results derived by implementing Crowdsourced testing:

• In-House testing may be restrictive in regards to availability of resources and exposure to certain situations. This is where crowdsourced testing is preferred.
• Very cost-efficient as you need to pay only for valid bugs that are reported through crowdsourced testing
• Testers are unbiased towards the internal concerns of the company
• Testers are diverse with variations like different devices, operating systems, environments with low or high bandwidth and areas that help to fix localization issues as well as different environment issues.

Disadvantages of Crowdsourced Testing:

• Data security and confidentiality must be managed by crowdsourced testers
• Prompt communication with crowdsourced tester is difficult as they are located at different places and do not directly interact with customer
• Testers are paid based on number of bugs they log in the report and they may log large number of less effective bugs and skip over critical and difficult to reproduce bugs
• Test coverage is difficult to maintain if compared with traditional testing