This paper aims to prove the following hypothesis Problem Statement: HYPOTHESIS (1) User Experience collection of mobile applications can be done using the Crowdsourcing mechanism; (2) User Experience collection of mobile applications are influenced by the mindset of Crowdmembers, culture/ethnicity/social background, ease of interface use and rewards, among other factors.
The authors of this paper, did a literature review first to find if Crowdsourcing was applicable and a used method to solve problems in Software Engineering. This helped us to narrow down the application of Crowdsourcing to the Requirements Engineering-Usability (User Experience) collection. User experience collection of two Malayalam language-based mobile applications, AarogyaSetu and BevQ was done as the next step. Incorporating findings from Study I, another study using AarogyaSetu and Manglish was launched as Study II. The results from both cases were consolidated and analyzed. Significant concerns relating to expectations of Crowd members with User Experience collection were unraveled and the purpose of Study was accomplished.
(1) Crowdsourcing is and can be used in Software Engineering activities. (2) Crowd members have expectations (motivating factors) of User Interface and other elements that enable them to be an effective contributor. (3) An individual’s environment and mindset (character) are influential in him becoming a contributor in Crowdsourcing. (4) Culture and social practices of a region strongly affects the crowd-participating decision of an individual.
This is purely self-done work. The value of this research work is two-fold. Crowdsourcing is endorsed significant in Software Engineering tasks, especially in User Experience collection of mobile applications. Two, the Crowd service requesters can be careful about designing the questionnaire for Crowdsourcing. They have to be aware and prepared to meet the expectations of the Crowd. This can ensure the active participation of potential contributors. Future researchers can use the results of this work to base their research on similar purposes.
1. Introduction
Published work in Crowdsourcing defines Crowdsourcing as an open-call, free-to-choose mechanism that calls on individual contributors who are skillful, experienced and willing to contribute to a particular piece of work or service (Hosseini and Mahmoud, 2014; Estellés-Arolas et al., 2015; Kietzmann and Jan, 2017). This mechanism also involves controlling and rewarding participants of a work (Chandler and Mueller, 2013; Goh et al., 2017; Cappa et al., 2019). The group of participants is called a “crowd” and individual participants, “crowd members”. Crowdsourcing is a suitable option when organizations fail to find people with suitable skill sets or lack other resources like software, hardware, appropriate tools, people experienced with such kind of work, etc. within the organization. In most such cases, trying to gather resources will be much more expensive, will take time and not be a worthy solution to the problem. We made a thorough investigation of the literature and found that although there was much research done on applying Crowdsourcing in Software Engineering (Asiegbu Baldwin et al., 2017; Stol et al., 2017; LaToza and Van Der Hoek, 2015; Khan et al., 2021), especially user experience collection, there was no such work on the application of crowdsourcing in user experience collection for any Malayalam mobile application and this led to the work consolidated in this paper. Malayalam is an ancient Dravidian language, native of the state of Kerala (NIC for Government of Kerala, 2021b [online] https://www.kerala.gov.in by Naitional Informatics Centre (NIC) for Government of Kerala, access date 15/08/2021), India.
2. Methodology
The problem was viewed from a Software Engineering perspective. Requirements Engineering, the first phase of the Software Lifecycle model was considered.
Step 1: Exhaustive literature study was conducted on the application of Crowdsourcing in Software Engineering. The purpose of this step was to narrow down the focus to a specific area and specific application types.
Step 2: From the results of the study in Step 1, the focus was shifted to a more specific domain in Software Engineering. A detailed literature review was done on the thus revealed focus area and application type. From the results of this study, a clearer view of the need for an empirical study using Crowdsourcing in User Experience collection was obtained.
Step 3: Crowdsourced user experience collection of Malayalam Mobile Applications was done in two steps. A crowdsourcing questionnaire was prepared, crowd members identified, the questionnaire distributed and feedback collected from Crowdmembers. Applications considered were AarogyaSetu and BevQ Malayalam mobile applications. Result analysis was done.
Step 4: Crowdsourcing questionnaire was prepared, crowd members identified, questionnaire distributed and feedback collected from Crowd members for AarogyaSetu and Manglish applications. Result analysis was done.
Step 5: Consolidation of User Experience feedback was done, based on the results of both studies.
Step 6: Analysis of consolidated feedback was done and useful insights were obtained.
3. Methodology implementation
3.1 Literature review on the application of crowdsourcing in software engineering
A literature study was conducted on 30 publications on the topic. A tabular consolidation of the 12 most relevant papers on Crowdsourcing for Requirements-related aspects of Software Projects is given in Tables 1–3.
After collecting the details of work and challenges identified in applying Crowdsourcing to Software Engineering, the word cloud service of MonkeyLearn (MonkeyLearn Team, 2021[online] https://monkeylearn.com/word-cloud/access date 15/09/2021) was used to find out the most preferred work as well as the most referred challenges in the relevant published work in this area. The word with the maximum occurrence and significance was considered to be the most pivotal. The Monkey Learn plot of concerns addressed in the published literature on applying Crowdsourcing to Software Engineering is shown in Plate 1.
From the plot of concerns addressed in the published literature on applying Crowdsourcing to Software Engineering as in Plate 1, it is evident that the majority of the study was on applying Crowdsourcing to Software Requirements/Requirements Engineering.
MonkeyLearn plot of challenges in the context was done next to find existing problems that need to be addressed in implementing Crowdsourcing in Software Engineering. The plot is as in Plate 2 below:
The plot on challenges uncovered during various studies indicates that the challenge majority of the researchers faced in using Crowdsourcing with Software Engineering are collecting feedback from the Crowd and aligning/sequencing tasks or the process of Crowdsourcing.
With the vision obtained from the above-explained Literature review, the focus was narrowed down to Software Requirements Engineering. In the second step, a detailed literature review was conducted on the application of Crowdsourcing in the User experience collection of mobile applications. Also, a blunt search for such work with Malayalam mobile applications was done.
3.2 Literature review on the application of crowdsourcing in software requirements engineering/management
Step 2 was the study of literature on state-of-the-artwork application Crowdsourcing to Software Requirements Engineering/Management. The focus was narrowed down to software requirements based on results from the literature review detailed above. The scenario considered was that of mobile applications. This choice was made since a majority of such work was done with mobile applications. A sneak peek was also done at the work in this concerning Malayalam mobile applications. The consolidation of major work done in this area and relevant aspects are listed in Tables 4–6 below:
The column “Aspects covered” in Tables 1–6 above was plotted using MonkeyLearn. This was done to find the most significant/most occurring term in the set and this represents the concern addressed by the majority of published work in this area.
Plate 3 gives a clear indication that the focus was on the user and then on a prototype. From the literature, language was not a factor anywhere because a majority of the work we came across was in English. If at all a very few in other languages, they were negligible and didn’t have any remarkable contributions. The consolidation contained work in English Language only. A trace of Malayalam could not be found, however, till 2021.
Next, the MonkeyLearn plot was done in the “Challenges Mentioned” column of Table 4–6. This column contained challenges listed in existing works on the use of Crowdsourcing with User Experience collection of mobile applications. Plate 4 depicts the research work carried out in this regard.
Plate 4 is the plot of the challenges. The majority’s concern was related to crowd members and the interface given to the Crowdmembers for working with the request and contributing.
3.3 The outcome of the initial literature studies
Two sets of literature studies were conducted, one on Crowdsourcing for requirements-related aspects of Software Engineering and the other on the use of crowdsourcing in User Experience collection of mobile apps. Majority of the work across the software lifecycle phases focused on requirements and testing. In the process of the literature study, we came across the use of crowdsourcing relating to different aspects of a software project. Interestingly, we found that we never came across such attempts made on mobile applications in the Malayalam Language. From the initial two literature reviews conducted followed by the result analysis, we could get a clear understanding that a qualitative case study on this would be highly useful and necessary. This would aid and guide future research in this area.
With the insights from the literature explorations, the task of user experience collection of three Malayalam mobile applications, Aarogya Setu, BevQ and Manglish was done. First was a study of the user experience collection of two Malayalam mobile apps, Aarogya Setu and BevQ. Second was a study of the user experience collection of two Malayalam mobile apps, Aarogya Setu and Manglish (NIC for Government of Kerala, 2021b [online] https://www.kerala.gov.in by NIC for Government of Kerala, access date 15/08/2021, AarogyaSetu [online] https://www.aarogyasetu.gov.in by NIC for Government of India, access date 15/09/2021, NIC, 2021 [online] https://bevco.in/ Kerala State Beverages Corporation Ltd., Govt. Of Kerala, access date 15/09/2021, Clusterdev, 2021 [online] http://manglish.app/onlineby clusterdev, access date 15/08/2021).
3.4 About the applications used for user experience collection
Aarogya Setu is an application that is used in around twelve Indian languages including Malayalam. Our focus was on the Malayalam Aarogya Setu application – BevQ is a mobile application for token booking in the virtual queue of Beverages Corporation. The Manglish mobile application is used to key-in Malayalam words in English and get the Malayalam language notation equivalent of that. It is used widely with social media applications. The two studies were conducted by preparing a questionnaire, distributing it to the crowd and collecting feedback from Crowdmembers (the “crowd”). A comparison of crowdsourcing feedback was done between the two studies and conclusions were arrived at on what needs to be considered as the most influential factors in designing applications and interfaces for user experience collection of Malayalam mobile applications. What the Crowd expects from the service/work requester’s side was also uncovered. This can be considered a factor in attracting the crowd.
Details of the three applications used for the study are as given in Table 7 below.
The first study was conducted with Aarogya Setu and BevQ applications. A questionnaire with mixed question types – yes/no, choice and think-and-answer by selfwere distributed using the survey website of SurveySparrow. Many people viewed the questionnaire, but only very few attempted it and even fewer completed the task of answering the questionnaire completely.
3.5 Implementing the study using AarogyaSetu and BevQ (study I)
A Questionnaire was prepared after studying similar questionnaires for the purpose (Roy and Ganguli, 2008; Hao et al., 2016; Díaz-Oreiro et al., 2019) and this questionnaire was distributed to the Crowd. The Crowd we used here was immediate friends and friend groups who could be possible contributors. We requested them to pass it on to their known people who could be potential contributors. Table 8 below consolidates the results of this study.
3.5.1 Observations from the study
A questionnaire was prepared with 16 questions. The questionnaire consisted of think and answer type of questions. Very few were choices. Study 1 was viewed by 127 people, but only 34 people attempted the questions. Out of these 34, only 9 completed answering all the questions, i.e. 26.47%. The average time taken by these 9 people to complete the questionnaire was 13 min and 12 s, obviously not appreciable for use by experts and genuine users of the applications. This statistic is presented in Plate 5 below.
3.6 Implementing the study using AarogyaSetu and manglish (study II)
The methodology adopted was the same as that of study I, except for the change in strategy that there were no think-and-answer, text-type questions. The answers to all questions had to be just chosen from a list of choices. This decision was made based on the analysis of results from Study I. In Study II, many potential contributors viewed the questionnaire as compared to the study and more people attempted and completed the questionnaire. The consolidation of excerpts from the study is as in Table 9 and 10 below:
Plates 8–12 depicts the graphical representation of different observations from the study applying Crowdsourcing to the User Experience collection of Malayalam mobile applications (Study II). Possible reasons are also listed.
4. Discussion
Findings of Crowdsourced User Experience collection of Malayalam mobile applications AarogyaSetu and BevQ were conducted as the first study. From the observations of the study, improvements were made on questions, question types and other interactive items produced to the Crowd members for giving feedback. A second study was conducted by applying Crowdsourcing to the User Experience collection of the Malayalam mobile applications, AarogyaSetu and Manglish. A comparison of a few serious concerns between the two studies is presented in Table 11 below. All observed facts and possible reasons for the same are also listed thereof.
5. Conclusions
The first two literature reviews drilled down to the significance and necessity for a case study on the usage of Crowdsourcing in usability (in terms of User Experience) collection of mobile applications used by people in a specific cultural background. From the Literature reviews, an analysis of concerns and challenges uncovered this need. It also indicated that crowdsourcing can be used as an effective mechanism for collecting interested, skilled and experienced people’s evaluations of mobile application usage experience. From the two case studies conducted, many interesting conclusions were arrived at. Factors which attract the crowd were absent other than an obligation for a few. Many conclusions could be arrived at relating to how to design the questionnaire, how the questionnaire or the evaluation item could reach maximum Crowd, the necessity of keeping optimal control over the Crowdsourcing process, etc. where the most prominent ones. A few are listed below:
Platform used a hierarchical reach mechanism and Internet reach would have given more and hence quality-improved results. Design – Design the feedback mechanism in such a way that the user interface and choices are unambiguous and distinct.
The controls used in the interface shall also provide ease of use.
Rewards – People work either because of compulsion or motivation. To attract stakeholders or nonstakeholders external to the system, a rewarding system must be included. Identify the most influential factors. Control – With hierarchy levels, controls may go loose. There were many visitors, but very few attempted and even few completed in case of Study I. Schedule – Keep a process in place to make the flow systematic- plan milestones and deliverables..
6. Future work
When reward is involved/time is too short/anonymity is not maintained and the crowd is obliged to the requester, there is a greater possibility that the textual expression we receive regarding the User Experience will not be close to the truth. Emotions in a text can be an indication of the sanctity and dependability of User Experience collected using Crowdsourcing. One of the most important future directions in this research is adding credibility and value to the User Experience (data) collected by giving weight to assessing emotional correctness and dependability.
Malathi Sivasankara Pillai is an Assistant Professor in the Department of Computer Applications, Cochin University of Science and Technology, Kochi, Kerala, India. She is an Master of Computer Applications (MCA) and an MTech in Software Engineering from reputed Universities. Her areas of research interest are software engineering, software quality assurance, software project management, education and speech, and audio processing.
Dr Kannan Balakrishnan is an Emeritus Professor in the Department of Computer Applications, Cochin University of Science and Technology, Kochi, Kerala, India. He holds MSc in Mathematics, MPhil, MTech in Computer Science and PhD from reputed Universities. He has produced many PhDs under his guideship and has many international publications to his credit. He is also a journal peer review team member and a book author. Dr Kannan’s areas of research interest include but are not limited to graph networks, graph networks in computer applications, artificial intelligence, neural networks, deep learning and intelligent computing.
