Effects of a Gamified Software Intervention on Mathematics Scores Among Jamaican Sixth-Grade Students Free

The unsatisfactory performance of students in mathematics in the Jamaican educational system has been a cause for concern for both government and its stakeholders inside the public and private sectors. This is a major concern that requires urgent attention with the advent of science, technology, engineering and medicine (STEM) becoming a driving force for entering emerging job markets. In Jamaica, the sentiment is shared that an insufficient number of persons are equipped with the proper skills and understanding required to function effectively in life after school. In effect, many are unable to apply the mathematics skills they learned in a meaningful way. This poor attitude is also evident among many students as some are of the view that mathematics will be of little use to them outside of school (Thompson, 2017).

It is also true that in today’s society, computers and other gaming systems have fulfilled human needs that the real world is unable to satisfy (McGonigal, 2011). This is because, in the gaming world, you are able to score rewards, whereas you would not usually earn these in the real-world setting. The integration of technology can be good or bad which is dependent on the reader’s point of view (Adimabua, 2015). Today, we are living in a world that is moving at a fast pace where change is becoming accelerated in incomprehensible ways. Research in the field of education is not static, as the desire for new and improved methods of learning means that research is always evolving (Schaaf & Mohan, 2014).

There has also been a major shift in the field of education where teachers are moving from more traditional methods of learning to a more 21st century style of teaching, where the instruction is more learner centered and where the learner plays a more active role (Garris, Ahlers, & Driskell, 2002). Children in this era have become accustomed to tablets and other electronic devices. There is, however, no evidence that such familiarity has any bearing on how students perform academically if their curriculum should be converted to game format on its own.

We now live in a “gamified” world where some classrooms are redesigned to facilitate an online environment as a way to keep students engaged and interested in the course content. The process of gamification is one such method. While there has hardly been any academic attempt at a formal definition, Kapp (2012) defined gamification as a process whereby one uses “game-based mechanics, aesthetics and game thinking to engage people, motivate action, promote learning, and solve problems” (p. 10). An argument can be made, however, that “gamification” goes much further than that description. Gamification can be looked at as an approach used by instructors to facilitate learning using gaming elements that force students to think. Marczewski (2013) defines gamification as “the application of gaming metaphors in nongame contexts to influence behavior, improve motivation and enhance engagement” (p. 21).

While there have been many discussions from scholars on the effects of gaming systems and how such systems affect a student’s academic performance, there has been little empirical research in the Caribbean to substantiate this claim. Technology is not a “one-size-fits-all” situation and can be very costly to implement. No research evidence has been conducted to substantiate the claim that special learning benefits can be gained from one specific medium over another when delivering instruction to students (Clark, 2012). Clark famously declared that the media are mere vehicles that deliver instruction but have no bearing on student achievement no more than a truck that delivers our groceries has any bearing on our nutrition.

The issue of worsening math scores has plagued the Jamaican education system for several years at both the primary and secondary levels. This current research was designed to determine the effects of a gamified software intervention in mathematics achievement among sixth-grade students in a small inner-city elementary school. For 2017, it was reported by the Jamaica Information Service that the average score for mathematics (Grade Six Achievement Test) was 62.4%, while the scores since 2016 stood at 58.2% (Smith-Edwards, 2016). The pass rates since 2013 stood at 61%, 60% in 2014 and 56% in 2015. While the 62.4% pass rate was an increase compared to 2014-2015, the pass rate is still low. Against those statistics, the low pass rate is still a major concern for both administrators and educators alike.

The present study provides empirical data on the effects of the implementation of a gaming system as an experiment and also provides recommendations on how such a system could be implemented in the general Jamaican school system. While there are numerous studies that investigate the impact of gamification in higher education and high school settings, this current research focused on the impact of gamification in a primary middle school school setting in a Caribbean third-world context. By applying gamification in the classroom, students could be motivated to learn new ways, or they could enjoy tasks that they once found difficult. The final result discussed the relationship between motivation and its impact on student scores in an online gamified learning intervention experiment.

The study sought to answer two research questions:

1. How has implementation of gamification systems in the classroom improved mathematics scores of K–6 grade students?

2. What recommendations would you give on how to design the mathematics curriculum in the future to increase student interest and scores in math using technology?

The theory that was used to guide this research is the self-determination theory of motivation (SDT) (Ryan & Deci, 2017). The SDT is a theory of motivation, human behavior, and development. This theory of motivation was used to explain how students are motivated by using gamification to improve their math scores. The SDT theory is focused primarily on varying types of motivation that were subsumed as part of this theoretical framework. In addition, the theory is also concerned with “how sociocontextual factors support or thwart people’s living through the satisfaction of their basic psychological needs for competence, relatedness and autonomy” (Ryan & Deci, 2017, p. 3). The SDT over the years has been refined and discussed by many scholars from different institutions (Ryan & Deci, 2017). The hypothesis is that human beings in general require three basic needs: competence, autonomy, and relatedness (Ryan & Deci, 2017).

The theory further explains that it is in our innate nature as human beings to attain greater degrees of healthy psychological, social, and behavioral functioning. This allows us to realize our natural talents. This information elucidates what humans need from being in their psychological and social environments in order to be fully functional in order to succeed (Ryan & Deci, 2017). A similar approach was utilized for the study at hand to gather information on participants.

The primary use of SDT theory within this study was to help interpret why students cognitively behave the way they do, relating the findings to using games to improve math performance. The premise of the study is that the use of gaming technology in the classroom will increase intrinsic motivation of the students to learn, and by extension, their math performance will significantly increase. The SDT theory applies both to the learner’s motivation and their intention to learn using technology (Fathali & Okada, 2017). The researcher examined students’ individual ability to choose how to satisfy their individual needs, and further investigate their actions in the classroom that requires some degree of self-regulation. Therefore, the use of technology as a nontraditional learning method is described by SDT as an out-of-class learning method (Ryan & Deci, 2017). This study investigates how well motivational factors described by SDT applies to and explains sixth-grade student learners’ intention to use technology and gamification in their classroom to improve their math scores.

Gamification as a concept was coined in 2002 by Nick Felling (Marczewski, 2013). However, the term was first used publicly in 2008, and since then it has been used on numerous occasions (Scepanovic, Zaric, & Matijevic, 2015). Gamification is also used to explain the integration of gaming elements, frameworks and mechanics into nongame scenarios (Johnson, 2014). Gamification is usually used in game contexts to engage people and to solve problems. It is argued that with gamified systems in the classroom, this can increase the participation and the motivation of the learner (Sahin & Namli, 2016). Gamification should also be differentiated from game-based learning because gamification takes the entire learning process and turns it into a game (Al-Azawi, Al-Faliti, & Al-Blushi, 2016).

Mathematics is a subject area where technology has not been used to its full potential. Kapp (2012) noted that each activity involved in the game must have an intrinsic goal (such as learning to solve a problem) and extrinsic elements (such as points and rewards) with a clear end point in mind, designed to elicit a specific outcome (such as perform better on a specific skill). One of the main elements and features of video games and other types of games is the frequency and intensity of the feedback provided from games. This feedback is oftentimes provided in real time where the player is informed of his/her progress throughout the game. There are also strict rules inherent in most games. Marczewski (2013) noted that rules are a vital component of any game. No matter the mechanics of the game, there will always be rules that must be adhered to.

Schaaf and Mohan (2014) noted that a properly designed game can give players an opportunity to experience intrinsic rewards at different levels of play. This is more likely to occur when students apply enough effort and apply the problem-solving skills that they learned in order to elevate to the next level on the leaderboard. If done effectively, the impact of gamification can be used to align the interest of the designer with the interest of and how motivated the players are (Buckley & Doyle, 2014). Garris et al. (2002) views the gaming cycle as an interactive process whereby players are allowed to repeat their play without judgment and make corrections based on the feedback. Feedback in this aspect is important because it informs the player/learner if he/she did the right thing or used the wrong approach and suggests different ways how they can correct it without stating the answer. A major advantage of this feature is that, in the gaming world, feedback is immediate.

This may occur as soon as you have completed a level and notified (Marczewski, 2013). The way to reap the benefits of games is to view games not as mere tools and accessories for teaching but rather artifacts and content to be studied whereby you can learn something (Whitton, 2012).

Is gamification a “now” hype, or a system that can be commercially viable for many years to come? In discussing the capabilities of gamification in improving math scores and student motivation to learn math, it is imperative to examine Gartner’s hype cycle for emerging technologies. The Gartner hype cycle tracks technologies overtime and see how they evolve or not. The chart describes how technology progresses with the “hype” of having the newest product (over enthusiasm) to be disinterested or a period of disillusionment (Gopaladesikan, 2012) (see Figure 1). In the first stage the “technology trigger” stage, a potential technology is introduced which comes with the hype in the market. After this, you may get to the stage called the “peak of inflated expectations.” At this stage according to Gartner, the technology can either produce a high level of success accompanied by a host of failures as the expectations were not met. At the stage of “trough of disillusionment,” issues usually arise where the technology is overemphasized and pushed beyond its limits as the technology does not live up to the hype. This then leads to the “slope of enlightenment,” where the technology will either live up to its name or the relaunching of second or third generation phases of the product. The final stage, the “plateau of productivity,” is when the technology “takes off” in the industry and is adopted in the mainstream business (Gopal-adesikan, 2012). Sandusky (2015), however, surmises that “gamification in education and e-learning is still considered as an emerging technologyThis is the opposite of what the Gartner hype cycle suggests. So, based on this, will the “hype” of gamification live up to its name?

Gamification is at the stage of Peak of Inflated Expectations, based on the 2012 version of Gartner’s hype cycle. This is because, some businesses have also adopted using gamification and gamelike elements in their training and development exercises (Marczewski, 2013). In November 2012, Gartner had predicted that by the year 2015, at least 40% of global organizations will adopt gamification as their primary mechanism of conducting business (Kumar, 2013). However, it was also reported that by the year 2014 at least 80% of gamified applications would fail to meet the “hype” (Fenn & Raskino, 2008). This, Gartner surmises, will be mainly due to poor design and the added focus on badges, rewards and leaderboards. Gopal-adesikan (2012) reported that gamification had done its time at the peak stage and was on its way to the Trough of Disillusionment.

Lane Mout Primary school was selected for the study because of its unique characteristics. The school is located in an Eastern Kingston inner-city community on the border of Kingston and St. Andrew. In 2014, Lane Mout Primary invested in software called Edufocal that would allow the students to adequately prepare for their exams.

The Edufocal gaming software was developed in 2010 as an online learning community focusing on using technology to enhance the learning experience outside of the classroom. The learning software has over 15,000 preparatory questions for the GSAT exam that is presented in a manner that is akin to the traditional role-playing game. When students answer each question correctly, they are rewarded with points and medals which allows them to “level up” in the game. The higher the ranking, the more difficult the questions become. The more answers you get correct, the more rewards you are able to unlock, and the more prizes you will win (Edufocal, 2017).

The questions for the software were created and put together by qualified educators in the field ranging from different subject areas such as social studies, mathematics, English language, and communication studies, which are the main subjects taken in GSAT. The software also includes questions from past papers where students are able to work solution for hundreds of questions. This research only focused on the math component of the program.

The researcher randomly assigned students in two classes based on their respective grades that they receive from the pretest (diagnostic math test) at the beginning of the school year (September 2017). This diagnostic test was used as the pretreatment. Grade 6 was purposefully selected because, at that stage in Jamaica, students are preparing to sit the GSAT exam. From the scores, students were randomly placed in two Grade 6 classes. From this, one grade was exposed to the gamified software treatment, while the other class was used as the control group and received traditional math lesson. There were four classes in total. The treatment group was given complete access to the Edufocal gamification software, while the other classes (control group) did not have access (Table 1).

The Diagnostic Test for Entry to Grade 6 test is given at the beginning of the semester (usually in September) to all students entering Grade 6 at the institution. The GSAT is an exam given to students in Grade 6 and the results are used to evaluate the performance of students to determine their suitability for high school placement. Midterm and mock exam test scores were also collected.

Two groups were used for the experimental study: an experimental group (using a gaming system) and a control group (taught via traditional methods). This study sought to compare the math scores of students who have used gaming applications (experimental group) and those who have not used the application (control group) to examine if there have been any changes in their math scores All students that participated in the study were from Grade 6 and preparing to sit the GSAT exam in March 2018.

The treatment for this study involved a set of mathematical instructional games created using the GSAT curriculum. The procedure for the experimental group was as follows; (a) math motivation and attitude toward math survey to identify student attitude toward math (A1) (b) school district benchmark diagnostic test (pretest) (A2) used to place students in each group/ class (note, the treatment for this group is traditional classroom instruction throughout), (c) intervention of the treatment (Z) (A3), (d) GSAT final exam (posttest) (A4) for math assessment. To measure the effect of the treatment Z on the math achievement and motivation of the group toward using technology to improve math scores, the researcher compared the test results of the experimental group at the beginning of the study and the end of the study to the control group. A similar set of tests measuring the same six strands used at the beginning of the study was also administered at the end when the posttest exam is taken (A1+A4) (see Table 2).

Data were analyzed using STATA. Multiple regression test was analyzed using the raw test scores. With this test, the outcomes were observed for two groups over specific time periods. In this situation, one of the groups was exposed to a treatment during the time period. In this study, the treatment (Z) is using Edufocal. The second group (control group) was not exposed to the treatment during the time period. The researcher calculated the mean outcome in group A in both periods and calculated their difference. The researcher then obtained the mean scores from group B in both the pre and post.

Null. There is no significant difference in math test scores of the diagnostic and final GSAT test for students who used the gamified software compared to those who did not.

Directional. Students exposed to the gamified intervention had a significantly higher gain score in standardized tests than students in the control group.

The researcher examined the effectiveness of a gamified software intervention in mathematics achievement among sixthgrade students in a selected elementary school. The intervention involved the use of Edufocal gaming software to solve math questions.

The district benchmark pretest was used as the instrument to show equivalency between the experimental and control groups. When the diagnostic scores were analyzed for the experimental group, the results show a SD of 17.74 and a mean score of 53.85. For the control group, the SD score was 16.15, with a mean score of 53.95. Students were divided based on scores and account for equal gender (see Table 3). The district benchmark exam was used to place students in the experimental group and all other students placed in the traditional classes. Each benchmark test consisted of approximately 80 multiple choice questions on average covering all six strands in mathematics; (a) numbers, (b) measurement, (c) geometry, (d) statistics, (e) algebra, and (f) probability.

The randomization process was done using STATA to determine the number of each student based on class assignment that was used in the study. The results in Table 2 (4) shows a p value of 0.982 for diagnostic score which was not significant based on the significance value of .05. Upon analyzing the scores from students in the treatment group at the end of the fifth grade, it gave a mean value of 53.94 and a SD value of 12.13. In the control group on the other hand, the mean was not significant at 58.18 mean with a SD score of 11.48

The data were student scores that were given using standard diagnostic tests, end-of-term exam, mock exams, and GSAT final exams. When the diagnostic scores were analyzed for the experimental group, the results show a SD of 17.74 and a mean score of 53.85. For the control group, the SD score was 16.15, with a mean score of 53.95. After the final analysis, the results show a p value of 0.982, which was not significant based on a value of 0.05. Upon analyzing the scores from students in the treatment group at the end of the fifth grade, it gave a mean value of 53.94 and a SD value of 12.13. In the control group on the other hand, the mean was still not significant at 58.18, with a SD score of 11.48, as represented in Table 4.

The regression analysis model was used as the test which includes a treatment dummy variable and compares the average outcome of treated individuals to the average outcome of untreated individuals. The coefficient on the treatment dummy in this study tells us the average difference in treated students’ math performance relative to the average math performance of untreated students. As such, the coefficient on the treatment dummy is equivalent to the difference in means from a standard t test when students are randomly assigned to treatment. A major advantage of the regression model over the standard t test is that we are able to control for any other variables that could potentially affect a student’s math scores other than the intervention of the software.

Table 5 shows the impact of the technology intervention on students academic performance on three separate exams (pre, during, and post). The results suggest that in the December exam, treated students earned a score of (p > 0.91) percentage higher than students that were not exposed to the technology in the December final examination. Similarly, students earned a score that was 5.39 and 1.35 higher in their February Mock Exam and the standardized external GSAT mathematics examinations, respectively. While these results indicate that the intervention students scores were positively affected, the estimated impact is not statistically significant at the conventional 5% and 10% significant levels. We failed to reject the null hypothesis at the .05 level and, hence, the intervention did not statistically improve students’ performance in the short or longterm. The R² across the three models suggest that about 48 to 67% of the variation in student test score on these exams can be explained by their intervention treatment status, Grade 5 test scores, and their Grade 6 diagnostic scores. As such, these variables explain a large amount of the variation in students performance in mathematics.

Ke (2009) argued that with the recent bombardment of gaming systems in the classroom, it has not met the anticipated potential it carries as a motivation and learning tool. The study concluded that there exists an overall significant impact of technology on student achievement, attitude and motivation toward math, but the results vary based on the intervention method used.

Technological teaching aids can help to enhance students’ ability, which will eventually increase their confidence in learning a subject. Turgut and Temur (2017) inferred that the use of gamelike elements to teach mathematics can have a positive effect on how students perform academically. Using any type of technology to learn can be less effective or ineffective when the learning objectives are unclear and the focus is on the use of the technology itself and not the curriculum. To effectively implement technology in the classroom, instructional technologists should pay closer attention to the learner, the learning environment, professional competency, system capacity, community connections, technology capacity, and accountability. Once these factors are taken into consideration, then the technology can be used as an effective tool to enhance the learning experience (Schacter, 1999).

The analysis also concluded that technology does not play a role in facilitating student learning. The argument is that the inclusion of technology should not serve as a replacement to the traditional face-to-face curriculum but rather an enhancement to the learning process. Buckley and Doyle (2014) found that gamified learning intervention had a positive effect on student learning. However, the effect varies between each student, such as their level of motivation and willingness to participate. Clark’s (2012) perspective is important in this debate because, no matter how many media tools the instructor uses and invests in, it is the contribution of the material/content, the methods used, and the way in which it is designed that causes a student to learn. In addition to this, the learners must feel confident that they can be successful in the subject. If students feel that they can achieve success in a particular game, then they will feel more motivated to succeed in that game (Kapp, 2012).

The results showed that overall, the intervention did not statistically improve students performance in mathematics whether in the short or long term. One possible explanation could be that the tests at the end of the year became repetitive and thus students were regurgitating what they already learned from the first time they used the software and thus became disengaged after awhile. It could be that after an entire semester, students were already exposed to all the questions and were familiar with the answers and no new questions were introduced that challenged their learning abilities. This was especially evident when it was closer to the exam in March 2018. In order for the game to be effective in the learning process, it should present at minimum an optimal level of difficulty in tasks and its interaction with the learner (Chen, Yeh, & Chang, 2016).

Three limitations were identified:

1. To strengthen the validity of the study, one elementary school located in an inner-city community was used.

2. The study used only sixth-grade math scores.

3. Current research gap was limited to the understanding of the potential nature of gamification as a learning tool to improve student math scores.

A mixed method perspective could be considered where focus groups and interviews are included along with the traditional quantitative method of data collection. This strategy would allow researchers to get an opportunity to speak face-to-face with users and ask the “why” questions. This will include conducting research involving administrators, instructors, computer personnel, math teachers and everyone involved in the process including the maker/owner of the software.

Research should also focus more on how to improve student academic performance rather than what method is best to use. Gamification and its benefits should not be used only as a control mechanism; however, it should serve the task of motivating and engaging the users (Marczewski, 2013). In further discussing what teaching strategies and media to be used, Simonson, Smaldino, Albright, and Zvacek (2012, p. 135) points out that “it is important to utilize students in this process as students can seek to provide insight into the design of the learning experience.”

A longer time period with different geopolitical educational boundaries/districts could be implemented at different time periods. This would include implementing the software in at least one school in all seven districts focusing on low performing schools. This would include a view to ensuring a more proportionate distribution of the island population rather than focusing on only one school.

A new study should be conducted examining one or several other variables, including parent’s educational background and social class. When studies that look at achievement outcomes are designed, there are other variables that could be included in the study. It was also recommended that when using interventions (i.e., computer games), that those interventions focus more on teaching math geared toward evidence-based instructional strategies. A new study should also be conducted related to using technology to do math and its application to daily life as compared to other subjects.

The researcher used standardized test scores to measure students’ achievement across three full terms of study. A survey was also used to collect data at two different periods to examine student attitudes toward math and student attitude toward using technology to learn math. A pretest and posttest gain score analysis model was used to measure student progress throughout the three school terms. The data were analyzed using mean scores to determine if any significant difference exists between the percentage of student test scores as explained by their treatment status, diagnostic score and final GSAT scores. The results indicate that the intervention students scores were negatively affected and the estimated impact was not statistically significant, as we failed to reject the null hypothesis at the .05 level. As such, the intervention did not statistically improve students performance in the short or long term. The self-determination theoretical framework was used to study the effects of gamified intervention on students test scores and by extension, effect of student motivation when using technology to learn math.