Applying Deep Learning to a Sign Language Progress Monitoring System

Progress monitoring (PM) is used to measure a student’s academic performance level and determine whether students are making adequate academic progress (e.g., McMaster et al., 2017). In recent years, several computerized PM systems have been developed (e.g., AIMSweb [Shinn & Shinn, 2002], FastBridge [2019], Dynamic Indicators of Basic Early Literacy Skills [Kaminski & Good, 1998], etc.), and; research on PM has increased (cf. Ysseldyke & Tardrew, 2007).

However, many web-based systems are invalid for students with special needs, such as those who are deaf or hard of hearing, as computerized PM systems are often auditory based. To address this issue, we developed AvePM, a PM software suite designed to emphasize students who are deaf or hard of hearing (Hooper et al., 2018). AvePM uses curriculum-based measurement, which includes brief, easy to complete, and produces reliable and valid data that can be used to improve academic outcomes (Deno, 1985; Hosp et al., 2016). Curriculum-based measurement has been used successfully in education for many years (MiuraWayman et al., 2007; Shinn, 1989).

This article describes our attempts to automate scoring of an app that is part of the AvePM software suite. The suite (http://avepm.com) includes eight measures of student performance addressing tasks such as emergent and early reading, American Sign Language (ASL), and English development, as well as more traditional reading and writing measures. Two of the eight measures are scored by computer algorithms, but the other six are scored manually using custom tools developed for each measure.

One of the measures in the AvePM suite is named “PictureNaming.” Picture naming is a commonly used progress-monitoring task that measures students’ ability to name pictures—an essential first step in the reading process. PictureNaming was designed to accommodate students who are deaf or hard of hearing. The app presents randomly selected images, and students sign or speak the image names; a video recording is stored online and made available for manual scoring. The tool takes video input, thereby allowing students to record responses in sign language.

Although some PM tasks (e.g., cloze tests) can be scored reliably by a computer (Fuchs et al., 1992; Miller et al., 2008), to date, tasks recorded using the PictureNaming app have been scored manually rather than by machines due to the complexity of scoring ASL. In the current project, we developed a deep-learning algorithm to auto-score picture naming. Our goal was to provide immediate feedback for students and transform the scoring and subsequent data analysis processes for teachers.

Deep learning, a subset of machine learning in artificial intelligence, is used for image classification, object detection, and face recognition. The term “deep” in deep learning refers to the number of layers that transform the data. These layers extract high-level features from the raw input. Features are the variables or attributes of a dataset, such as grayscale values of pixels, edges, objects, and points. For example, when classifying a person as a male or a female, the features might include height, weight, body mass index, hair length, speech pitch, and so forth.

The term neural network is used widely in deep learning. Neural networks are computing systems designed for pattern recognition. A neural network is a collection of connected units known as neurons. Computer neurons are similar to biological neurons: connections between neurons are like synapses that transmit signals between neurons in the brain. There are many types of neural networks. Each functions according to the task it must perform, such as text, speech, or image processing.

A neural network consists of many layers. The first layer, the input layer, takes features of an object as input; the last layer, the output layer, classifies or recognizes the object. All other layers (known as hidden layers) process the input data (which is in the form of features) to produce some form of predication. Recent developments in deep learning have greatly improved the accuracy and precision of visual object recognition (LeCun et al., 2015). The current project applies deep learning to ASL recognition.

Advancements in deep-learning technology and the availability of computational power have resulted in considerable interest in video and image recognition. For example, algorithms have been developed to train machines to differentiate between cats and dogs (Branson et al., 2014), path tracking for automated selfdriving cars (Paden et al., 2016), and so forth. In each case, an effective recognition model depends on the generation of a dataset aligned with the problem being solved.

However, little research has been conducted on sign language recognition (Chai et al., 2013; Zafrulla et al., 2011); thus, datasets need to be constructed to build an effective recognition model. Prior work has used the Microsoft Kinect sensor to record videos that segregate features such as hand color, shape, and size and use pixel values to track body movement (Li, 2012; Ren et al., 2013). A scarcity of data has limited prior research, difficulties associated with identifying human features (e.g., differences in hand shape, size, and color), and limitations associated with developing an effective recognition algorithm.

Our short-term goal was to develop a deep-learning algorithm capable of automatically scoring video recordings and generating useful feedback. Our long-term goal is to integrate the algorithm into the picture naming app, to improve scoring efficiency and reliability and reduce teacher-scoring time.

The current picture-naming tool within the AvePM system includes three components: A student interface that captures videos of students signing the names of images; a manual-scoring tool to view and score the students’ videos; and a data chart displaying student progress over time. To replace the manual scoring tool, we developed DeepSign to auto-score students’ videos. The design of DeepSign occurred in two phases. Initially, we developed a video data collection tool. We then created a recognition model capable of scoring students’ sign language videos and providing immediate feedback. In the following sections, we describe PictureNaming and DeepSign in more detail.

The PictureNaming App measures a student’s ability to sign or say the names of images. We identified 121 images to use in the measure. Images were selected based on their relevance in sign language or their importance in developing students’ early knowledge.

The PictureNaming recording process involves using a webcam to capture a video of a user signing the meanings of images presented on a computer. Initially, the student locates his/ her torso appropriately in front of the camera (see Figure 1). Images are presented to the student continuously for one minute, at progressively increasing speeds; initially, one image every 5 seconds, then one image every 4 seconds, and then one per 3 seconds (see Figure 2). Progression through the speed levels is determined by the student demonstrating mastery on three successive trials.

We designed an interface for teachers to view and score completed PictureNaming tasks (Figure 3). After scoring, performance data are added to a database and then to a chart from which students’ progress may be monitored (see Figure 4).

Our first task was to develop a software tool to capture videos of experts signing the meaning of 10 images (see Figure 5 for a description of the steps involved in capturing signed input). The tool displays core ASL vocabulary images (e.g., apple, banana, ice cream, milk, grape, coat, cap, shirt, pant, shoe (see Figure 7). ASL signers signed the images in front of a webcam. Each image was recorded five times to provide sufficient variance for a deep-learning algorithm to recognize similarities and differences within and between images (see Figure 6). The three signers captured 150 sample videos (see Figure 8).

We used the videos captured by the videorecording tool to build a deep-learning model capable of recognizing ASL gestures in real-time. The videos were divided into frames that were converted into grayscale images. These images were then loaded into a convolutional neural network (CNN) to train the model. A CNN is an artificial neural network commonly used in deep learning for image recognition. We chose a CNN over other neural networks as a CNN is effective for recognizing visual objects and is memory efficient. It is also less complex than other deep-learning neural network models (LeCun & Bengio, 1995; Oyedotun & Khashman, 2017).

The model determines the accuracy of student performance from video input. The model uses information about the user’s hand shape, size, and color to create grayscale images (see Figure 9). The algorithm compares the patterns of input data with patterns in the videos of the data set. The model’s accuracy is influenced by the amount of training data supplied to the neural network: the more data supplied, the more accurate the model.

We developed a prototype tool (known as DeepSign) to test the recognition reliability of the model. The tool presents images and records video while a user signs an object depicted in an image. The deep-learning algorithm converts the video into frames that are compared to previously stored videos. Signs are classified as correct when the pixel value of a student’s video is similar to videos stored in the model.

We used 90 of the 150 videos to train the deep learning model and to examine whether the model can recognize ASL vocabulary reliably. The other 60 videos were used to test the accuracy of the recognition model. The images in Figure 10 illustrate how the model is capable of recognizing differences between images. For example, it can recognize small differences between the gestures of apple, ice cream, and milk. At present, the model can recognize 10 ASL gestures. The rate of successful recognition reached 94.5%.

DeepSign is one of the first tools that use deep learning to recognize ASL signs. After all training videos were analyzed and imported to a CNN, the model could classify several ASL signs in real-time. The results of this pilot study suggest the deep learning method has the potential to assess picture-naming tasks reliably in a computerized PM system. The tool will provide students with immediate feedback, help teachers monitor student progress efficiently and effectively, and reduce the time for teachers to correct students’ work.

The pilot study was limited to 10 images from the PictureNaming software. We will extend the recognition capabilities of the software to include all 121 PictureNaming images and embed the deep learning model into the PictureNaming app. Future research will compare the results of auto-scoring with the current manual scoring tool and examine whether automating the scoring/assessment process will speed up the rate at which students learn ASL.

We will refine the deep-learning model to improve scoring accuracy, efficiency, and reliability. Data plays an important role in machine learning. We anticipate the model will continue to improve as additional data are added to the system. Also, we will change the image recognition algorithm from the histogram approach to the transfer learning algorithm to reduce the influence of variables such as changing light conditions and users’ physical differences (e.g., hand size, shape, and color).

Acknowledgments: The contents of this report were developed under a grant from the U.S. Department of Education, #H327S170012. However, those contents do not necessarily represent the U.S. Department of Education policy, and you should not assume endorsement by the federal government (Project Officer, Terry L. Jackson). Special thanks to Dr. Susan Rose and Alicia Hart and the ASL signers who helped record the videos for picture naming.