Providing timely and helpful writing feedback for student writers, especially those at the middle and high school level, can present an unwieldy challenge for teachers. Yet, feedback is necessary for students’ growth as writers. There is an increased interest and use of automatic writing programs to provide students with writing feedback. However, little is known about students’ perception of such feedback. This study examined the experiences of an eighth-grade class (n = 21) focusing on the experiences of one case study student representative of the class findings. Findings showed the automated essay scoring (AES) feedback did offer the student revision and editing assistance, yet the student articulated her the need to have teacher feedback to guide her understanding of the AES reports and about aspects of writing not available on the reports. Also addressed are considerations and cautions about using such programs.
Feedback about my writing is helpful usually when I’m done with the first draft or when I’m struggling to start something.
(Dez, eighth-grade writer)
To develop their writing skills, adolescents need ample opportunities to write. Students also need feedback to guide them forward. While the teacher is not and should not be the sole reader of students’ work, teacher feedback can help students advance their writing intentions (Graham & Perin, 2007; Graham et al., 2011). However, regular ongoing teacher feedback is often in short supply due to the significant time involved in providing feedback to each student, especially the types of quality feedback necessary for writing growth (Wilson et al., 2014; Yagelski, 1995). This becomes more challenging considering middle and high school teachers often work with upwards of a hundred students daily.
To help teachers lessen this burden, it is important to explore potential resources available for feedback. Peer feedback as a practice improves students’ writing from first to final draft (Graham et al., 2015), though it comes with its own set of challenges. For peer review to be effective students need instruction about what to look for and how to provide feedback to their peers (Philippakos, 2017). In addition, peers’ limited knowledge or expertise can limit the scope of peer feedback.
Computers offer another resource in the form of digital feedback through AES programs, which have become more readily available in recent years via both free and commercial products. Typically, students enter their writing into the program and receive a score accompanied by holistic feedback or feedback broken down into specific traits, offering a hybrid of feedback, not solely revision, but not solely editing either.
AES, too, comes with challenges. The National Council of Teachers of English (NCTE, 2013, 2018) raised concerns about the complexity of writing and computers’ inability to judge writing beyond surface features, such as logic and development of ideas. They also expressed concerns that computers can be “gamed” which may result in assessment of students’ abilities to use “machine-tricking strategies” rather than an assessment of students’ writing abilities. Others have acknowledged similar concerns (Lee et al., 2019). Despite expressed concerns, AES programs have the potential to offer students an immediate source of feedback to assist their writing.
In this study, we explored the use of AES as a resource to complement teacher feedback. We wanted to explore to what extent AES programs could support students in their writing by examining the student perspective and how students utilize information from these programs. There is limited research on how students respond to such feedback (Wang et al., 2020). In this qualitative case study (Merriam, 1988), we examined one student’s experiences with AES feedback to determine what was helpful and actionable for her writing. We guided the study with one main research question and three subquestions: What is one student’s experience with and perceptions of feedback from AES programs as tools for editing and revising her writing? Specifically, which AES programs did she prefer and why? What editing and revisions decisions did she make and why? What, if any, insights about her own writing did she gain from AES feedback?
Literature Review
In this review of the literature, we describe the existing literature in several areas related to our case study. We begin by contextualizing the current research on revision and editing, followed by research on feedback in writing. Next, we examine the research on AES and its potential applications for writing instruction. We conclude our literature review by situating our study within a sociocultural theoretical framework.
Revision and Editing
As less experienced writers, many students have struggled to revise, lacking a clear understanding of how and why to revise (McBeth, 2015; Morgan et al., 2015; Witte, 2013). Instructionally, teachers have not always explicitly taught revision (Yuknis, 2014) or they often taught it in combination with editing. A survey of middle school writers found students did not always know what was expected of them during the revision process and many expressed confusion with revision and editing, using the terms interchangeably (Hrubik & Morgan, 2019).
Fletcher (2017) stressed the difference between revision and editing, describing revision as a process of composing that involves reshaping, adding, cutting, and switching sections around. Revision is an opportunity to “re-see” a piece of writing. Katie Wood Ray (1999) explained that writers revise “not because the drafts are bad and need fixing, but to see what else is possible” (p. 60). Revision is an opportunity to think critically about what the writer has written to ensure ideas are expressed to the reader in a clear and organized manner. Editing, on the other hand, focuses on polishing a piece, attending to the mechanics of language such as grammar, spelling, and punctuation.
Due to its complex nature and higher level skills, students struggle more with revision than with editing. Explicit strategy instruction helps students revise their writing (Graham & Perin, 2007). From an instructional standpoint, teaching revision strategies allows students to consider the clarity of their writing and offer ways to improve their current and future writing.
Explicit instruction helps students to develop the essential skill of revision by teaching students to evaluate writing, discover ways to solve common issues and problems, and to learn about the overall craft of writing (MacArthur, 2019). It is important that students understand revision is not just about finding problems in their writing, but opportunities to expand and clarify their ideas (MacArthur, 2019).
Even when taught revision strategies, students need help looking at their writing critically to use these moves effectively. Students require information and direction in the form of feedback about where to focus their attention during revision. To benefit as writers, students need a firm understanding of what revision entails, along with instructional feedback followed by time to work through the feedback during the writing process.
Feedback
Feedback is a critical component to the development of student writers (Dikli, 2006; Graham et al., 2011; Zumbrunn et al., 2016). Students benefit from having an outside reader provide insights into the clarity and purpose of their piece, helping them to “see new possibilities and options in steps they might take next” (NCTE, 2018). Quality feedback provides students with an idea of the current status of their writing along with actionable ideas of how to move it forward. This is an important instructional support during the writing process (Graham & Perin, 2007; Wilson, 2019); however, there is no consensus on the best timing or frequency of feedback. Feedback is most effective when given in a timely manner and when students are given time to use the feedback to revise their writing (Philippakos, 2017; Wilson, 2019). Despite recognizing the importance of instructional feedback to student writing, teachers often reserve feedback until after the final submission, becoming more of an evaluation of student writing, at which point it does little to encourage further revision and editing.
Wilson and Czik (2016) defined instructional feedback as information provided by an agent (such as the teacher, a peer, or computer) that offers information regarding correctness, incorrectness, and ways to improve. Aside from variations in the feedback agent, feedback also varies by type and level. Feedback types are the way feedback is presented to students. Wilson and Czik (2016) identified three main types of feedback: direct, indirect, and praise. Direct feedback involves the agent making corrections or giving actionable suggestions. This type of feedback, because it offers solutions, is most likely to be implemented by student writers (Patchan et al., 2013). Indirect feedback encourages the students themselves to make decisions about revisions by framing feedback as questions or other thought-provoking comments including opinions or reactions to the writing, without explicitly requiring revisions. Finally, praise is a type of feedback focusing on approval and pointing out positive elements within the writing.
Researchers classify feedback as either higher level or lower level based on the focus of the feedback (Wilson & Czik, 2016). Lower level feedback focuses on seven components of writing skills: spelling, capitalization, punctuation, sentence structure, grammar, formatting, and word choice. Higher level feedback focuses more on content level suggestions including development of ideas and elaboration, organization, style, and feedback pertaining to the students’ writing process or experiences. Similarly, some divide these into revisional or editing feedback (Peterson et al., 2004) where revisional aligns with higher level feedback and editing with lower level feedback. Yagelski (1995) showed teacher feedback may be too focused on lower level features, rather than higher levels which have more impact on improving writing performance. Instead of focusing solely on one level, feedback should stretch across both levels (Wilson, 2019). For the purpose of this study, we refer to lower level feedback leading students to editing tasks and higher level feedback as resulting in revision.
Automated Essay Scoring Programs
While AES programs vary in the types of feedback offered and the delivery of such feedback, they operate similarly, using Natural Language Processing (NLP) tools to extract linguistic, structural, semantic, and rhetorical text features connected to writing quality. Programs use algorithms to assign holistic scores and/or evaluate traits of writing to provide feedback (Roscoe et al., 2017). Exact feedback varies from program to program, though they typically offer constructive, formative feedback on writing qualities such as usage, mechanics, sentence structure, organization, and development. Most programs do not examine content, which is not easily quantifiable.
The intended use of AES is to complement, not replace teacher feedback (Lee et al., 2019: Wilson & Czik, 2016). Despite this intention, some fear AES threatens to replace the teacher as the primary agent of feedback. Another concern among teachers is the possibility that programs might be fooled by the appearance of longer, complex sentences or the use of highlevel vocabulary, providing higher scores even if the writing is lacking in sense or content. These controversies are compounded by the use of AES to assess student writing on standardized tests.
Potential Use of AES Programs
AES programs typically provide low-level and some higher level feedback, thus potentially allowing the teacher to devote more instructional time to the craft of writing such as elaboration and style (Fink, 2016). The benefits of AES are the immediate reach of many students at once (Lee et al., 2019), efficiency, and flexibility (Wilson & Czik, 2016). While there is no consensus on the best timing of instructional feedback, immediacy is often preferable. AES allows students control over feedback timing. Programs are available whenever the student wants feedback, whether in the midst or after completion of a draft. AES makes revisions and edits immediately actionable, thus accelerating the practice-feedback loop (Foltz et al., 2013). The immediacy of automated feedback through essay scores and individualized suggestions offers opportunities for students to improve their writing while also offering assistance to teachers who are generally unable to provide such immediate feedback due to time constraints and the number of students in the class.
Despite controversies over the use of AES as tools for assessing student writing, AES programs have the potential to supplement teachers’ writing instruction. In a study by Wilson and Czik (2016), teachers reported spending about half as much time providing feedback when also using AES programs compared to teachers who were the sole provider of feedback. Further, the amount of feedback provided by both sets of teachers were the same, but the teachers with the added support of AES focused on higher level feedback. They examined content and supporting details more deeply because structural, mechanical, and grammatical writing elements had been identified by AES programs. While no one source or type of feedback should be used exclusively; explicit instruction in revision and editing strategies, peer review, and AES can offer additional support, complementing teacher feedback, thus prompting students to revise and edit more (Wilson & Czik, 2016).
Research about students’ AES experiences indicate students view computer feedback as non judgmental (Shermis et al., 2008). Fink (2016) referred to AES feedback as “a lowstakes process,” where students felt “comfortable experimenting with words, thoughts, and structure” (p. 51). Knowing a computer, rather than a person, was reading their writing, students did not feel that the AES feedback was judging them. Wilson et al. (2014) reported overall improvement in the quality of student writing from first to final draft with repeated in-class exposure to the AES program, PEG Writing (now called MIWrite).
Existing studies have frequently examined AES as tools for diagnosis and assessment of student writing (Shermis et al., 2008; Wilson, 2018). Researchers have begun studying students’ writing when using an AES program (Wilson & Czik, 2016; Wilson et al., 2014). However, missing from much of this research are student voices to examine which programs they prefer and how AES tools assist them as writers.
Theoretical Framework
For this study, we draw upon a sociocultural theoretical perspective to teaching writing, viewing learning as both social and interactive (Vygotsky, 1978). Sociocultural theories emphasize the collaborative nature of learning, acknowledging that learning is inherently social. Students make meaning through dialogic exchanges and through such collaboration, student knowledge develops and matures. This learning often occurs with a more capable other, sometimes peers and often the teacher (Vygotsky, 1978). In writing, this learning can occur through opportunities such as conferring with the teacher, receiving written feedback on their piece, or possibly using AES programs. Scaffolds are an important part of such learning with the ultimate goal being the students’ ability to complete such tasks independently (Wood et al., 1976). Sociocultural approaches to teaching writing emphasize the use of tools to support learning to write; this learning can occur through opportunities such as conferring with the teacher, receiving written feedback on their piece, or possibly using AES programs.
Methodology
In this section, we describe the context and participants, the instructional design, the selection of the case, data collection, and data analysis.
Context and Participants
Jessica is a teacher in a suburban middle school in Northeast Ohio where the student body is 93.9% White, non-Hispanic, 1.5% Hispanic, and 3.5% multiracial with an economically disadvantaged population of 30.1%. The middle school, consisting of approximately 425 students in Grades 5–8, is in its own wing within a PK–8 building. Each grade level has approximately 100–120 students. The district has a history of high achievement on the state report card.
Jessica taught an eighth-grade advanced 65-minute language arts class. All 21 students (10 male, 11 female) agreed to participate in this study. Students in the course are gifted learners or have a history of high achievement via report cards and/or scores on standardized tests. Jessica implements a workshop approach to teaching writing (Atwell, 2015). Denise collaborated with Jessica on the study.
Instructional Design
Prior to the start of data collection, students had just completed via Google Docs their first drafts of a research focused essay on topics of their choice. This was their fourth major writing assignment of the year. At the point of the study, students had completed their first draft and were ready for feedback. For this study, we selected three different AES programs to examine: MIWrite (www.measurementinc.com), ProWriting Aid (www.prowritingaid.com), and PaperRater (www.paperrater.com). Students had previous experience with MIWrite (PEG Writing) as a paid subscription used across grade levels in the school. They had no prior experience with PaperRater or ProWriting Aid. Table 1 lists the types of feedback each program provided, and Table 2 compares the types and levels of feedback offered by each program using the types and levels described by Wilson and Czik (2016).
On Day 1, students completed Google Form Questionnaire 1 to gather their initial thinking about feedback asking about their expectations, preferred types, and when feedback is helpful. They then used the remaining class time to run reports from the three different AES programs, saving each report as a PDF to a shared Google Drive folder.
On Day 2, students analyzed each of the programs using their own AES reports. First the author led this analysis using a chart document comparing the features, examining each program individually with a sample report projected for all students to see, talking through the feedback, and answering any questions from students regarding the meaning or use for the feedback. Then students examined their own feedback and completed the AES Comparison Chart and a reflection (Google Form 2) to identify which programs they found most helpful and how the feedback would inform their next moves as writers.
On the third day, students selected their preferred AES program and used the feedback from their report to revise and edit their drafts. Students documented their use of AES revisions and editing with Google Form 3, which asked questions before, during, and after use of the feedback. The “before” questions identified which program the student chose and why. “During” questions looked at where they began and the moves they made as writers. The “after” questions reflected on the effectiveness of the program in aiding their writing process.
On the last day, students continued to work on their research essays and could work with the same AES program from the previous day, use feedback from another AES program, use their revision strategies in their writer’s notebooks, work with a peer, and/or get assistance from the teacher. At the end of class, students completed a final reflection describing how they spent their class time, writing moves they made, and identifying the types of feedback and information guiding their writing that day. In this reflection, students revisited their first and current drafts to determine the level of changes made and identify work still needing to be done.
Selection of the Case
We examined the data from the entire class to note patterns in their responses to guide our selection of our case. We focused on the perceptions and experiences of one student to gain a more robust understanding of her use of AES for revision and editing. We purposefully selected a case study student who matched the findings from the class data in four ways:
The student’s preferred program was the same as the class majority, PaperRater (n = 12);
The student noted that AES can provide insights about patterns not previously recognized in their writing;
The student noted the limitations in using these AES; and
The student expressed some confusion and uncertainty regarding how to take action with the feedback provided.
From these criteria, we selected Dez as our case study student. In this qualitative case study, we centered on Dez’s thoughts and perceptions of AES feedback.
Data Collection
Data for this study included the following: four Google Form Reflection Questionnaires about students’ writing needs and opinions about the AES feedback provided at various points during the study, three AES reports for Dez’s initial draft, her completed student comparison chart of AES programs (see Appendix A), initial and revised research essay, and a transcribed interview.
The post study interview with Dez occurred after she completed and submitted her research drafts, and she received feedback from Jessica. We designed the interview to obtain further insight on Dez’s opinions regarding each AES program and her overall thoughts about AES feedback in comparison to other feedback agents. Jessica employed Rubin and Rubin’s (2012) responsive interviewing model, which emphasizes interviewer flexibility to change and add questions in reaction to the interviewee’s responses. During the interview, Jessica reviewed each of Dez’s AES reports and talked through what she liked and disliked about each, and how she used or did not use the feedback. In addition, they examined her final paper to discuss her overall changes.
Data Analysis
We imported all questionnaire responses into Google Sheets. We analyzed the data using constant comparative analysis (Strauss & Corbin, 1998). Initially, we analyzed Dez’s data as part of the whole class data and then we reexamined them for closer analysis. We used the class findings as a starting point to focus our analysis of Dez’s data. Initially, these broad codes helped shape our thinking. We looked at examples where Dez identified instances such as confusion, misunderstandings, inaction, and insight through these experiences. We read and reread all data to examine Dez’s thoughts about her AES experiences and to determine what she found helpful and applicable to her writing. Separately, both authors read the transcript and wrote memos about initial noticings using codes to further determine categories and patterns in Dez’s responses regarding her opinions and experiences with feedback. Next, we reviewed our codes and through discussion reached agreement when we had varying codes.
Findings
We selected Dez because she was representative of the major findings from the larger class. We examined her personal preference for one program over the others, what she believed she gained from this feedback, where she expressed confusion and uncertainty about the feedback, and what she felt she needed to guide her writing. We also examined her editing and revision decisions she made from that feedback.
Preferred AES Program
Each of the three programs used in this study offer similar information in varying ways. The reports ranged in length from eight pages from PaperRater to 11 pages for both MIWrite and ProWriting Aid. However, seven pages of the MI Write report was Dez’s essay text marked with codes to guide action. All three programs offer evaluative feedback in the form of a score. For example, when entering the same text (her first draft) in all three programs, Dez received a score of 89 B from PaperRater, a 26/30 from MI Write, and a 65/100 from ProWriting Aid. Beyond evaluation, Table 1 shows the kinds of feedback offered in each program organized by the level of feedback (Wilson & Czik, 2016).
Dez identified all three AES programs as “helpful,” but like her classmates (12 of 21), she preferred PaperRater. Dez liked the overall visual layout and organization of PaperRater and felt it directed her to take action in her writing. Feedback in each section included a color bar scale showing a continuum from red (weakness) to yellow (average) to green (strength) as a visual indicator of progress. Dez liked these tables and graphs which to her indicated “Okay, you’re bad. Okay, you’re good.” She felt this program looked “more appealing’ compared to “a big jumble of words from MI Write” and she was less than pleased with her overall “grade” function from ProWriting Aid where she received a 65/100.
Additionally, Dez liked the way that Paper-Rater provided specific actionable feedback she could apply to her writing. She recalled, “Honestly, PaperRater was probably my favorite. I’m pretty sure out of all the other ones just because it does give examples of what exactly needed to be fixed.” The program not only showed issues but directed Dez to specific examples in her writing. This allowed her to “find and fix” the targeted sections. For example, in the word choice category, the program identifies up to 10 “bad phrases” (e.g., vague words, clichés, overused words) used in the draft, allowing Dez to search for these words and replace them with more precise word choice.
Perceived Benefits of AES
Dez appreciated the immediate usability of some of the lower level editing tasks provided in her feedback reports such as spelling, grammar, and mechanics. She also noted when the feedback identified specific words or sentences in her writing. Dez reported that she first started with the “spelling, grammar, and vocab corrections.”
Beyond these “corrections,” Dez indicated this feedback helped her identify patterns not previously recognized in her writing. She knew she could be repetitive but had been “completely blind” to the examples AES identified. All three programs also identified Dez’s use of passive voice and lack of sentence variety. Dez shared she did not “necessarily know what it [passive voice] was,” but because it was an issue that all three programs identified, it raised her awareness of this pattern in her writing. She shared:
I know this was like always or at least somewhat mentioned in, like, every single one. And so I think that’s a helpful thing. Maybe one teacher notices it but another teacher doesn’t. It’s like a common human error, but when it comes to a program, all of them notice it and because they are immediately programmed to notice these things, and so I think it’s helpful in that sense, to help you recognize patterns of where you might be going wrong.
AES also helped Dez to consider her sentence variety. After viewing a chart analyzing her sentence length, Dez clicked on a resource underneath the chart titled “Effective Use of Sentence Length,” where she read about the importance of varied sentence length and learned some tips for varying her own sentence length. She shared:
I feel this helped my writing. Now I know that a good mixture of length is good in your writing. Instead of writing all long or short sentences, I can now focus on creating flowing sentences that get my point across to the reader.
She explained how helpful PaperRater was in helping her vary her sentence beginnings which she found to be both “important” and “very hard.” She explained:
So I think honestly sentence beginnings are very important. Like, really important. And so I’m glad this pointed it out because despite all my efforts. I always have a very hard time starting new sentences with different things. And so even if it’s a new paragraph, usually I’ll still reuse the same word from the paragraph prior and so even though it says that it’s good, I honestly am glad it still gives examples of, “Okay you kept repeating this word over and over and again. Fix it.”
PaperRater helped her to see a tendency to rely on simple sentence starts, rather than using a variety of sentence structures. These increased her awareness of both her sentence structure and sentence length.
Perceived Limitations of AES
Dez also highlighted areas where the feedback was less than helpful. She noted that the feedback did not address content and wondered if the system could be “gamed.” She elaborated:
[The programs are] not going to tell you, you went on for four or five paragraphs about dogs and then immediately switched over to cats and then talked about dogs again, without any connection between those two sources. And so, they are probably just telling you “Oh, well, you messed up this word about this cat or you messed up this word about this dog.” They are not telling you: “Hey you talked about dogs here, but then started to talk about cats and then talk about dogs again.” Like, are you sure you meant to do this?
She thought if the paper was written well, the program would not realize the sudden shift in topic.
In addition, Dez thought program feedback could discourage students, causing them to think that their writing is not “good.” Unlike AES programs, she felt that teachers would be more likely to “take into account who that person is, and whether they actually tried hard and could give better encouragement, better feedback saying you could improve on this, but you did really well on this.” She expressed concern regarding how “these programs automatically decide, you did this wrong, you did this, you did this wrong.” Because these programs did not take into account the writer and identifying “what is wrong” it could really “discourage someone from their writing.” Dez ended by saying, “I just think all around having someone who is the teacher grading, it is just a better way.”
Dez said she simply “skipped over” information that she did not find to be helpful, specifically some of the spelling suggestions for words in her paper not recognized by the programs including names, places, and content-specific vocabulary such as “animé.” During both days of in-class revisions while using AES, Dez asked for teacher feedback. She asked for specific feedback unrelated to AES but sought Jessica’s opinion on the content and flow of her introduction. In another instance, she asked for clarification and assistance to implement AES feedback in her draft concerning the programs’ not recognizing some of her words as spelled correctly. Dez did not feel comfortable relying solely on AES for her writing assignments. She shared: “I don’t think I could just use these [AES] and turn in my work with complete confidence knowing I did my absolute best until I actually consulted a teacher about my work or about any questions I might have.” Additionally, Dez preferred an opportunity for feedback via discussion, as opposed to just receiving feedback. She explained:
I do like having conversations and asking questions.… So, for me as a person, I do not feel comfortable or think I improve as much [with the AES feedback] … I just don’t see me improving with that [AES] over talking to a teacher, asking them questions, having them explain it clearly to me and then asking them any other questions that I might be confused on.
Changes in Dez’s Drafts
Nearly half of the students in the class reported only minor changes from their first draft to their draft after use of AES feedback. Insights from her questionnaires and interview indicate that most of Dez’s revision as a result of the AES feedback was lower level, with most changes in word choice and individual sentences. These “minor fixes” suggest many changes were editing tasks rather than revision. In the final questionnaire and also during the interview, Dez explained that she started with the spelling and grammar feedback as a way to “improve immediately.”
Comparing her initial draft to the draft after use of the feedback, we categorized Dez’s changes as at the word and sentence level revisions consisting of deleting, replacing, or adding ideas. See Table 3 for an overview of changes within one of Dez’s paragraphs. Overall, in addition to a couple of spelling corrections, Dez’s changes ranged from one word to one sentence in length. None of the changes involved whole paragraphs or anything longer than one sentence. Dez deleted information 28 times. She had expressed a concern about being repetitive and her draft being too long. She made 37 changes where she replaced a word, phrase or sentence with a new word, phrase, or sentence. In 24 instances, Dez added new information, including the addition of a photo with a caption. In comparing her two drafts there were no significant revisions in content, structure, or order of information.
Unintended Consequences
There were some clear unintended consequences with the AES feedback. As a gifted and high achieving student, Dez was not used to receiving low grades. For example, when she examined her score of 65 out of 100 from ProWriting Aid, Dez was disappointed. After all, on the grading scale she was accustomed to, this would almost be a failing grade. Dez expressed surprise at her score, “Oh my gosh, I thought it would have gotten better than that,” and “I have a lot to improve on,” before sharing how that score made her feel: “Bad. I feel really bad about myself.” Dez used the word “wrong” multiple times during the interview to describe the AES feedback she received. She felt the programs focused on what she did “wrong” in her writing. Dez needed reassurance and a reminder that the feedback was a suggestion.
Dez identified another potential issue that the feedback could provide a false sense of confidence. She mentioned being “in the green,” referring to PaperRater’s visual use of the red to green continuum to indicate strengths and weaknesses in their feedback categories. She warned:
Despite being in like the green, despite being good. There’s always, like, that little fault. And so instead of just saying, okay, you’re in the green, you’re good, you don’t need to do anything else. I think it still should be able to detect, like, little faults that you still can improve on. Despite being into the green. I mean, you’re giving someone, like, a false sense of, okay, I did really well, and making them, not making them, but not allowing them to improve on things they may have acknowledged before or maybe not even acknowledged yet. And it’s just giving them a tougher time or just making it longer for them to improve as a writer by not just giving them straight out places you could improve on just because they’re in the green, just because they’re better than most people.
These false messages indicated a need for instruction or assistance to aid students in interpreting and using AES feedback.
Discussion
Our work with Dez allows us to gain insights regarding her specific experiences and opinions regarding AES feedback. Feedback supports learning (Hattie, 2008), and immediate and task specific feedback is more beneficial for students (Kulik & Fletcher, 2016; Shute, 2008). AES can serve as a feedback agent; however, most provide lower level feedback to students (Wilson & Czik, 2016). Feedback, in essence, should help students understand the criteria for good performance and understand how their current attempt stands in relation to these criteria, but most importantly, the feedback should help students understand how to close this gap (Sadler, 1989).
Though AES feedback has the potential to act as a more knowledgeable other in student learning, Dez’s commentary serves as a reminder that a truly knowledgeable other must include characteristics that only another human being can possess. The AES programs we explore in this study had limitations in that there was no content feedback. All feedback including feedback from a more “knowledgeable other,” peers, and computerized feedback has its own strengths and weaknesses (Lee et al., 2019). From a sociocultural perspective, this feedback can act as a scaffold, but it appears it is an insufficient scaffold unto itself. Dez illuminated where she could take action and when she could not.
While AES was effective in making such patterns visible, Dez needed additional help to take action, seeking out clarification from the teacher. Strategy instruction can prepare students with tools to apply feedback from AES or any other feedback agent. There are helpful aspects of AES feedback that students can use independently, especially implementing direct lower level feedback when it provides clear actions to take. It is possible that if not for the AES feedback, Dez might not have learned that pattern within her writing because in their conferences, Jessica chose to focus on something else within her writing. And it is important to note, that understanding a particular pattern in her writing was within Dez’s ability as a writer at that time to understand independently.
Dez’s reactions and experiences with AES provide some interesting considerations for teachers who wish to explore such programs. Dez worried about the ease in which she could game the system. The programs utilized are designed to analyze quantitative elements associated with writing. Each report was multiple pages long. That can be overwhelming for even an advanced language arts student.
Dez described that she felt the programs did not take into account the person behind the writing. A teacher with a trained eye and relationship with the student writer is more likely to take “who” the writer is into account when providing feedback, thus making a more meaningful connection with the writer and her needs. Dez knew that she could approach Jessica and she appealed for help with some of this information. In addition, this was not the only feedback she received as Jessica provided her with more content-based feedback. In the end, even though Dez indicated she made improvements to her writing, most of these changes were lower level components of writing such as spelling, mechanics, and word choice (Wilson & Czik, 2016).
These findings center the teacher’s responsibility in developing students’ ability to use AES feedback in a productive manner. First, the reports can be overwhelming to many students. Students need to know where to start and which tasks have higher leverage to advance their writing. It might be satisfying to correct 12 spelling or punctuation errors, but it has lower payoff than understanding a tendency to start your sentences the same way. It is possible that teachers can direct students to look at particular aspects of the report or help prioritize the information they do receive. Teachers also need to address the issue of “wrongness.” The sheer amount of feedback can shake a writer’s confidence especially if viewed through the lens of “bad” (lots of things to fix) and “good” writing (few things to fix). It may not be helpful that some programs give students’ scores on their papers. For example, Dez was surprised by her 65/100 score. Even though this was not a grade, this is a failing percentage in a school-based grade system. Even if AES feedback is considered low stakes, that can be a startling moment for a writer.
The limitations of AES illustrate the importance of the teacher’s role in students’ writing growth. The programs tended to provide what to work on without always indicating how to do it or where in the writing to apply it. In several instances, Dez’s uncertainty about what AES feedback meant or how to implement it in her writing, points to a need for teacher instruction on how to interpret some AES feedback before students can be expected to apply the feedback to their writing. In addition, students need explicit strategy instruction for revision that emphasizes the importance of revision and how to do it (Dinkins, 2014; Philippakos, 2017; Witte, 2013; Yuknis, 2014).
At the same time, AES can ease the burden of time for teachers. It is a tool that students can utilize at any point of their writing process for immediate feedback, which the teacher cannot generally provide (Foltz et al., 2013). Overall, AES offers more lower level feedback than higher level feedback, allowing students to get some of the easier changes or “quick fixes” out of the way, often working at the word and sentence level. However, this could support teachers spending more time providing higher level feedback. According to Link et al. (2020), teachers who did not use AES feedback provided more lower level feedback than when AES was utilized.
A limitation for this study is our focus on one individual student. With so much to navigate within a piece of writing and specific feedback for that piece of writing, our decision to focus on one student does not illuminate the experiences of all students. In addition, teachers consider Dez an advanced writer. Although her ability to utilize some aspects of feedback and her confusions about other parts provide helpful insights for teachers to consider if using AES in their classrooms. However, our findings suggest AES can be helpful to both students and teachers when used as a feedback agent during the writing process but simply does not replace teacher feedback and instruction.
We content that more studies are needed to hear from students about their writing processes and understandings. As AES becomes more common and readily available, we need more research investigating what students find helpful and how they use feedback, which may provide insights regarding how to best support students’ writing development. In addition, identifying what is missing from computer feedback and identifying the areas students need concerted attention may help educators in planning targeted instruction.
Conclusion
With feedback a nearly universally recommended practice for writing instruction, coupled with the demands of time and complexity, it is important to explore ways to lessen the burden on teachers through exploration of other feedback agents such as AES. This study sought to include students into this conversation and highlight one student’s voice as an important stakeholder in her own education and development as a writer. While AES has benefits, it only one additional tool to aid teachers in providing feedback. As Dez has illustrated, AES feedback alone was not sufficient to address her needs as a writer. Future research could explore what other tools might support teachers in providing personalized feedback to their students.
Acknowledgment: The authors report there are no competing interests to declare.
