Using Rubrics in A University EFL Process Writing Program : An Exploratory Case Study

While rubrics have been widely recognized as an effective instructional tool for teachers to evaluate students’ writing products, fewer studies explored how students use it for their writing process in an EFL university academic writing classes. This study explores the application of process-oriented rubrics in two EFL writing programs, and investigates whether English language proficiency, motivation to writing, and their previous experiences with writing programs would significantly affect the use of the rubrics. The participants (N=190) were from two student cohorts, each of which had 95 participants. The data set includes students’ self-, peeruse and the instructor’s use of the rubrics, and students’ written reflection upon peer feedbacks. The data showed that the rubrics can guide students to practice a writing process, and that the 20-item rubric was statistically reliable. The data of rubrics also showed that the participants were more critical on their peers’ writing, and the reflection data showed students’ awareness of revision strategies. The qualitative data seemed to suggest that peer reviews and reflections upon such reviews could enhance students' revision strategies. This article will conclude itself by providing some pedagogical suggestions in EFL contexts.


Introduction
The use of rubrics in academic writing classrooms has gained attention since the 1980s (Broad, 2003).A rubric is "a guide listing specific criteria for grading or scoring academic papers, projects, or tests, and an instrument that describes a specific level of performance within a scale (Crusan, 2015, p.1)".In other words, it is an assessment tool with a clarified description of evaluation criteria and performance levels.
There are two major opinions regarding the use of rubrics in writing programs.Opponents (Kohn, 2006;Wilson, 2007) question its effectiveness as an assessment and instructional tool since learners' higher order needs are more likely to be ignored, and teachers do not always improve their instruction.This group of scholars pointed out that a rubric, in essence, legitimizes standardized writing practices, instead of displaying its expectation of students' self-monitored writing practices.Proponents, on the contrary, believe that rubrics play an important role in writing classes.In particular, rubrics can make the scoring process transparent and consistent (Bradford, Newland, Rule, & Montgomery, 2016;Ene & Kosobucki, 2016;Jonsson, 2014;Jonsson & Svingby, 2007;Li & Lindsey, 2015;Montgomery, 2000;Panadero & Jonsson, 2013;Sluijsman, Dochy, & Moerkerke, 1999), and can have the potential to improve learning and/or instruction (Jonsson & Svingby, 2007;Panadero & Jonsson, 2013).Specifically, rubrics can enhance learners' critical thinking in their writing process, and inform them of course goals and expectations.To instructors, rubrics are an effective approach to have more understandings of students' needs.
Although it has been in constant debate upon the effectiveness of rubrics as an instructional tool in writing programs, there are still many attempts to employ rubrics as an instructional tool (Jonsson & Svingby, 2007;Panadero & Jonsson, 2013).Most discussions are on how to design valid and reliable rubrics for teachers to assess students' writing (Bai, 2012), yet fewer studies discussed how students use rubrics to evaluate and improve their writing practices.
Among the few discussions on the use of rubrics as an instructional tool, major discussions are on (1) how to use it in peer-, self-, and teacher-assessment (Ashton & Davis, 2015;Bradford, Newland, Rule, & Montgomery, 2016;Li & Lindsey, 2015;Matsuno, 2009;Leggette, McKim & Dunsford, 2013;Lindblom-Ylänne, Pihlajamäki, & Kotkas, 2006;Wang, 2016), and (2) whether it should be used with other activities such as corrective feedbacks (Diab & Balaa, 2011;Ene & Kosobucki, 2016;Ghadi & Khodabakhshzadeh, 2016;Lee, 2011;Trace, Meier, & Janssen, 2016).Those discussions have undoubtedly provided important insights, yet one question remains unanswered: how to engage students to employ rubrics in their multi-drafted writing?Also, those studies mainly discussed the use of rubrics on students' writing products instead of writing process, although one purpose of using rubrics is to promote students' understanding of writing as a process instead of a product (Dochy, Segers, Sluijsmans, 1999;Leggette, McKim, & Dunsford, 2013;Sluijsman, Dochy, & Moerkerke, 1999).Therefore, more studies are needed to the use of rubrics on multiple drafts in process writing classes.This study attempts to explore the use of in multipledrafted writing and with other activities (i.e., peer feedbacks) in a processing writing program at an EFL university.

Literature review
Process writing has been a popular teaching approach in writing classes since the 1980s (Graham & Sandmel, 2011), which emphasizes multiple drafts of writing (i.e., pre-writing, writing, and post-writing).Ideally, a rubric of process writing makes course goals transparent and guides students through the stages of writing (Jonsson, 2014).Interestingly, most studies explored the use of rubrics on a one-drafted writing, and fewer studies discussed the role of rubrics in students' multi-drafted writing process.Among the few, it was found that rubrics, applied with other tools, can promote students' application of revision strategies and lead to better accuracy.For example, Diab and Balaa (2011) found that students could effectively revise their second draft with the use of comments, grades and the rubric.Greenberg's (2015) study found that students improved their writing quality through the use of rubric and peer-assessment.Lam (2013) found that students could develop insights into the gap between their current and desired level of performance, and take appropriate remediation, through self-, peer-, and tutor-assessment.In other words, when used with peer-assessments, comments and grades, rubrics can be more effective in promoting students' revision abilities in process writing.
While it is generally agreed that rubrics can be more effective when used by self, peer and teachers, disparities were also found among the three groups.Three reasons have been suggested.One reason is their varied interpretation on rubrics.Li and Lindsey (2015) found that students usually focus on key words whereas teachers have a broader picture of evaluated skills.The second reason is the bias that the three groups show in the use of rubrics.Matsuno (2009) found that teachers usually have their patterned bias in rating, while they do show higher consistency in rating, while students' peer-assessment shows the lower level of patterned bias.
Thirdly, contradictory findings have been reported regarding students' self-and peerassessment.Matsuno (2009) found that the students tended to rate their own writing lower, and their peers' writing higher, whereas Topping (2013) found that the students rated their writing higher.For the disparities between students' self-, and peerassessment, some studies explored the reasons.Lindblom-Ylänne, Pihlajamäki, and Kotkas (2006) found that the learners felt difficult to remain objective in rating their own writing, while it was equally difficult to remain critical in rating their peers' writing.Torres-Guijarro and Bengoechea (2017) found that gender may be a factor since female students tend to judge their writing too harshly.
While the aforementioned studies have been exploring the three groups' use of rubrics, another group has been discussing students' self-assessment, in particular, whether or not training of scoring can decrease the disparity.Ashton and Davies (2015) found that students with no training of scoring are less likely to differentiate between the advanced, the intermediate and the novice writings.
Another group of studies have been investigating the application of rubrics with other tools such as corrective feedback (Ene & Kosobucki, 2016;Panadero & Jonsson, 2012).Panadero and Jonsson (2012) suggested that the use of rubrics can facilitate students' learning if combined with other meta-cognitive activities such as self-and peer-assessment.Ene and Kosobucki (2016) found that corrective feedback as a supplementary tool can satisfy the learners' higher order needs.Ghadi & Khodabackhshzadeh (2016) found that the employment of technology in peerassessment can also improve writing performance and positive belief in writing.
In conclusion, rubrics can be an effective instructional tool since it engages students to envision the gap between drafts and course goals.While there are disparities among the teacher-, self-and peer-uses of rubrics, several strategies have been discussed in order to decrease the disparities such as training, oral feedbacks and more.Those studies have provided important insights, yet most studies have focused on ESL learners and English-as-the-first-language context, whereas fewer studies have investigated in English-as-a-foreign-language context (Ghadi & Khodabakhshzadeh, 2016;Wang, 2016).And also, few studies looked at the use of rubrics for multiple drafts in advanced English academic writing programs.

Context and Participants
This research was based on a 16-week Advanced Academic Writing course offered to 190 undergraduate students at a southern university in China in the year of 2015.Students of this university have been widely recognized by higher proficiency in English language.Two groups of participants (N=190) were selected due to their major and English language proficiency.One group was the third-year students from School of English and Education (SEE), and another was the second-year students from an accounting program (ACCA).
The SEE participants (N=95) were from a pre-service English language teacher program admitted to this university in the year of 2012.The English language proficiency of this group was generally higher than that of the ACCA group, according to the score of English subject from the entrance examination to university.This group was divided into four classes for administrative convenience, each of which had approximately 25 students.This group had been offered a two-stage writing program: The first-stage program focused on the expository writing, and the second stage on academic writing.The course discussed in this study was the second-stage program, which took place at the students' third academic year university study.
The ACCA participants (N=95) admitted to this university in the year of 2013.They were from an enhanced ACCA program, in which all the students needed to excel on a screening test of Math and English.Therefore, this group had a higher level of English proficiency compared to their peers from the same institute.This group was divided into two classes due to administrative reasons, each of which had more than 40 students.This group also took more English-related courses than their peers, including a one-year English academic writing offered to this group only.Compared to the SEE participants, this group did not have the first-stage writing program.The second-stage writing program was the only writing program they received to enhance their English academic writing skills.

The Course
This 16-week academic writing course was the first part of the second-stage writing program.It focused on academic writing skills, while the second part was on writing a research paper.
In this 16-week course, students learned to write a three-drafted 1500-word book review, which should have the elements of summarizing, paraphrasing, narratives, as well as specific strategies for writing multiple drafts.As shown in Figure1, the course was divided into four stages.In the first week, the participants were explained that they would be evaluated by a rubric (see Appendix A) designed by the instructor/the researcher.The 20-item rubric was explained item by item.Sample writings from the previous year were used in order for a better understanding of the rubric.At this stage, the course focused on general brainstorming strategies and some writing strategies such as how to write an introduction, paragraphs and a conclusion.
In the 8th week, which was also at the second stage, the students submitted the first draft.During the week of 8-11 weeks, students were introduced more strategies such as summarizing and paraphrasing.
From the 11th week, students should submit the second draft; finally, in the 16th week, they should submit the final draft.Between the 11th and 16th week, the course focused on revision strategies.Issues emerged from their first and second drafts were selected, categorized and explained in the class.During these weeks, the course also had oral reports, students' book presentation and peer reviews in order for students to have more opportunities to engage in critical thinking and revision for their final drafts.

Figure 1: General Course Schedule
Both groups had the same course schedule and assignments.Both groups received written feedbacks from the instructor regarding their first, and 2nd draft.Also, both groups had group presentations on their book review.However, there were some differences in instructional plans.Between the week 11 and week 16, the ACCA group did not have oral peer feedbacks and oral book report, whereas the SEE group had one oral book report and three oral peer feedbacks.
The book presentation that both groups had was an 8-minute presentation on a chosen book.The students chose a book, introduced its main contents, and explained its significance in English.
The activities that only the SEE group had were (1) oral report and (2) three peer oral feedbacks (see Table 1).The oral report was in a lecture format: each student reported to the class his/her writing plan, while the rest listened and took notes.After the report, the students were asked to write a reflection regarding (a) what they have learned from other students' oral reports, and (b) what they would do next to improve their own writing.
The oral peer feedbacks had three sections, each of which had a different focus.During these sections, usually the instructor would take 40 minutes out of 80minute class explaining to the students the major issues emerging from their second draft.Then the students would take the rest 35 minutes discussing and giving feedbacks.The participants chose one peer in the same class, with whom they needed to work for the three sections.During the discussion time, the students were encouraged to use both L1 and L2, so language proficiency should not be an issue.Afterwards, they took 5 minutes to write a quick reflection regarding (a) their opinions they provided to their peer, and (b) suggestions they took from their peer(s).

Table 1: Oral peer feedback schedule # Focus
The 1 st An oral report + a written reflection The 2 nd  Introduction and conclusion+ a written reflection The 3 rd  Paragraphs + a written reflection The 4 th  Sentences and vocabulary + a written reflection

The rubric
The instructor designed an analytic rubric based on the course goals, one of which was to enhance students' understanding of writing process and abilities to revise their writing.This 20-item rubric had four major parts: the three drafts and the revision process: The item 1-11 were about the final draft, item 12-16 about the revision process, item 17-18 the 1st draft, and 19-20 about the 2nd draft (Table 2).Each item had five points with 1 point the least and 5 the most.The language of the rubric was English.

Data collection and analysis
The data included quantitative and qualitative data.The quantitative data was two cohort groups' self-, peer-, and teacher-assessment of the three-drafted writing.
The qualitative data included the SEE group's four reflections from the one oral report and the three peer feedbacks.The SEE group (N=95) was divided into four classes for the administrative convenience.One class (N=23) was randomly selected for the thematic analysis.The reflections were written after each peer feedback.
The quantitative data was analyzed using software SPSS 16.0, and the qualitative data was analyzed by thematic analysis approach.

Findings
This purpose of this study was to explore the use of rubrics in an EFL academic writing program at university level.Both the two cohort groups used the rubrics to guide their writing process, while the ACCA group used the rubrics only and the SEE group used the rubrics in combination with oral report and peer feedbacks.Both groups received teacher's written feedbacks and guidance on revision.The findings suggested that it is pedagogically possible to apply rubrics in both larger and smaller size classes.While oral report and peer feedbacks seemed to promote the SEE group's decisions in revision, it did not show significant difference in their self-assessment compared to the ACCA group.

The reliability of the rubrics
The students' self-assessment data (N=190) were used to test the reliability of the rubric by Software SPSS 16.0.The Cronbach's Alpha scores were .848.Therefore, the rubrics were deemed appropriate for further discussion.

Self-, peer-, and teacher-assessment
Figure 2: The Overall Score Both groups' assessment did not show significant difference, implying that other activities (i.e., oral report and peer feedbacks) did not necessarily affect students' use of rubrics.As presented in Figure 4, both the two groups' self-assessments were close and the SEE's peer-assessment was also close to the two groups' self-assessment: The ACCA's self-assessment mean score was 85.54, the SEE self-assessment was 84.76, and the SEE peer-assessment was 83.Teacher's assessment on the SEE group was significantly higher (N=81.97)than the ACCA (N=73.51).

Self-assessment vs teacher's assessment
Comparing students' self-assessment to teacher's assessment, the findings showed that both the students evaluated their writing higher compared to the instructor's.
Specifically speaking, about 15 out of 20 items showed statically significance (see Figure 5).The students' (N=190) self-assessment showed that almost all the four parts showed statistical significance.
However, when we compared the SEE and the ACCA group students' self-assessment with each other, we found some interesting differences, which will be further discussed in the following section.

Two groups' self-assessment a. Perception of the multi-drafted writing process
The two groups' self-assessment data showed their different understandings of three drafts in a writing process.Both groups looked at the three-drafted writing as twodrafted writing, yet their perceptions of draft one, two and three varied.
The SEE group assessed their final draft significantly different from the 1st and the 2 nd draft, while no significant difference was found between the 1 st draft and the 2 nd draft.It can suggest that the SEE group regarded their 1 st draft similar to the 2nd draft, which may suggest that the SEE group regarded the three-draft writing process as twodraft writing.
The ACCA group assessed the final draft significantly different from the 1 st draft, but not from the 2nd draft, while no significant difference was found between the 1st draft and the 2nd draft.This may imply that the ACCA group did not regard the 2nd draft necessary.
The findings suggested that both groups regarded the three-drafted process writing as two-drafted writing, yet their perception of draft two varied.

b. Writing strategies
The two groups' self-assessment were mostly similar to each other, which suggests that the four oral peer feedbacks that the SEE group received may not make a distinct impact on the rating of their writing.
Also, the data showed that the two groups rated the five items (i.e., 1,4,9,11,14) significantly different.The four items (1, 4, 9, 11) were about the final draft, while item 14 was about authors' voice in revision.In particular, the two groups showed different understanding regarding (a) the elements of a book review and the required format (item 1 & 9), (b) the strategies of writing an introductory paragraph (item 4), and (c) verb tenses and punctuation (item 11).

SEE: self-and peer-assessment
The SEE group's self-and peer-assessment showed some interesting results.First, they rated their peer's writing lower, yet the gap did not show statistic difference but item 3 and 20 did.While item 3 was about the author's voice in the final draft, the item 20 was about the revision skills applied in the second draft writing.This suggests that the participants, after four oral peer feedbacks and reflection writings, may have enhanced more critical readership as a reader of their peers' writing than of their own writing.

The Reflection
The qualitative data showed that the SEE group paid more attention to improving overall organization as well as sentence structure, compared to the flow of thoughts, words as well as verb tenses (see Table 3).Overall organization "The introduction is attractive with scenarios and it shows that the book is about woman issues.The conclusion can enlighten the meaning Splendid Suns.The paragraphs show the author's understanding from different perspectives, but there are some strong words, and some contradictions in showing author's opinions.The author thinks men would hire men but there is no fact/data to support, and she write that being a tomboy is frustrating, but later she expresses her admiration toward Hillary.
They are contradictory to each other." "The review is well-constructed and the structure in the review is clear.I think it is better to explain the unique and new concept to the reader.Also the pronouns should keep consistent.Finally, if she can try to add some content to make the book and reader closer, the review will be more complete." "Shorten the story part and change the standpoint." "The conclusion is not convincing enough.…The introduction should be rearranged….Shorten the story summary and change the perspective from Agguie to ordinary people."

Sentences
"She has put important information in a sub-ordinate clause, but the length of sentences are too long, taking 2 or 3 lines.And also because she involved many information in one sentence, it is sometimes not easy to understand." "Some sentences are too long and contain too many information."

Conclusion
This study attempted to explore the application of a process-oriented rubric with and without other meta-cognitive activities in an EFL writing program at university level.
The findings suggest that it is pedagogically possible to implement process-oriented rubrics in both larger and/or smaller size EFL classes, yet more instructions are needed regarding the roles of each draft in process writing.
The quantitative data show that the participants needed more training in order to rate their multiple-drafted writing process.It also suggests that they did not regard the writing process, critical thinking and the revision differently from each other.The slightly lower peer-assessment may suggest students' rising awareness of a writing process and revision process.The qualitative data shows that rubric-based peer feedback enhances students' awareness of revision plan at both macro-, and microlevels.The students were aware of further revision at both organization and sentence levels, yet this reflection did not necessarily improve their abilities to evaluate their own revision process.
The findings suggest that self-, peer-, and teacher-use of rubrics can promote teachers' pedagogical decisions in that it shows students' understanding of different drafts in the process of writing.Also, the findings suggest that meta-cognitive activities promoted students' critical readership and revision strategies, yet such promotion was not revealed in the students' self-, and peer-assessment.This suggests that future research should explore how to improve the rubrics in order to guide students through their writing process.Future research should also focus on students' understanding of multiple drafts in process writing.
This study has its limitations.First, the participants were not trained to self-and peerassess their process of writing, although they had a brief explanation in the first week.Second, the two major groups were from different levels (SEE senior, and ACCA sophomore).Third, the SEE students' oral feedback could have been recorded or videoed for further study, so we could know more about how they implemented their plan of revision in their drafts.However, these also showed the reality of EFL

Figure 4 :
Figure 4: the ACCA and the SEE self-assessment

Figure 7 :
Figure 7: SEE's selfand peer-assessment final draft has the essential elements of a book review: Introduction, Summary of the main contents, Discussion, and Conclusion.final draft shows the student's attention by writing: pointing out misunderstanding, filling the gap and contributing further understandings (one of the three).Summary resonates to the purpose of the book review (i.e., audience, purposes).5 4 3 2 1 6.The Conclusion shows the student's critical thinking to the so-what questionflow of thoughts is smooth by application of Key words, Topic Sentences and organization skills.5 final draft has a significant proportion of original work (i.e., personal comments instead of hearsay)obvious errors are found in the use of tenses and punctuations.5 4 3 2 1 II: Revision Process: From 1st draft to the Final Draft 12. Efforts on improving sentence variety are evident.5 4 3 2 1 13.The flow of thoughts is significantly improved.5 4 3 2 1 14.Critical thinking is obvious through the process of revision.5 4 3 2 1 15.Basic revision activities are apparent through the three drafts such as word choice, tone, tense and etc.5 second draft demonstrates the student's abilities to apply revision skills taught in the class.5 4 3 2 1

Table 2 :
The rubric