A Framework for Analyzing Features of Writing Curriculum in Studies of Student Writing Achievement
Saved in:
| Title: | A Framework for Analyzing Features of Writing Curriculum in Studies of Student Writing Achievement |
|---|---|
| Language: | English |
| Authors: | Oddis, Kyle, Burstein, Jill, McCaffrey, Daniel F., Holtzman, Steven L. |
| Source: | Grantee Submission. 2022 6:95-144. |
| Peer Reviewed: | Y |
| Page Count: | 50 |
| Publication Date: | 2022 |
| Sponsoring Agency: | Institute of Education Sciences (ED) |
| Contract Number: | R305A160115 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Writing Instruction, Writing Achievement, Curriculum Evaluation, College Instruction, Scoring Rubrics, Writing Evaluation, Writing Exercises, Course Descriptions |
| DOI: | 10.37514/JWA-J.2022.6.1.05 |
| Abstract: | Background: Researchers interested in quantitative measures of student "success" in writing cannot control completely for contextual factors which are local and site-based (i.e., in context of a specific instructor's writing classroom at a specific institution). (In)ability to control for curriculum in studies of student writing achievement complicates interpretation of features measured in student writing. This article demonstrates how identifying and analyzing features of writing curriculum can provide dimensions of local context not captured in analysis of student-generated texts alone. Using a dataset of 48 curricular texts collected from 21 instructors teaching in five disciplines across six four-year public universities in the United States, this article: (1) presents a set of curriculum scoring rubrics developed through qualitative analysis, (2) describes a protocol for training raters to use the rubrics to score curricular texts to achieve rater agreement and generate quantitative data, and (3) explores how this framework might be amended to more deeply consider feature relationships between curriculum and student writing in studies of student writing achievement. Literature Review: The literature review provides an overview of existing studies that our research expands upon; grounds rubric development in genre theory, threshold concepts in writing studies, and design thinking; and explores how conducting curriculum analysis in tandem with feature analysis of student writing can benefit writing analytics research programs. Research Questions: (RQ1) What identifiable features of writing curriculum might affect how students approach situated writing tasks? (RQ2) How can we categorize features of writing curriculum to help us better understand its role in student writing achievement within and across disciplines? (RQ3) Can we produce a quantitative measure of curricular features that can be used in conjunction with natural language processing (NLP) data gathered on features of student writing? (RQ4) How can a set of usable, theoretically grounded rubrics offer insight into what research teams interested in studies of writing achievement might consider going forward? Methodology: The first phase of this study involved qualitative analysis of curricular texts as a guide for creating scoring rubrics. The scoring task in the second phase of the study consisted of three components: (1) Development of scoring rubrics. Rubric development was based on observations from two exploratory rounds of qualitative coding of texts in our dataset. Rubrics addressed five features of writing curriculum: "accessibility," "applicability," "actionability," "situational clarity," and "overall quality." (2) Annotation protocol training. Three research assistants with experience annotating linguistic features in texts served as raters and were trained to annotate and score the curricular texts according to the rubrics. (3) Application of scoring rubrics. Each trained rater scored the curricular texts in batches, and the study lead (first author) served as an "expert rater" who also scored the texts so final rater agreements (quadratic weighted kappa) could be calculated. Results: Qualitative analysis revealed that features of writing curriculum varied widely across learning sites, attesting to a lack of standardization or consistency of writing curriculum at and across institutions. Quantitative results speak to challenges in producing "usable" statistical data with a limited dataset. Discussion: Our study illustrates the challenges of applying rubrics to curricular datasets which offer only a partial picture of the realities of teaching and learning writing in multiple disciplines at various institutions. The potential to observe relationships between features of curricular texts and features measured in student writing requires collecting more robust datasets that include assignment grading rubrics, assignment sheets/instructions, and syllabi across disciplines in local contexts where writing happens. Future studies would need to include a sufficiently large number of courses where faculty provide a complete set of relevant curricular materials to allow for course-level analysis. Conclusions: This study's design is promising for application to larger datasets which may be drawn from single and multi-institutional contexts. Our limited dataset offers inconclusive results for demonstrating relationships between student writing features and features of associated writing curriculum (e.g., student writing motivation and applicability of curriculum). However, insights from this process suggest that in order to understand student writing achievement more comprehensively, we must develop more diversified data collection and analysis practices. This would afford deeper insight into the complexities of teaching and learning writing, specifically in terms of how students orient themselves to writing tasks delivered in curriculum. Future approaches to similar kinds of research can offer more insight into how curriculum affects student writing achievement and broader outcomes (e.g., college GPA). |
| Abstractor: | As Provided |
| IES Funded: | Yes |
| Entry Date: | 2022 |
| Accession Number: | ED618640 |
| Database: | ERIC |
| FullText | Text: Availability: 0 CustomLinks: – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED618640 Name: ERIC Full Text Category: fullText Text: Full Text from ERIC |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: ED618640 AccessLevel: 3 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: A Framework for Analyzing Features of Writing Curriculum in Studies of Student Writing Achievement – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Oddis%2C+Kyle%22">Oddis, Kyle</searchLink><br /><searchLink fieldCode="AR" term="%22Burstein%2C+Jill%22">Burstein, Jill</searchLink><br /><searchLink fieldCode="AR" term="%22McCaffrey%2C+Daniel+F%2E%22">McCaffrey, Daniel F.</searchLink><br /><searchLink fieldCode="AR" term="%22Holtzman%2C+Steven+L%2E%22">Holtzman, Steven L.</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Grantee+Submission%22"><i>Grantee Submission</i></searchLink>. 2022 6:95-144. – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 50 – Name: DatePubCY Label: Publication Date Group: Date Data: 2022 – Name: SourceSuprt Label: Sponsoring Agency Group: SrcSuprt Data: Institute of Education Sciences (ED) – Name: NumberContract Label: Contract Number Group: NumCntrct Data: R305A160115 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Journal Articles<br />Reports - Research – Name: Audience Label: Education Level Group: Audnce Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink> – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Writing+Instruction%22">Writing Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Achievement%22">Writing Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Curriculum+Evaluation%22">Curriculum Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22College+Instruction%22">College Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring+Rubrics%22">Scoring Rubrics</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Exercises%22">Writing Exercises</searchLink><br /><searchLink fieldCode="DE" term="%22Course+Descriptions%22">Course Descriptions</searchLink> – Name: DOI Label: DOI Group: ID Data: 10.37514/JWA-J.2022.6.1.05 – Name: Abstract Label: Abstract Group: Ab Data: Background: Researchers interested in quantitative measures of student "success" in writing cannot control completely for contextual factors which are local and site-based (i.e., in context of a specific instructor's writing classroom at a specific institution). (In)ability to control for curriculum in studies of student writing achievement complicates interpretation of features measured in student writing. This article demonstrates how identifying and analyzing features of writing curriculum can provide dimensions of local context not captured in analysis of student-generated texts alone. Using a dataset of 48 curricular texts collected from 21 instructors teaching in five disciplines across six four-year public universities in the United States, this article: (1) presents a set of curriculum scoring rubrics developed through qualitative analysis, (2) describes a protocol for training raters to use the rubrics to score curricular texts to achieve rater agreement and generate quantitative data, and (3) explores how this framework might be amended to more deeply consider feature relationships between curriculum and student writing in studies of student writing achievement. Literature Review: The literature review provides an overview of existing studies that our research expands upon; grounds rubric development in genre theory, threshold concepts in writing studies, and design thinking; and explores how conducting curriculum analysis in tandem with feature analysis of student writing can benefit writing analytics research programs. Research Questions: (RQ1) What identifiable features of writing curriculum might affect how students approach situated writing tasks? (RQ2) How can we categorize features of writing curriculum to help us better understand its role in student writing achievement within and across disciplines? (RQ3) Can we produce a quantitative measure of curricular features that can be used in conjunction with natural language processing (NLP) data gathered on features of student writing? (RQ4) How can a set of usable, theoretically grounded rubrics offer insight into what research teams interested in studies of writing achievement might consider going forward? Methodology: The first phase of this study involved qualitative analysis of curricular texts as a guide for creating scoring rubrics. The scoring task in the second phase of the study consisted of three components: (1) Development of scoring rubrics. Rubric development was based on observations from two exploratory rounds of qualitative coding of texts in our dataset. Rubrics addressed five features of writing curriculum: "accessibility," "applicability," "actionability," "situational clarity," and "overall quality." (2) Annotation protocol training. Three research assistants with experience annotating linguistic features in texts served as raters and were trained to annotate and score the curricular texts according to the rubrics. (3) Application of scoring rubrics. Each trained rater scored the curricular texts in batches, and the study lead (first author) served as an "expert rater" who also scored the texts so final rater agreements (quadratic weighted kappa) could be calculated. Results: Qualitative analysis revealed that features of writing curriculum varied widely across learning sites, attesting to a lack of standardization or consistency of writing curriculum at and across institutions. Quantitative results speak to challenges in producing "usable" statistical data with a limited dataset. Discussion: Our study illustrates the challenges of applying rubrics to curricular datasets which offer only a partial picture of the realities of teaching and learning writing in multiple disciplines at various institutions. The potential to observe relationships between features of curricular texts and features measured in student writing requires collecting more robust datasets that include assignment grading rubrics, assignment sheets/instructions, and syllabi across disciplines in local contexts where writing happens. Future studies would need to include a sufficiently large number of courses where faculty provide a complete set of relevant curricular materials to allow for course-level analysis. Conclusions: This study's design is promising for application to larger datasets which may be drawn from single and multi-institutional contexts. Our limited dataset offers inconclusive results for demonstrating relationships between student writing features and features of associated writing curriculum (e.g., student writing motivation and applicability of curriculum). However, insights from this process suggest that in order to understand student writing achievement more comprehensively, we must develop more diversified data collection and analysis practices. This would afford deeper insight into the complexities of teaching and learning writing, specifically in terms of how students orient themselves to writing tasks delivered in curriculum. Future approaches to similar kinds of research can offer more insight into how curriculum affects student writing achievement and broader outcomes (e.g., college GPA). – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: CodeSource Label: IES Funded Group: SrcInfo Data: Yes – Name: DateEntry Label: Entry Date Group: Date Data: 2022 – Name: AN Label: Accession Number Group: ID Data: ED618640 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED618640 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.37514/JWA-J.2022.6.1.05 Languages: – Text: English PhysicalDescription: Pagination: PageCount: 50 StartPage: 95 Subjects: – SubjectFull: Writing Instruction Type: general – SubjectFull: Writing Achievement Type: general – SubjectFull: Curriculum Evaluation Type: general – SubjectFull: College Instruction Type: general – SubjectFull: Scoring Rubrics Type: general – SubjectFull: Writing Evaluation Type: general – SubjectFull: Writing Exercises Type: general – SubjectFull: Course Descriptions Type: general Titles: – TitleFull: A Framework for Analyzing Features of Writing Curriculum in Studies of Student Writing Achievement Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Oddis, Kyle – PersonEntity: Name: NameFull: Burstein, Jill – PersonEntity: Name: NameFull: McCaffrey, Daniel F. – PersonEntity: Name: NameFull: Holtzman, Steven L. IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2022 Numbering: – Type: volume Value: 6 Titles: – TitleFull: Grantee Submission Type: main |
| ResultId | 1 |
