Reading Apprenticeship Writing Connections (RAWC): Evaluation

Research & Evidence

Reading Apprenticeship Writing Connections (RAWC)

Overview and Context of the RAWC Evaluation

RAWC included an evaluation to provide project leaders with formative and summative data for the purposes of program improvement and impact assessment of this new blended model of discipline specific professional learning. The evaluation included a randomized controlled trial (RCT) on a subset of 27 of the more than 200 participating schools to more closely examine the impact of Reading Apprenticeship on students’ academic reading and writing skills and dispositions. Data sets and measures included: relevant state standardized literacy tests; an ETS-developed computer-administered reading and writing test; and a student survey of metacognitive literacy knowledge and dispositions toward literacy and learning.

Reading Apprenticeship: Writing Connections (RAWC) allowed SLI to design and test a model that shortened our discipline-specific professional learning model from ten days to seven days, and explore the potential of online technologies for ongoing learning and implementation support. As such, it shortened the time for teacher learning from two years (as had been the model for prior efficacy studies) to one year.

To carry out a rigorous impact evaluation of this model within the three-year grant cycle, it was necessary to begin the RCT in the first year of the hybrid program development and implementation. Further, to enable control teachers to participate in the professional development opportunities at the completion of data collection, it was necessary to measure student learning impacts in the same, first year of teacher learning, before teachers had completed the entire program of professional learning. The design of the RCT was therefore unlike previous efficacy studies of Reading Apprenticeship professional development in several ways. It enabled program developers to examine a number of questions that would shed light on ways to streamline the model, examining questions such as:

  • Could teachers learn to implement new pedagogies as the school year ran along, and do so well enough to impact student learning outcomes in the same year?
  • Could the hybrid elements offer enough ongoing support to accomplish the deep changes in instructional practice that have previously been measured over two years?

Findings and Limitations of the Study

Not surprisingly, given the grant timeline constraints, attrition, small number of participants, and other limitations noted by the evaluator (evaluating the project in a pilot year, studying teacher practice change prior to completion of the professional learning, and inconsistent issuing of the student assessment), the randomized control trial did not find statistically significant impacts on all aspects of implementation and student achievement for this particular blended model of professional learning during its first year of implementation. However, many positive impacts of the project for teachers and students were noted and align with Reading Apprenticeship efficacy studies.

Selected Findings from the RCT and Scale-Up Survey

Although not statistically significant, evaluators found moderate to large positive effect sizes for the following teaching practices and student actitivies consistent with the professional learning goals:

  • Fostering student independence
  • Students practice of metacognitive inquiry strategies
  • Student practice of comprehension strategies
  • An increase in time students engage in collaborative activities
  • Student engagement (proportion of students who completed homework and/or paid attention in class)

The findings described above were supported by results of the scale-up survey issued to all teacher participants. Below are a few of these results:

  • 91% of teachers increased the number of classroom assignments that including reading and writing activities
  • 90% reported that the quality of their students’ work improved
  • 86% said they raised their expectations for students’ reading comprehension level
  • 83% raised their expectations for students’ subject area learning
  • 83% reported that Reading Apprenticeship helped them support diverse learners, especially the students who are frequently ‘disengaged’
  • 82% of teachers agreed that students incrased active participation in class

The full evaluation report is available from IMPAQ, Int.