RAISE: Evaluation Findings

Research & Evidence

RAISE: Evaluation Findings

Summary of Impact and Scale-Up Evaluation Reports
Reading Apprenticeship Improving Secondary Education (RAISE)
i3 Validation Grant, 2010–2015

Over five years, while serving more than 1,700 high school teachers and nearly 700,000 students, Reading Apprenticeship helped 274 schools make progress on one of the country’s most crucial challenges: improving academic literacy. Research shows that the Reading Apprenticeship Improving Secondary Education (RAISE) project improved teachers’ ability to strengthen students’ reading of complex texts in science, history, and English classes — a key solution to removing barriers to college and career readiness.

Two independent evaluations demonstrate Reading Apprenticeship’s effectiveness in improving secondary literacy instruction and achievement at scale. One evaluation — a comprehensive impact study conducted by Impaq International — used a random assignment design, with 22 intervention schools and 20 control schools in Pennsylvania and California, to assess the impact of Reading Apprenticeship on teachers’ instructional practices, students’ classroom experiences, and students’ literacy skills. Evaluators used multiple data sources, including teacher and student surveys and an assessment that simulates subject-area reading and learning tasks, developed by ETS. A second study — conducted by Empirical Education — used surveys, focus groups, and case studies to identify lessons from managing the scale-up across four of the five states.

Key Findings Regarding Impact

“A student who would walk into one classroom and not engage at all walked into this [RAISE] classroom and it was amazing to see the engagement. The level of conversation and critical thinking the students were doing was impressive.”” – Eric Turman, Principal, Reading High School, PA

Teaching Practice

  • RAISE had statistically significant impacts on teachers’ use of core Reading Apprenticeship practices and on their confidence in delivering literacy instruction.
  • Intervention teachers fostered greater student independence, engaged students more often in metacognitive problem-solving processes, and provided more opportunities for students to practice comprehension strategies and collaborate to comprehend subject-area texts. Intervention teachers also reported greater confidence in their literacy instruction.

“We noticed that students were engaging with the tests a lot more, writing on passages, circling key words in science, and ‘talking to the text.’ We also noticed they were using the skills we taught in English class in science class.” – Heather Arena, Teacher Leader, Exeter High School, PA

Student Literacy Learning and Achievement

  • Students in Reading Apprenticeship classrooms reported significantly greater opportunities to share reading processes and problem solving, and indicated that reading instruction was more integrated into their content-area learning. These outcomes parallel impact findings on teacher practice.
  • Consistent with prior studies indicating that Reading Apprenticeship can improve students’ ability to comprehend and reason from subject-area texts, RAISE had a positive and statistically significant impact on student literacy in science classes, as measured by the ETS assessments. Although results for the other two subjects were not statistically significant, there was a meaningful, positive effect for ELA classrooms.
  • For all students (pooled across subject areas and grade levels) and for key subgroups (English language learners, low-income students, low prior performers, and non-White students), RAISE had positive but not statistically significant impacts on student literacy skills, as measured by the ETS assessment.

The results of this impact evaluation suggest the RAISE intervention had a significant impact on instructional practices that are the hallmark of Reading Apprenticeship — practices that support students in gaining experience and greater independence in doing the work required of them by today’s higher standards.

The strong results for science suggest that science teaching can be an especially productive area for improving student comprehension and reasoning from text, aligned with the higher standards for literacy and college and career readiness being adopted by many states. The RAISE project had greater impact with science students than with ELA or history students, and had greater impact in Pennsylvania schools than in California schools. These differential impacts provide important opportunities to improve future efforts to scale up Reading Apprenticeship — opportunities, for example, to examine the circumstances and supports that best enable the Reading Apprenticeship approach to be effective.

Key Findings Regarding Scale-Up

“The support and trust from our administrators was great. If we asked to try a new strategy, they would sit in to learn about it and then support our doing it.” – Nick Burch, Teacher Leader, Quakertown Community High School, PA

Formative Feedback

  • Teachers rated Reading Apprenticeship professional development highly, and other data indicated high levels of initial implementation and the promise of sustainability, including teachers and principals planning for implementation in the following year.
  • Teachers and administrators demonstrated high levels of buy-in and commitment, reporting that Reading Apprenticeship was an appropriate literacy approach for their classrooms and a means of improving student achievement in their schools.
  • Teachers and administrators identified competing priorities as the biggest challenge to implementation of Reading Apprenticeship.

“What I’ve learned from other teachers throughout the country has been incredible in terms of what it’s done for my classroom. Listening to others troubleshoot problems and come up with bright ideas to deepen student learning has been amazing.”” – Rob Cushman, Biology Teacher and Teacher Leader, Wyomissing Senior High School, PA

Scale-Up Processes

  • Early establishment of team meetings proved to be an important predictor of which schools would grow in their implementation of Reading Apprenticeship and which would atrophy.
  • Teachers identified collaboration with colleagues as the most effective activity in building their capacity to implement Reading Apprenticeship.
  • Over the years studied, schools fell into differential patterns of scale, with some adding additional teachers over time, others “staying the course,” and still others losing teachers involved in the initiative.

The evaluation demonstrated that the Reading Apprenticeship approach can successfully reach and positively impact large numbers of teachers and students. The RAISE project’s investment in site-based team meetings and teacher leader development proved key to the successful implementation of Reading Apprenticeship at scale.

Download the full impact and scale-up reports.