Reading Apprenticeship Across the Disciplines (RAAD): Evaluation

Research & Evidence

Reading Apprenticeship Across the Disciplines (RAAD)

Overview and Context of the RAAD Evaluation

 

RAAD included extensive evaluation components to provide project leaders with formative and summative data for the purposes of program improvement and impact assessment.

I. Context

Prior studies of Reading Apprenticeship examined the impact of our discipline-specific teacher professional learning, meaning science teachers learned primarily with other science teachers about science literacy, English teachers with fellow ELA teachers, and so on. However, most districts contract with WestEd for cross-disciplinary Reading Apprenticeship, and for fewer days than are afforded by grant funding. Thus, in RAAD we studied a model of Reading Apprenticeship that is more feasible and widely used:

  • Less teacher PD time. Prior studies had five to ten days of in-person PD, whereas RAAD had five days in-person.
  • Cross-disciplinary PD sessions included teachers from all subject areas.
  • School year teacher support via online learning, both synchronous and asynchronous.
  • Refined support for teacher leaders, including regional Teacher Leader Meeting agendas, materials for teachers to lead school team meetings, and a 15-hour an online course called “Leading for Literacy.”
  • Local educational agencies as coordinators and leaders of the regional Teacher Leader meetings.

II. Impact and Formative Assessment Studies

RAAD included extensive evaluation activities to assess the project’s impact and improve implementation, including: (1) a randomized controlled trial (RCT) conducted by IMPAQ Int. to examine teacher practice change and student achievement (40 school sample from California, New York, Texas, and Wisconsin); (2) a Formative Assessment study conducted by WestEd which examined all 500+ schools in all states and included: (a) a case study of a large urban district; (b) surveys of all teachers and focus groups with Teacher Leaders; and, (c) an analysis of online professional learning communities designed to support teacher practice over the school year.

Highlights of RAAD outcomes, accomplishments, and contributions to the field follow.

Goal 1: Increase the number of highly effective middle and high school teachers serving high needs students by engaging 2,000 teachers in Reading Apprenticeship professional learning.

We exceeded our participation objectives and engaged 2,240 teachers from 570 schools in six states (California, Illinois, Michigan, New York, Texas, and Wisconsin) in evidence based professional learning, thus improving the numbers of effective teachers in the United States.

Featured Teacher Outcomes:

  • The RCT found significant positive impacts on some teacher literacy practices, including modeling of collaboration and differentiated instruction, and that high participation in the PD correlated with high implementation.
  • Among the 1,333 teachers who participated in the spring formative assessment teacher surveys, 91.9% of Cohort 1 and 79.9% of Cohort 2 teachers attended all five days of the RAAD face-to-face institutes. Over 85% of these teachers rated institutes as effective or highly effective.
  • Teachers’ satisfaction levels with online PLCs was more varied, and many reported that the PLCs were less helpful than face-to-face PD. WestEd studied PLCs closely to target our improvement efforts and conducted additional facilitator training to improve PLC quality and consistency.
  • Teacher and student uptake of Reading Apprenticeship practices differed by subject areas, both among the subjects and as compared to prior studies that focused on subject-specific PD. In RAAD, the ELA teachers modified their practice more than STEM teachers, whereas in prior studies of Reading Apprenticeship, science teachers and students made the biggest changes.

GOAL 2: Improve high needs middle and high school students’ reading comprehension, academic achievement, metacognitive skills, and positive learning dispositions.

RAAD teachers served 231,400 secondary students during the grant period, and the majority of teachers (83%) reported that student learning improved as a result of their participation in the PD. Seventy percent of the participating teachers served high need students.

Featured Student Outcomes:

  • The RCT found no impact on student’s reading identity or standardized test scores.
  • The RCT found significant impacts on: (1) teacher and student collaboration practices; (2) student use of reading strategies; (3) student use of metacognitive inquiry strategies.
  • In interviews and focus groups, teachers reported that improvement in student engagement and reading lead to improved student learning and confidence.
  • Most teachers attributed the improved student confidence to the fact that students felt part of the learning process, a strong indicator that many teachers were able to turn active learning over to students.

Goal 3: Build local capacity for strong and sustained implementation of effective academic literacy practices through school and regional network support.

Featured Capacity-Building and Sustainability Outcomes:

  • Expanded and diversified facilitator pool. A key factor in implementing PD with fidelity at scale, just like classroom teaching, is the quality and diversity of the facilitators. With the RAAD grant, WestEd was able to develop ten new certified Reading Apprenticeship facilitators from the participating states. Each of these facilitators leads evidence-based PD in their own region and across the country.
  • The local support offered by Regional Partners varied given their different organizational roles, experience level, and competing initiatives. Identifying ways for their staff to quickly instantiate Reading Apprenticeship principles of inquiry-based teacher learning into their local contexts is key to sustaining the work over time, but takes more than one year of experience.
  • Teacher Leaders Had Stronger Uptake and Impact. Teacher Leaders were school site teachers who attended additional PD and built relationships with their school colleagues, one another across school sites, and with their Regional Partner. They participated in about five days additional PD than the average teacher, and the formative assessment work and RCT both found that Teacher Leaders and their students reported more frequent implementation of Reading Apprenticeship routines.

III. Conclusion and Contributions to the Field

The project findings align with current research which shows that the duration, content focus, and modality of PD impact the level of practice change and student learning outcomes. The evaluations raise questions as to whether shortened PD in cross-subject teacher groups is of enough intensity and duration to shift teachers’ practices and lift student outcomes at the significance level seen in prior studies of subject-specific models of Reading Apprenticeship.

In addition to the many educators who directly benefitted from the professional learning and leadership development, RAAD has impacted the field of teaching and learning through extensive dialogue and dissemination of this work in peer-reviewed research and practitioner publications, presentations, conferences, websites, and social media. School and district implementation and success stories have appeared on www.ReadingApprenticeship.org. Below are a few of the more than 25 publications and presentations informed by or featuring RAAD.

A. Publications During the Grant Period

Goldman, S., Britt, M.A., Brown, W., Cribb, G., George, M., Greenleaf, C., Lee, C.D., Shanahan, C., & Project READI. (2016) Disciplinary literacies and learning to read for understanding: A conceptual framework for disciplinary literacy. Educational Psychologist, 51(2), 219-246. http://dx.doi.org/10.1080/00461520.2016.1168741.

Goldman, S.R., Greenleaf, C., Mariya Yukhymenko-Lescroart with Willard Brown, Mon – Lin Monica Ko, Julia Emig, MariAnne George, Patricia Wallace, Dylan Blum, M. Anne Britt, and Project READI. (2019) Explanatory modeling in science through text-based investigation. American Educational Research Journal, 56(4), 1148-1216. DOI: 10.3102/0002831219831041.

Greenleaf, C. & Katz, M. L. (2019) “Releasing responsibility for what? Developing learning environments for text-based inquiry in the disciplines in secondary schools. In M. McVee, Ortlieb, E. Reichenber, J. & P.D. Pearson (Eds.), The Gradual Release of Responsibility in Literacy Research and Practice (pp 37-52). Emerald Press.

Greenleaf, C., Litman, C. & Marple, S. (2018) The impact of inquiry-based professional development on teachers’ capacity to integrate literacy instruction in secondary subject areas. Teaching and Teacher Education, 71, 226-240.

Greenleaf, C. & Valencia, S. Missing in action: Learning from texts in subject-matter classrooms. In K.A. Hinchman & D.A. Appelman, (Eds.) (2016). Adolescent literacy: A handbook of practice-based research (pp. 135 – 155). NY: Guilford Press.

Katz, M.L., Stump, M., Charney-Sirott, I., and Howlett, H. (September/October 2019). “Traveling with Integrity: Translating Face-to-Face Teacher Professional Learning to Blended Spaces.” Journal of Adolescent and Adult Literacy, Vol. 63(2): pp. 217-223. https://doi.org/10.1002/jaal.976

Schoenbach, R., Greenleaf, C., & Murphy, L. (2016) Leading for literacy: A Reading Apprenticeship approach. San Francisco: Jossey-Bass Wiley.

Schoenbach, R. & Greenleaf, C. (2017) Leading for literacy: Engaging schools and districts in transforming subject-area literacy. Phi Delta Kappan, 9 (3), 59-64.

B. Select Presentations During the Grant Period

Greenleaf, C. “Apprenticing Adolescents to Academic Literacy in the Subject Areas: The Reading Apprenticeship Framework.” (2015) Panel, Wisconsin State Reading Association, Milwaukee, WI.

Greenleaf, C. “All that Matters: Building Learning Culture, Learner Dispositions, and Literacy Identities” Keynote, (2015), Wisconsin State Reading Association, Milwaukee, WI.

Katz, M.L., with Anne Pagel, Darcy Budnik, Alissa Davis, and Brenda Vela. (February 8, 2018). “Equipping Students for Disciplinary Literacy Success Through Reading Apprenticeship Across the Disciplines.” Panel at Wisconsin State Reading Association Conference, Milwaukee.

Katz, M. and Loyd W. “Spotlight on State and Local Initiatives: WestEd, SLI Reading Apprenticeship Coaching Model.” SEED Teacher Feedback and Coaching Online Community of Practice. (July and October, 2017)

Stump, M and Greenleaf C. Engineering Solutions to High Level Literacy Instruction Across the Disciplines: The Role of Data. (May 23, 2016). TQP Annual Conference, Washington D.C.

 

Download IMPAQ Report on the Randomized Controlled Trial

Download WestEd’s Formative Assessment Report