In this paper, Lori Nathanson, Meghan McCormick, and James Kemple of the NYU Research Alliance for New York City Schools analyze the district’s annual school climate survey. Given every spring since 2007 to all teachers, parents, and grade 6-12 students, the survey invites respondents to evaluate their school’s academic expectations, communication, engagement, and safety/respect. Data from the questionnaires make up 10-15 percent of each school’s A, B, C, D, F Progress Report grade (the remainder of the grade comes from students’ academic achievement and progress). Last year, the surveys were completed by 476,567 parents, 428,327 students, and 62,115 teachers. Response rates have been remarkably high: 78 percent of students and 83 percent of teachers (in 2010) and 53 percent of parents (in 2012). Here are sample items from the 2012 survey, most using 5-4-3-2-1 agree/disagree scales

Parents:

-   I feel welcome in my child’s school.

-   My child’s school has high and appropriate expectations for my child.

-   My child’s school provides instruction that enables my child to reach his or her potential.

-   My child is safe at school.

-   My child’s school is clean.

-   School staff treat my child with respect.

-   There is gang activity in my child’s school.

-   How often during this school year have you received information about what your child is learning in school?

-   How often during this school year have you been invited to a workshop, program, performance, or other event at your child’s school?

-   How satisfied are you with the level of assistance your child receives when he or she needs extra help?

Teachers:

-   The principal at my school knows what’s going on in my classroom.

-   The principal at my school is an effective manager who makes the school run smoothly.

-   The principal at my school places the learning needs of children ahead of personal or political interests.

-   I feel respected by the principal at my school.

-   School leaders give me regular and helpful feedback about my teaching.

-   My school sets high standards for student work in their classrooms.

-   My school has clear measures of progress for student achievement throughout the year.

-   In my school, it’s easy to speak up about what is on your mind.

-   In my school, people are eager to share information about what does and doesn’t work.

-   In my school, we have so many different programs that I can’t keep track of them all.

-   Teachers in my school work together in teams to improve their instructional practice.

-   I usually look forward to each working day at my school.

-   I would recommend my school to parents seeking a placement for their children.

Students:

-   The adults at my school look out for me.

-   My teachers encourage me to succeed.

-   I need to work hard to get good grades at my school.

-   Teachers at my school expect me to continue my education after high school.

-   Most students at my school treat teachers with respect.

-   My teachers enjoy the subjects they teach.

-   My teachers give me extra help when I need it.

-   Most students in my school help and care about each other.

-   How often do students threaten or bully other students at school?

-   How often do adults at my school yell at students?

-   How often do students use alcohol or illegal drugs while at school?

-   I am safe in my classroom.

-   I am safe in the hallways, bathrooms, and locker rooms.

-   My school is kept clean.

-   How comfortable are you talking to teachers and other adults at your school about something that is bothering you?

Nathanson, McCormick, and Kemple drew the following conclusions from their analysis of three years of NYC surveys:

            • The surveys provided statistically reliable indicators of school climate, and the items were highly correlated with one another. In fact, the four reporting categories were statistically indistinguishable from one another. This means they could be combined into a single “school environment” measure without diminishing the current breadth of information about schools. Because many of the items were so highly correlated with one another, it would be possible to come up with a single measure of school environment with about half the number of items.

            • Survey scores were significantly associated with students’ test scores and graduation rates, but these correlations were not consistent from year to year and across students, teachers, and parents. “When there were significant associations,” say the authors, “relatively large differences in School Survey scores were associated with relatively small differences in test scores.” But in high schools, it was clear that improving aspects of school climate could increase the number of students on track and ultimately graduating.

            • Responses from teachers, parents, and students were quite distinct, reflecting their unique perspectives. A single school environment score for each group might provide richer information about a school than four separate scores (for academic expectations, communication, engagement, and safety/respect).

            • Data from teachers’ assessment of their school’s climate produced clearer distinctions between schools than data from parents and students. This led the authors to suggest that teachers’ scores should have greater weight in Progress Report grades.

            Nathanson, McCormick, and Kemple conclude that New York City’s school surveys have great potential in three areas: giving parents, students, and teachers a voice in assessing their schools; providing information for school-improvement efforts; and holding schools accountable. But some of this potential remains untapped. Here are their recommendations, some of which have already been implemented by the school department:

            • Eliminate redundant items in the surveys.

            • Reduce the time required to complete the survey by creating more-consistent question formats and response categories.

            • Choose different, separate measures for the parent, student, and teacher surveys to capitalize on their distinct perspectives on their schools.

            • Incorporate new school-environment measures that are more likely to distinguish between schools and are associated with other school-performance indicators (for example, ask teachers to assess their principal’s instructional leadership).

            • Incorporate measures that more effectively gauge parents’ satisfaction and engagement with their child’s school (for example, “I would recommend this school to other parents”).

            The authors close with the following recommendations for other school districts considering surveys of this kind:

-   Bring researchers into the process early on so surveys are valid and questions tap areas that research indicates are most influential within schools.

-   Continuously improve surveys in response to comments and suggestions, even if this makes it harder to make year-to-year comparisons.

-   Give school-environment scores more weight in school accountability. “There is good and growing evidence that a school’s environment, including safety and student and family engagement, is important for improving academic outcomes,” say the authors.

-   Broaden survey questions to assess student motivation, academic aspirations, and problem-solving; these are important correlates of student success.

-   Allow students to assess their teachers. “It may make sense,” say the authors, “to integrate student ratings, collected through a school survey, as a small part of teacher evaluation systems.”

“Strengthening Assessment of School Climate: Lessons from the NYC School Survey” by Lori Nathanson, Meghan McCormick, and James Kemple from The Research Alliance for New York City Schools, New York University, June 2013; the report and a technical compendium are at http://steinhardt.nyu.edu/research_alliance/publications/SchoolSurv...; 2012 surveys are at http://schools.nyc.gov/Accountability/tools/survey/2012surveysamples. 

From the Marshall Memo #490