|Abstract: The field of after-school programming remains rife with unanswered questions. What constitutes quality in after-school programs? Are after-school opportunities valuable for participants regardless of their quality? Are differences in quality associated with differences in participant benefit? This sub-study of the longitudinal evaluation of The After-School Corporation (TASC) looks at how after-school opportunities with varying features affect urban middle-grades (6-8) adolescents who live in impoverished circumstances. Supported by the William T. Grant Foundation, the study explores the associations between after-school project features and the social and cognitive outcomes of disadvantaged middle-grades participants in TASC programs. The study relies on data collected during the 2001-02 and 2002-03 school years in eight TASC projects serving middle-grades students.|
Recent research on youth development increasingly concludes that, especially for disadvantaged youth, high-quality out-of-school-time opportunities are very important. These opportunities help youth overcome risk factors that might otherwise impede healthy development and also introduce positive opportunities, experiences, and supports. From this literature, it is possible to identify program features and practices that are associated with positive youth-development outcomes. For example, the National Research Council’s Committee on Community-Level Programs for Youth (Eccles & Gootman, 2002) identified eight features of positive developmental settings: physical and psychological safety; appropriate structure; supportive relationships; opportunities to belong; positive social norms; support for efficacy and mattering (making a difference); opportunities for skill-building; and integration of family, school, and community efforts. Likewise, existing evidence indicates that structural features of after-school programs affect staff practices. For example, Rosenthal and Vandell (1996) found that
· higher child-staff ratios are associated with more negative staff-child interactions;
· larger group sizes are associated with lower child ratings of program climate, emotional support, and support for autonomy and privacy;
· higher levels of staff education are associated with fewer negative staff-child interactions.
This study explored the associations between after-school project features and the social and cognitive outcomes of disadvantaged middle-grades (grades 6-8) participants in after-school programs supported by The After-School Corporation (TASC) in New York City. Middle-grades youth may be receptive to external supports and opportunities because they are old enough to understand and pursue their own interests, but young enough to change course easily toward a more positive future if persuaded of the value of doing so. Evidence from sources such as the TASC evaluation suggests that after-school services can particularly benefit this age group by promoting high levels of school attachment (as measured by school attendance) and, to a lesser extent, improvements in achievement (Reisner, White, Russell & Birmingham, 2004).
We identified for this study five dimensions of cognitive and social outcomes: students’ attachment to the TASC project, relationships between youth and adults, peer relationships, cognitive development, and attachment to school. For each of these five outcome areas, we explored associations with the following features that can directly shape the out-of-school time experiences of youth and provide the structure necessary to maintain high-quality relationships and activities:
· Practices to promote positive relationships , including positive staff-youth relationships, positive peer relationships, and connections with families and the community
· Rich content-based program activities , including a mix of academic and non-academic (physical and recreational) enrichment activities that build skills
· Learning- and mastery-oriented content-delivery strategies that provide both structured and unstructured learning opportunities and promote participant autonomy, choice, and leadership
· Staff qualifications and support , including staff education and training, expertise, turnover, and supports
· Group size and configuration , including youth-staff ratio and group size
· Program resources , including financial resources, space and facilities, equipment and materials, and accessible location
· Program partnerships , linkages, and connections, including relationships with parents, participants’ schools, communities, and membership in a larger network of programs
The process and content features identified by the study as core elements of effective programs reflect evidence from youth-development research and also from teaching and learning research about the content and instructional strategies that promote learning. McLaughlin (2000) observed that after-school programs that capture youths’ interest and promote their learning are “not happenstance.” Instead, positive outcomes emerge when adults deliberately create opportunities in which both the content of activities and the instructional processes are “knowledge-centered” and “youth-centered.” More generally, researchers investigating human learning point to the importance of providing learners with rich content-based experiences, led by instructors or coaches who encourage mastery and use both structured and unstructured teaching strategies to promote learning (Bransford, Brown, & Cocking, 1999).
Data was collected from eight TASC after-school projects that served middle-grades youth in the 2002-03 school year. The eight projects in this study had operated since at least 1998-99 in space provided by New York City public schools. All eight projects employed TASC’s model of program services, including sponsorship and operation by a community-based or other nonprofit organization, employment of a full-time project coordinator, regular communications between the after-school project and the host school, extensive opportunities for staff development, and focus on participants’ academic and social growth. However, the projects varied in terms of project goals, approaches, and services offered in ways that this study hypothesized were related to students’ cognitive and social outcomes. For example, some projects offered a comprehensive list of activities from which participants could choose, including arts-based and academic enrichment activities focused on mastery. In contrast, other projects offered a more traditional after-school curriculum of homework help supplemented by sports or games.
The eight schools hosting the TASC projects in this study served some of the most disadvantaged middle-grades students in New York City, as illustrated in Exhibit I. In particular, more students in these schools than citywide were eligible for free- or reduced-price lunch and were non-white. Fewer than a third of the students in the schools performed at grade level on the city and state English Language Arts (ELA) and math assessments.
Characteristics of All New York City Public Middle Schools,
Schools Hosting TASC Study Sites, and TASC Participants,
2001-02, in Percents
Exhibit reads: Eighty-three percent of TASC project participants in the study qualified for free-or reduced-price lunch, compared to 70 percent of middle-school students citywide and 78 percent of the students enrolled in the study schools.
a Citywide figures are from schools designated by the New York City Department of Education as middle schools.
b Participant-level data are not available for one project.
Data collectionSurveys were administered to program participants (N=399 in eight projects) in spring 2003, and to site coordinators (N=8), host school principals (N=6), and program staff (N=126 in seven projects) in spring 2002. We also relied on the student information system of the New York City Department of Education (DOE) for data on school attendance, end-of-year achievement test scores, and participant characteristics. Analyses focused on changes in student-level educational-performance data between the 2000-01 and 2001-02 school years. These data were available for 726 students in English Language Arts and 853 students in mathematics in seven of the eight study sites. TASC program attendance records for 2001-02
were available for 1,219 participants in seven of the eight study sites. In addition, we conducted site visits to each of the eight projects during the 2002-03 school year, which included structured observations of program activities and interviews with site coordinators.
Data analysis focused on identifying practices that varied across sites in order to determine the particular practices that were associated with achieving a notably more positive outcome at one or more sites relative to other sites.
To measure variation across sites, we examined the distribution of indicators of each outcome, program practice, or project characteristic to determine its prevalence in the eight middle-grades projects. If the indicator was not prevalent in at least 20 percent of cases, it was not considered for future analysis, because it was not sufficiently common to warrant a search for patterns of association. Conversely, an indicator variable was dropped if it was present in more than 80 percent of the cases because it was deemed to be too prevalent to permit distinctions among sites. Analysis of survey responses categorized the mean student response to each attitudinal scale as a positive response if respondents rated the items in the scale at the midpoint of the scale range or higher (e.g., on a scale where the possible scores ranged from 4 to 16, a mean response of 10 or higher was classified as positive).
To establish the variation across sites and enhance the interpretation of the differences in data by site, we compared the prevalence of the indicator at each site to its prevalence across the other seven sites in order to compute an effect size. When differences were statistically significant, we interpreted an effect size of +0.20 as a notably positive difference between an individual project and the other projects, and -0.20 as a notably negative difference.
We then used these effect-size calculations to look for patterns across sites and to determine the project practices and characteristics associated with particular youth outcomes. For each outcome measure analyzed, we looked for practices and characteristics that were common (based on a notable effect size, or the presence or absence of a dichotomous variable) to the projects demonstrating a notably positive effect size on that outcome and, in particular, practices and characteristics that were present in those projects but not in projects with notably negative effect sizes on the outcome.
An effect size estimates the size or importance of differences. Statistical significance assesses whether there is a difference that is greater than would be expected by chance. However, when large samples are used, minor differences can meet the threshold of statistical significance. The study team used differing methods to estimate the effect size for continuous measures and for dichotomous measures, as appropriate (Lipsey & Wilson, 2001). The statistical literature contains extensive discussion about how to interpret effect sizes of different magnitudes. The standard works suggest that an effect size of 0.20 is small, 0.50 moderate, and 0.80 large (Cohen, 1977). However, some researchers have pointed to the need to calibrate the interpretation of effect sizes to the expected impact of the program being studied. These authors often point to the medical study of the benefits of aspirin in reducing heart attacks, where the effect size was 0.03, yet was deemed important enough to influence health policy (Prentice & Miller, 1992). This study has adopted a threshold of 0.10 for a small effect size in analysis of the association between participation in a TASC after-school project and changes in school attendance.
The eight projects included in this study varied in important ways on each of the student outcome measures analyzed, as summarized in Exhibit 2.
Variation on Participant Outcome Measures
a For all outcomes except school attendance, notably positive is defined as a difference with a
Z-score that is statistically significant at the p<0.05 level and an effect size of +.20 or greater. Notably negative is defined as a statistically significant difference with an effect size of -.20 or less. For school attendance, +.10 and -.10 were considered notably positive and notably negative effect sizes.
b,c Gains on assessments are measured in terms of changes in the percent of the possible scale score points between 2000-01 and 2001-02.
d Change in school attendance rates is measured for participants between 2000-01 and 2001-02.
We measured attachment to the TASC program through program attendance rates and an attitudinal scale measuring students’ perceptions of the sense of community in the program (developed by the Child Development Project, Developmental Studies Center). In the after-school projects where middle-grades students demonstrated notably positive attachment to the TASC program, we found certain common policies and practices. In particular, project staff set clear goals and expectations for students, encouraging them to take ownership of their after-school experience. The projects also set policies that encouraged regular program attendance, and offered a rich array of activities to foster student engagement. These projects gave participants opportunities to choose activities and supported social development activities such as conflict resolution and life skills instruction. In addition, these projects enjoyed a strong relationship with their host school, in which after-school staff, for example, discussed student progress with school-day teachers and involved the school community in after-school events. One site coordinator summed up this philosophy by saying, “It’s important that kids see you as a resource in the school–as a part of the school, but something different.”
Staff instructional practices were important in the after-school projects that demonstrated notably positive staff-youth relationships, as measured through students’ reports of their level of trust of the after-school staff (survey scale developed by the Child Development Project, Development Studies Center) and through study team observations of constructive interactions between staff and students. We found positive staff-youth relationships in sites where the project staff modeled positive behavior for participants and actively promoted student mastery of the skills or concepts presented in activities. In these sites, project staff listened attentively to participants and frequently provided individualized feedback and guidance during project activities.
Participating in after-school projects provides middle-grades youth the opportunity to socialize and develop friendships, opportunities that may otherwise be in short supply. One site coordinator noted that, “Kids are less on edge about friendship groups and who they hang out with. I really feel it is a safe haven in that way.” We measured peer relationships through observations of cooperative and friendly interactions between youth, and through a peer aggression attitudinal survey scale (modified from Orpinas & Frankowski, 2001 ). In the after-school projects with notably positive peer relationships, project activities regularly included social development and athletic activities that provided students with the opportunity to interact in informal team-oriented ways. These projects often combined instruction in athletic skills with encouragement of positive behaviors such as self-discipline. In addition, project staff established clear expectations for interactions that were mature and respectful. One site coordinator emphasized that he communicates to youth that “this is their program. You work on it. It’s letting kids understand that it is not us that dictate everything that goes on in the program.”
We measured cognitive development outcomes through participant reports of academic benefits and through analyses of student performance on the New York city and state mathematics and ELA assessments. We used a statistical model to estimate the difference between a students’ expected and actual change in performance from 2000-01 to 2001-02, controlling for demographic characteristics as well as baseline achievement. The after-school projects in which participants experienced the most positive cognitive development outcomes tended to have an especially strong relationship with the host school, in some cases sharing staff. For instance, in one project, a school dean served as an assistant director of the after-school program, creating continuity in disciplinary expectations as well as in academics. Her familiarity with the school-day curriculum enabled her to advise after-school staff on the types of homework help that would be most beneficial to students. In addition, these projects offered enriched learning opportunities that were different from but complementary to the regular school day, including project-based learning activities. In each of these projects, the site coordinator also required most or all staff to submit lesson plans on a regular basis, thus creating a system for monitoring and improving program quality.
To estimate the relationship between participation in a TASC project and attachment to school, analyses examined whether the gap between after-school participants’ and non-participants’ school attendance rates increased between 2000-01 and 2001-02. (The attendance data were weighted to adjust for differences in distribution among grade levels between after-school participants and non-participants.) Projects that maintained a strong relationship to the host school and that offered hands-on learning enrichment activities were most successful in encouraging participants’ attachment to school, as measured through this analysis of school attendance. These projects generally had seamless transitions between the school-day and after-school, including regular collaboration with school-day teachers and sharing of staff. The projects also offered activities that showed participants how academics related to real-life experiences.
The distribution of scale scores on the math and ELA tests administered in New York City is neither identical across grade levels nor does it follow a regular progression. Therefore, to facilitate analysis, the study team standardized the scale scores across grades, so that the range of possible test scores extended from 0 to 100 at each grade level and the mid-point of the possible scale scores for each grade level was always 0.50. For more information, please refer to Reisner et al., 2004.
Patterns of associations measured in this study revealed four key features of after-school programming that were consistently related to positive outcomes for middle-grades students:
· Skilled and caring staff . In the study sites, project staff played a central role in engaging middle-grades students and promoting their social and cognitive development. Projects fostered a sense of community and positive peer and youth-adult relationships in sites where staff members established clear goals and high expectations for mastery while modeling positive behavior. Sites with experienced, qualified after-school staff were assessed as yielding strong academic benefits and encouraging students’ attachment to school.
· Student choice. All eight projects included in this study offered a variety of homework, enrichment, and recreational activities. However, the projects where students displayed the strongest attachment to the after-school program were also those that offered participants the opportunity to help design their own after-school experience by choosing activities, which in turn helped to create a sense of ownership and belonging within the program.
· Enrichment activities, including social development and athletic opportunities. In the projects where students demonstrated notably positive cognitive development outcomes, they were exposed to enrichment activities that frequently included hands-on learning, which complemented school-day academics. In addition, projects encouraged attachment to the after-school program and positive peer relationships through social development and athletic activities.
· Leadership that promotes a strong relationship between the after-school project and the host school. The experiences of the eight projects in this study suggest that stability in school and in project leadership were associated with support for student learning and development. In particular, experienced leadership led to a more intentional focus on integrating school-day and after-school programs, for example, by aligning policies, providing some continuity in staffing, and developing after-school activities that supported but differed from the school-day curriculum. In projects with these strong relationships to the host school, students showed evidence of notable achievement gains as well as a stronger attachment to both the after-school program and the school.
Relationships Between Student Outcomes and Project Features
Overall, the findings from this study suggest that after-school projects that serve middle-grades students can contribute to positive student outcomes by promoting program effectiveness through staffing decisions, student choice, high-quality enrichment activities, and leadership that promotes collaborative relationships. These elements of project quality create a supportive environment for student learning and development, and encourage the engagement of middle-grades students in their after-school program, in turn leading to socialand cognitive benefits.