We take cohorts of entering freshmen at the United States Air Force Academy and assign half to peer groups designed to maximize the academic performance of the lowest ability students. Our assignment algorithm uses nonlinear peer effects estimates from the historical pre-treatment data, in which students were randomly assigned to peer groups.
We find a negative and statistically-significant treatment effect for the students we intended to help. We provide evidence that within our “optimally” designed peer groups, students avoided the peers with whom we intended them to interact and instead formed more homogeneous subgroups.
These results illustrate how policies that manipulate peer groups for a desired social outcome can be confounded by changes in the endogenous patterns of social interactions within the group.
[Keywords: peer effects, social network formation, homophily]
…We first identify nonlinear peer effects at the United States Air Force Academy (USAFA) using pre-treatment data in which students were randomly assigned to peer groups (squadrons) of about 30 students. These estimates showed that low ability students benefited statistically-significantly from being with peers who have high SAT Verbal scores. We use these estimates to create optimally designed peer groups intended to improve academic achievement of the bottom one-third of incoming students by academic ability while not harming achievement of students at other points in the distribution. [This objective function was determined by USAFA senior leadership, who had a strong desire to reduce the academic probation rate, then at roughly 20%.] Using an experimental design, we sorted the incoming college freshman cohorts at USAFA into peer groups during the fall semesters of 2007 and 2008. Half of the students were placed in the control group and randomly assigned to squadrons, as was done with preceding entering classes. The other half of students (the treatment group) were sorted into squadrons in a manner intended to maximize the academic achievement of the students in the lowest third of the predicted grade point average (GPA) distribution. To do so, low ability students were placed into squadrons with a high fraction of peers with high SAT Verbal scores. We refer to these as bimodal squadrons. In the process, the sorting algorithm also created a set of treatment squadrons consisting largely of middle ability students. We call these homogeneous squadrons.
The reduced form coefficients (using the pre-treatment data) predicted a Pareto-improving allocation in which grades of students in the bottom third of the academic distribution would rise, on average, 0.053 grade points while students with higher predicted achievement would be unaffected. Despite this prediction, actual outcomes from the experiment yielded quite different results. For the lowest ability students, we observe a negative and statistically-significant treatment effect of −0.061 (p = 0.055). For the middle ability students, expected to be unaffected, we observe a positive and statistically-significant treatment effect of 0.082 (p = 0.041). High ability students are unaffected by the treatment.
High and low ability students in the treatment squadrons appear to have segregated themselves into separate social networks, resulting in decreased beneficial social interactions among group members. Survey responses following the experiment show that, compared to the control group, low ability students in the treatment group were much more likely to sort into study (friendship) groups with other low ability students. For the middle ability students, evidence suggests that the positive treatment effect occurred because these students did not interact with low ability students after being placed into the homogeneous squadrons.
…Well known difficulties exist in the application of policy to affect a desired outcome. General equilibrium responses as in 1976 or 2010 can undo effects predicted by more simple partial equilibrium models. Large policy interventions can also lead to political responses by actors and interest groups (). However, we see in our results a different mechanism at work; policy interventions can affect patterns of endogenous social interaction. As such, we believe that endogenous responses to large policy interventions are a major obstacle to foreseeing the effects of manipulating peer groups for a desired social outcome.
…Students in the control group were randomly assigned to one of the 20 control squadrons according to an algorithm that has been used by USAFA since the summer of 2000. The algorithm provides an even distribution of students by demographic characteristics. Specifically, the USAFA admissions office implements a stratified random assignment process where females are first randomly assigned to squadrons. Next, male ethnic and racial minorities are randomly assigned, followed by male non-minority recruited athletes. Students who attended a military preparatory school are then randomly assigned. Finally, all remaining students are randomly assigned to squadrons. Students with the same last name, including siblings, are not placed in the same squadron. This stratified process is accomplished to ensure demographic diversity across peer groups.