A Tale of Good Research, Strong Theory and Human Nature
by Barbara Elmore
The idea for improving the scholastic performance of lower-ability students at a prestigious military academy sounded excellent to James West and his research partners. They had strong research behind them that suggested success. They had the support of the U.S. Air Force Academy (USAFA), and an environment in which freshmen are controlled. Also, West was teaching there.
The researchers drew up a careful design. They randomly assigned 2007 and 2008 incoming freshman students to peer groups with varying levels of ability. Their ultimate goal was to reduce the Academy's 20 percent scholastic probation rate.
The cost of the projected improvement? An attractive, fat goose egg. In theory the project should have soared. But well-formed theory crashed hard into cold reality. Theory lost. Human nature prevailed. Or as West puts it, chuckling, "People have a mind of their own."
"We were wild-eyed and optimistic," West said. "We were sure we were going to improve the grades of the lower-ability folks. What we wanted to do was go in and change these groups slightly and drastically reduce the number of cadets on academic probation, and be heroes."
West, now a Baylor professor, says he and his research colleagues, Scott Carrell of the University of California-Davis, and Bruce Sacerdote of Dartmouth, were "dumbfounded" by the results. Still, West is sanguine and he and his colleagues are making lemonade from lemons.
The first sip of lemonade is a published paper. Although writing about research that fails to hit its intended mark is rare, an editor of the prestigious journal Econometrica invited them to submit after hearing a presentation by Carrell. The result is "From Natural Variation to Optimal Policy? The Importance of Endogenous Peer Group Formation."
"We felt very blessed to get into the journal," West said. Econometrica has a strong reputation for premier technical research and, as West puts it, "panache."
West, holder of the W.H. Smith Professorship in Economics, arrived at Baylor in 2011 from the USAFA, where he was a civilian professor of Economics. He, Carrell and Sacerdote believed if mixing cadets of differing performance levels to raise GPAs would work anywhere, it would be most likely to succeed at the Academy.
When Air Force Academy freshmen arrive, they normally go into one of 40 military squadrons with about 32 freshmen in each squadron. For the "endogenous group" study, half of the arriving cadets from 2007 and 2008 went into randomly assigned squadrons as freshmen normally do.
The other half went into groups designed by the researchers to maximize the academic achievement of the students in the lowest third of the predicted GPA distribution. Thus, low-ability students went into squadrons with peers who had scored high on the verbal portion of the SAT.
To engineer this, the researchers obtained a list of class members. They determined into which squadron each student would go. The project was a secret, West said. "If any knowledge had slipped out, it would have ruined the experiment."
West had good reason to believe in success. Previous research shows that if low-performing students spend their spare time with students who have SAT verbal scores one standard deviation higher, that correlates to raising GPA by about half a letter grade. "That's a large effect-huge," West noted.
The "endogenous groups" the researchers describe in their paper are small circles of five or fewer friends within a squadron. "What we mean by ‘endogenous' is the process of selecting the smaller group from within the larger group," West said.
Because they were certain the experiment would work, the researchers wrote an early draft about the project. "Raising achievement" was in the original title. Notably, the paper has gone through at least half a dozen different titles since.
The researchers used the Air Force Academy because West was a faculty member and because they saw the military institution as "the ideal environment" because of standardized classes and limited social contact of freshmen outside their military squadrons. In the final rewrite of the paper, the editor pushed the collaborators beyond the issue of an unsuccessful experiment to whether or not such an experiment could ever succeed. The thinking was, "if an experiment like this should have worked successfully, it should have worked at the Academy," West said.
One of the most important lessons coming from the project is the difficulty of explaining changing behavior. But the Econometrica paper does not discuss that idea much because the researchers did not want to stray far from their field of economics. Still, it's an important lesson.
"Coming from all the big experiments that have happened in this country in the last 40 or 50 years is the idea that if we can change peer groups, we can change achievement," West said. That belief supports activities like student busing and institutions like private schools. "We wanted to design an experiment that could carefully measure the effect of changing group peers upon student achievement. We had a clean experimental design. We could not conceive beforehand what could possibly go wrong. But people can undo what you intend."
This apparently is true even when the intentions are backed by good research. For example, in the field of economics, peer effects-the influence of peers on others in the group-has been an active subject for over 10 years. Sacerdote published the first major paper on the topic in 2001.
"We knew that substantial peer effects existed at the Air Force Academy," West said. "So we estimated that if you were in a squadron that had smarter people, you would do better. So what if we deliberately tried to surround lower-ability people with lots of smart kids-would we boost their performance? It's a natural question."
The researchers forecast that on average, the experiment would raise GPAs by .055-just large enough to be able to measure the increase if it existed.
"As we have gotten deeper into it and others saw our presentation, we are wondering if it can be done," West said. "You can't raise the GPA of lower-ability cadets by surrounding them with smarter ones. That was the big-picture takeaway. If they have freedom to choose whom they want to study with and we surround them with smart study partners and they choose not to study with them, there is not much else we can do. It appears it is not going to work."
Or as the paper puts it, "We conclude that social processes are so rich and complex that one needs a deep understanding of their formation before one can formulate optimal policy."
The experiment's failure did not stop the research, however. West and Carrell, who formerly was a military professor at the Academy, have produced nine or 10 co-authored papers. The experiment continued through the cadets' senior year, and the collaborators have plentiful data.
"This paper deals only with what happened in the cadets' first year," West said. "We can look at longer-term effects."