The Case of the Amazing Gay-Marriage Data: How a Graduate StudentReluctantly Uncovered a Huge Scientific Fraud
By Jesse Singal
The exposure of one of the biggest scientific frauds in recent memory didn’t start with concerns about normally distributed data, or the test-retest reliability of feelings thermometers, or anonymous Stata output on shady message boards, or any of the other statistically complex details that would make it such a bizarre and explosive scandal. Rather, it started in the most unremarkable way possible: with a graduate student trying to figure out a money issue.
It was September of 2013, and David Broockman (pronounced “brock-man”), then a third-year political-science doctoral student at UC Berkeley, was blown away by some early results published by Michael LaCour, a political- science grad student at UCLA. On the first of the month, LaCour had invited Broockman, who is originally from Austin, Texas, to breakfast during the American Political Science Association’s annual meeting in Chicago. The pair met in a café called Freshii at the Palmer House Hilton, where the conference was taking place, and LaCour showed Broockman some early results on an iPad.
The iPad thing was LaCour’s trademark. “He was sort of famous for taking his results from different studies he was working on, putting them on an iPad, and buttonholing people at the conferences and going over all of the research that he was doing, the different findings he had, and basically not letting the people go until they had an idea of what he was working on,” says Tim Groeling, a communications professor at UCLA, who is listed as one of LaCour’s references on his curriculum vitae. “It was infectious,” continues Groeling. “Really cool stuff was on
By Jesse Singal
The exposure of one of the biggest scientific frauds in recent memory didn’t start with concerns about normally distributed data, or the test-retest reliability of feelings thermometers, or anonymous Stata output on shady message boards, or any of the other statistically complex details that would make it such a bizarre and explosive scandal. Rather, it started in the most unremarkable way possible: with a graduate student trying to figure out a money issue.
It was September of 2013, and David Broockman (pronounced “brock-man”), then a third-year political-science doctoral student at UC Berkeley, was blown away by some early results published by Michael LaCour, a political- science grad student at UCLA. On the first of the month, LaCour had invited Broockman, who is originally from Austin, Texas, to breakfast during the American Political Science Association’s annual meeting in Chicago. The pair met in a café called Freshii at the Palmer House Hilton, where the conference was taking place, and LaCour showed Broockman some early results on an iPad.
The iPad thing was LaCour’s trademark. “He was sort of famous for taking his results from different studies he was working on, putting them on an iPad, and buttonholing people at the conferences and going over all of the research that he was doing, the different findings he had, and basically not letting the people go until they had an idea of what he was working on,” says Tim Groeling, a communications professor at UCLA, who is listed as one of LaCour’s references on his curriculum vitae. “It was infectious,” continues Groeling. “Really cool stuff was on
David Broockman in front of the Golden
Gate Bridge on the day he was offered a
job at the Stanford Graduate School of
Business Photo: Courtesy of David
Broockman
The results LaCour showed Broockman were, in fact, very cool, and
like everyone else who had come across them, Broockman instantly
knew they would be a hit. LaCour’s research involved dispatching
canvassers to speak with California voters at their homes. He
reported that a brief conversation about marriage equality with a
canvasser who revealed that he or she was gay had a big, lasting
effect on the voters’ views, as measured by separate online surveys
administered before and after the conversation. LaCour told
Broockman that he planned on getting a big name on the paper in
progress: Donald Green, a highly respected political-science
professor at Columbia who was also Broockman’s undergraduate
adviser at Yale.
that iPad.”
Part of why LaCour’s results were so noteworthy was that they flew in the face of just about every established
tenet of political persuasion. While past research had shown canvassing can be effective at influencing people in
certain ways, the sheer magnitude of effect LaCour had found in his study simply doesn’t happen — piles of
previous research had shown that, all else being equal, human beings cling dearly to their political beliefs, and
even when you can nudge them an inch to the left or right, people’s views are likely to snap back into place
shortly after they hear whatever message you’ve so carefully and cleverly crafted. Not so in this case: When
LaCour compared the before-and-after views on gay marriage in his study, he found that opinions had shifted
about the distance between the average Georgian and the average Massachusettsian, and this effect appeared to
have persisted for months.
So when LaCour and Green’s research was eventually published in December 2014 in Science, one of the leading peer-reviewed research publications in the world, it resonated far and wide. “When contact changes minds: an expression of transmission of support for gay equality (http://www.sciencemag.org/content/346/6215/1366) ” garnered attention in the New York (http://www.nytimes.com/2014/12/12/health/gay-marriage-canvassing-study-science.html?_r=0) Times (http://www.nytimes.com/2014/12/12/health/gay-marriage-canvassing-study-science.html?_r=0) and a segment on (http://www.thisamericanlife.org/radio-archives/episode/555/the-incredible-rarity-of-changing- your-mind) "This American Life (http://www.thisamericanlife.org/radio-archives/episode/555/the- incredible-rarity-of-changing-your-mind) " in which a reporter tagged along with canvassers as they told heart- wrenching stories about being gay. It rerouted countless researchers’ agendas, inspired activists to change their approach to voter outreach, generated shifts in grant funding, and launched follow-up experiments.
But back in 2013, the now-26-year-old Broockman, a self-identifying “political science nerd,” was so impressed by LaCour’s study that he wanted to run his own version of it with his own canvassers and his own survey sample. First, the budget-conscious Broockman had to figure out how much such an enterprise might cost. He did some back-of-the-envelope calculations based on what he’d seen on LaCour’s iPad — specifically, that the survey involved about 10,000 respondents who were paid about $100 apiece — and out popped an imposing
So when LaCour and Green’s research was eventually published in December 2014 in Science, one of the leading peer-reviewed research publications in the world, it resonated far and wide. “When contact changes minds: an expression of transmission of support for gay equality (http://www.sciencemag.org/content/346/6215/1366) ” garnered attention in the New York (http://www.nytimes.com/2014/12/12/health/gay-marriage-canvassing-study-science.html?_r=0) Times (http://www.nytimes.com/2014/12/12/health/gay-marriage-canvassing-study-science.html?_r=0) and a segment on (http://www.thisamericanlife.org/radio-archives/episode/555/the-incredible-rarity-of-changing- your-mind) "This American Life (http://www.thisamericanlife.org/radio-archives/episode/555/the- incredible-rarity-of-changing-your-mind) " in which a reporter tagged along with canvassers as they told heart- wrenching stories about being gay. It rerouted countless researchers’ agendas, inspired activists to change their approach to voter outreach, generated shifts in grant funding, and launched follow-up experiments.
But back in 2013, the now-26-year-old Broockman, a self-identifying “political science nerd,” was so impressed by LaCour’s study that he wanted to run his own version of it with his own canvassers and his own survey sample. First, the budget-conscious Broockman had to figure out how much such an enterprise might cost. He did some back-of-the-envelope calculations based on what he’d seen on LaCour’s iPad — specifically, that the survey involved about 10,000 respondents who were paid about $100 apiece — and out popped an imposing
number: $1 million. That can’t be right, he thought to himself. There’s no way LaCour — no way any grad
student, save one who’s independently wealthy and self-funded — could possibly run a study that cost so much.
He sent out a Request for Proposal to a bunch of polling firms, describing the survey he wanted to run and asking
how much it would cost. Most of them said that they couldn’t pull off that sort of study at all, and definitely not
for a cost that fell within a graduate researcher’s budget. It didn’t make sense. What was LaCour’s secret?
Eventually, Broockman’s answer to that question would take LaCour down. ***
Last week, Broockman, along with his friend and fellow UC Berkeley graduate student Josh Kalla and Yale University political scientist Peter Aronow, released an explosive 27-page report (http://stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf) recounting many “irregularities” in LaCour and Green’s paper. “Irregularities” is diplomatic phrasing; what the trio found was that there’s no evidence LaCour ever actually collaborated with uSamp, the survey firm he claimed to have worked with to produce his data, and that he most likely didn’t commission any surveys whatsoever. Instead, he took a preexisting dataset, pawned it off as his own, and faked the persuasion “effects” of the canvassing. It’s the sort of brazen data fraud you just don’t see that often, especially in a journal like Science. Green quickly fired off an email to the journal asking for a retraction; Science granted that wish yesterday, albeit without LaCour’s consent (http://news.sciencemag.org/policy/2015/05/science-retracts-gay-marriage-paper-without-lead- author-s-consent) . And while there’s no word out of central New Jersey just yet, there’s a good chance, once the legal dust settles, that Princeton University will figure out a way to rescind the job offer it extended to LaCour, who was supposed to start in July. (Princeton offered no comment other than an emailed statement: “We will review all available information and determine the next steps.”) LaCour, for his part, has lawyered up and isn’t talking to the media, although he was caught attempting to cover up faked elements of his curriculum vitae (http://nymag.com/scienceofus/2015/05/michael-lacour-made-up-a-teaching-award-too.html) earlier this week. His website claims (http://www.google.com/url? q=http%3A%2F%2Fwww.mikelacour.com%2F&sa=D&sntz=1&usg=AFQjCNE_D- LD79PJ6BWSHaMtWuRMmDGugA) that he will “supply a definitive response” by the end of the day today.
But even before Broockman, Kalla, and Aronow published their report, LaCour’s results were so impressive that,
on their face, they didn’t make sense. Jon Krosnick, a Stanford social psychologist who focuses on attitude
change and also works on issues of scientific transparency (http://www.google.com/url? q=http%3A%2F%2Fbps.stanford.edu%2F%3Fpage_id%3D987&sa=D&sntz=1&usg=AFQjCNGaX1UKo6ek6Ro7kCgE80qoxVro8A) , says that he hadn’t heard about the study until he was contacted by a "This American Life" producer who
described the results to him over the phone. “Gee,” he replied, “that's very surprising and doesn't fit with a huge literature of evidence. It doesn't sound plausible to me.” A few clicks later, Krosnick had pulled up the paper on his computer. “Ah,” he told the producer, “I see Don Green is an author. I trust him completely, so I'm no longer doubtful.” (Some people I spoke to about this case argued that Green, whose name is, after all, on the paper, had failed in his supervisory role. I emailed him to ask whether he thought this was a fair assessment. “Entirely fair,” he responded. “I am deeply embarrassed that I did not suspect and discover the fabrication of the survey data and grateful to the team of researchers who brought it to my attention.” He declined to comment further for this
Eventually, Broockman’s answer to that question would take LaCour down. ***
Last week, Broockman, along with his friend and fellow UC Berkeley graduate student Josh Kalla and Yale University political scientist Peter Aronow, released an explosive 27-page report (http://stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf) recounting many “irregularities” in LaCour and Green’s paper. “Irregularities” is diplomatic phrasing; what the trio found was that there’s no evidence LaCour ever actually collaborated with uSamp, the survey firm he claimed to have worked with to produce his data, and that he most likely didn’t commission any surveys whatsoever. Instead, he took a preexisting dataset, pawned it off as his own, and faked the persuasion “effects” of the canvassing. It’s the sort of brazen data fraud you just don’t see that often, especially in a journal like Science. Green quickly fired off an email to the journal asking for a retraction; Science granted that wish yesterday, albeit without LaCour’s consent (http://news.sciencemag.org/policy/2015/05/science-retracts-gay-marriage-paper-without-lead- author-s-consent) . And while there’s no word out of central New Jersey just yet, there’s a good chance, once the legal dust settles, that Princeton University will figure out a way to rescind the job offer it extended to LaCour, who was supposed to start in July. (Princeton offered no comment other than an emailed statement: “We will review all available information and determine the next steps.”) LaCour, for his part, has lawyered up and isn’t talking to the media, although he was caught attempting to cover up faked elements of his curriculum vitae (http://nymag.com/scienceofus/2015/05/michael-lacour-made-up-a-teaching-award-too.html) earlier this week. His website claims (http://www.google.com/url? q=http%3A%2F%2Fwww.mikelacour.com%2F&sa=D&sntz=1&usg=AFQjCNE_D- LD79PJ6BWSHaMtWuRMmDGugA) that he will “supply a definitive response” by the end of the day today.
But even before Broockman, Kalla, and Aronow published their report, LaCour’s results were so impressive that,
on their face, they didn’t make sense. Jon Krosnick, a Stanford social psychologist who focuses on attitude
change and also works on issues of scientific transparency (http://www.google.com/url? q=http%3A%2F%2Fbps.stanford.edu%2F%3Fpage_id%3D987&sa=D&sntz=1&usg=AFQjCNGaX1UKo6ek6Ro7kCgE80qoxVro8A) , says that he hadn’t heard about the study until he was contacted by a "This American Life" producer who
described the results to him over the phone. “Gee,” he replied, “that's very surprising and doesn't fit with a huge literature of evidence. It doesn't sound plausible to me.” A few clicks later, Krosnick had pulled up the paper on his computer. “Ah,” he told the producer, “I see Don Green is an author. I trust him completely, so I'm no longer doubtful.” (Some people I spoke to about this case argued that Green, whose name is, after all, on the paper, had failed in his supervisory role. I emailed him to ask whether he thought this was a fair assessment. “Entirely fair,” he responded. “I am deeply embarrassed that I did not suspect and discover the fabrication of the survey data and grateful to the team of researchers who brought it to my attention.” He declined to comment further for this
story.)
Krosnick is no outlier. Over and over again, throughout the scientific community and the media, LaCour’s
impossible-seeming results were treated as truth, in part because of the weight Green’s name carried, and in
part, frankly, because people — researchers, journalists, activists — wanted to believe them. There was a
snowball effect here: The more the study’s impact and influence grew, the greater the incentive to buy into the
excitement.
LaCour’s persuasive personality also seems to have played a role. Those who knew the short, bearded, impeccably groomed graduate student, a 2009 graduate of UT Austin, describe him as very well put-together and adept at presenting summaries of his research — he was the guy who really could win you over with a scatterplot on an iPad (even today, his homepage (http://www.google.com/url? q=http%3A%2F%2Fwww.mikelacour.com%2F&sa=D&sntz=1&usg=AFQjCNE_D- LD79PJ6BWSHaMtWuRMmDGugA) continues to feature a cool visualization of his gay-marriage-persuasion “data”). And if his responses sometimes seemed to lack depth when he was pressed for details, his impressive connections often allayed concerns. “He was both effective at [first impressions] and also flailing a bit,” says Dave Fleischer (http://www.leadership-lab.org/team/) , head of the Leadership LAB of the Los Angeles LGBT Center, an organization that worked with LaCour and Green on their canvassing efforts (Fleischer says he worked with LaCour for about two years in total). “So when he and I really had a disagreement, he would often rely on the kind of arguments where he’d basically invoke authority, right? He’s the one with advanced training, and his adviser is this very high-powered, very experienced person [Lynn Vavreck, his Ph.D. adviser at UCLA], and they know a lot more than we do.” (Vavreck said she couldn’t comment because of UCLA’s ongoing investigation of LaCour’s conduct.)
So LaCour was always able to dissuade people from looking too closely, from asking follow-up questions. This was a graduate student who successfully ran the gauntlet of the Princeton interview process with a publicly posted CV that contained wild falsehoods (http://nymag.com/scienceofus/2015/05/lacour-made-up-his- biggest-funding-source.html) about his grant receipts — he listed $793,000 worth of them, which is an all but impossible amount for a political-science graduate student — and included a teaching award that doesn’t exist (http://nymag.com/scienceofus/2015/05/michael-lacour-made-up-a-teaching-award-too.html) . Many of his fabrications, including the data for the Science study, lay in plain sight for years, and yet no one picked up on them until last week. What’s most odd is the way some of the falsehoods sat buried in piles of truths. LaCour’s made-up “Emerging Instructor Award,” for example, was listed alongside a number of other awards he really did win. LaCour seems to have a tendency toward dishonesty even in situations where there is no rational reason for it. “My puzzlement now is, if he fabricated the data, surely he must have known that when people tried to replicate his study, they would fail to do so and the truth would come out,” Green told me (http://nymag.com/scienceofus/2015/05/co-author-of-the-faked-study-speaks-out.html) shortly after the scandal broke. “And so why not reason backward and say, Let’s do the study properly?”
Why did it take so long for someone to discover the fraud? It’s a question many have asked in the last week and a half as observers both inside (https://ricochet.com/scandal-political-science/) and outside (http://fivethirtyeight.com/datalab/as-a-major-retraction-shows-were-all-vulnerable-to-faked-data/) of
LaCour’s persuasive personality also seems to have played a role. Those who knew the short, bearded, impeccably groomed graduate student, a 2009 graduate of UT Austin, describe him as very well put-together and adept at presenting summaries of his research — he was the guy who really could win you over with a scatterplot on an iPad (even today, his homepage (http://www.google.com/url? q=http%3A%2F%2Fwww.mikelacour.com%2F&sa=D&sntz=1&usg=AFQjCNE_D- LD79PJ6BWSHaMtWuRMmDGugA) continues to feature a cool visualization of his gay-marriage-persuasion “data”). And if his responses sometimes seemed to lack depth when he was pressed for details, his impressive connections often allayed concerns. “He was both effective at [first impressions] and also flailing a bit,” says Dave Fleischer (http://www.leadership-lab.org/team/) , head of the Leadership LAB of the Los Angeles LGBT Center, an organization that worked with LaCour and Green on their canvassing efforts (Fleischer says he worked with LaCour for about two years in total). “So when he and I really had a disagreement, he would often rely on the kind of arguments where he’d basically invoke authority, right? He’s the one with advanced training, and his adviser is this very high-powered, very experienced person [Lynn Vavreck, his Ph.D. adviser at UCLA], and they know a lot more than we do.” (Vavreck said she couldn’t comment because of UCLA’s ongoing investigation of LaCour’s conduct.)
So LaCour was always able to dissuade people from looking too closely, from asking follow-up questions. This was a graduate student who successfully ran the gauntlet of the Princeton interview process with a publicly posted CV that contained wild falsehoods (http://nymag.com/scienceofus/2015/05/lacour-made-up-his- biggest-funding-source.html) about his grant receipts — he listed $793,000 worth of them, which is an all but impossible amount for a political-science graduate student — and included a teaching award that doesn’t exist (http://nymag.com/scienceofus/2015/05/michael-lacour-made-up-a-teaching-award-too.html) . Many of his fabrications, including the data for the Science study, lay in plain sight for years, and yet no one picked up on them until last week. What’s most odd is the way some of the falsehoods sat buried in piles of truths. LaCour’s made-up “Emerging Instructor Award,” for example, was listed alongside a number of other awards he really did win. LaCour seems to have a tendency toward dishonesty even in situations where there is no rational reason for it. “My puzzlement now is, if he fabricated the data, surely he must have known that when people tried to replicate his study, they would fail to do so and the truth would come out,” Green told me (http://nymag.com/scienceofus/2015/05/co-author-of-the-faked-study-speaks-out.html) shortly after the scandal broke. “And so why not reason backward and say, Let’s do the study properly?”
Why did it take so long for someone to discover the fraud? It’s a question many have asked in the last week and a half as observers both inside (https://ricochet.com/scandal-political-science/) and outside (http://fivethirtyeight.com/datalab/as-a-major-retraction-shows-were-all-vulnerable-to-faked-data/) of
academia ponder what this scandal means for science. A big part of the answer, it turns out, lies in David
Broockman’s story.
In more than three hours of interviews with Science of Us, Broockman laid out, for the first time, the complete timeline of how he exposed LaCour. His story details his concerns with LaCour’s research dating all the way back to late 2013 — far earlier than previously reported — and includes several moments in which, were it not for an unlucky break here or there, the fraud could have been uncovered sooner, potentially forestalling a great deal of the disruption it inflicted on various careers and on social science as a whole.
Most important, in Broockman’s opinion, his experience highlights a failure on the part of political science to nurture and assist young researchers who have suspicions about other scientists’ data, but who can’t, or can’t yet, prove any sort of malfeasance. In fact, throughout the entire process, until the very last moment when multiple “smoking guns” finally appeared, Broockman was consistently told by friends and advisers to keep quiet about his concerns lest he earn a reputation as a troublemaker, or — perhaps worse — someone who merely replicates and investigates others’ research rather than plant a flag of his own.
“Academics are people who do everything by the book,” Broockman tells Science of Us. “And especially as graduate students and untenured people, the No. 1 thing you want to do is show that you know the book and you’re doing everything by it.” But “this was something where we’re like, ‘What’s the book? There’s no book for this. What do we do?’”
***
The first thing Broockman did, back in December of 2013, was get frustrated at his inability to run a survey like LaCour’s. On the 9th, Broockman decided to call uSamp (since renamed). This was when the first of those near-misses occurred — a year and a half later, a similar conversation would help bust the entire scandal wide open. But the first time he called uSamp, Broockman trod carefully, because he thought LaCour was still working with the company. “As far as I’m concerned, he still has an ongoing relationship with this company, and is still gathering data with them,” he explains, “and I didn’t want to upset the apple cart of whatever, and so I don’t recall in any way mentioning him. I just said, ‘I’ve heard you can do this kind of work. Can you do this kind of work for me?’”
The salesperson he spoke with, Broockman explains, said that they weren’t sure the firm could complete this kind of survey, but seemed under-informed and slightly incompetent. “And so I just kind of gave up, because I wasn’t on a witch hunt,” says Broockman. “I was just trying to get a study done.” Had Broockman mentioned LaCour by name or pressed for more details, he would realize later, LaCour’s lack of any real connection to the company might have revealed itself right away.
Things didn’t get any easier when Broockman sent his RFP out to dozens of other survey companies the next day. “We are seeking quotes for a large study we are planning that will necessitate enrolling approximately 10,000 new individuals in a custom internet or phone panel,” it started. The responses indicated that these companies had very little ability to pull off a study on the scale of LaCour’s. Broockman says he “found that pretty weird,
In more than three hours of interviews with Science of Us, Broockman laid out, for the first time, the complete timeline of how he exposed LaCour. His story details his concerns with LaCour’s research dating all the way back to late 2013 — far earlier than previously reported — and includes several moments in which, were it not for an unlucky break here or there, the fraud could have been uncovered sooner, potentially forestalling a great deal of the disruption it inflicted on various careers and on social science as a whole.
Most important, in Broockman’s opinion, his experience highlights a failure on the part of political science to nurture and assist young researchers who have suspicions about other scientists’ data, but who can’t, or can’t yet, prove any sort of malfeasance. In fact, throughout the entire process, until the very last moment when multiple “smoking guns” finally appeared, Broockman was consistently told by friends and advisers to keep quiet about his concerns lest he earn a reputation as a troublemaker, or — perhaps worse — someone who merely replicates and investigates others’ research rather than plant a flag of his own.
“Academics are people who do everything by the book,” Broockman tells Science of Us. “And especially as graduate students and untenured people, the No. 1 thing you want to do is show that you know the book and you’re doing everything by it.” But “this was something where we’re like, ‘What’s the book? There’s no book for this. What do we do?’”
***
The first thing Broockman did, back in December of 2013, was get frustrated at his inability to run a survey like LaCour’s. On the 9th, Broockman decided to call uSamp (since renamed). This was when the first of those near-misses occurred — a year and a half later, a similar conversation would help bust the entire scandal wide open. But the first time he called uSamp, Broockman trod carefully, because he thought LaCour was still working with the company. “As far as I’m concerned, he still has an ongoing relationship with this company, and is still gathering data with them,” he explains, “and I didn’t want to upset the apple cart of whatever, and so I don’t recall in any way mentioning him. I just said, ‘I’ve heard you can do this kind of work. Can you do this kind of work for me?’”
The salesperson he spoke with, Broockman explains, said that they weren’t sure the firm could complete this kind of survey, but seemed under-informed and slightly incompetent. “And so I just kind of gave up, because I wasn’t on a witch hunt,” says Broockman. “I was just trying to get a study done.” Had Broockman mentioned LaCour by name or pressed for more details, he would realize later, LaCour’s lack of any real connection to the company might have revealed itself right away.
Things didn’t get any easier when Broockman sent his RFP out to dozens of other survey companies the next day. “We are seeking quotes for a large study we are planning that will necessitate enrolling approximately 10,000 new individuals in a custom internet or phone panel,” it started. The responses indicated that these companies had very little ability to pull off a study on the scale of LaCour’s. Broockman says he “found that pretty weird,
because apparently uSamp had managed this in like a day.” “Some small part of my head thought, ‘I wonder if it
was fake,’” says Broockman. “But most of me thought, ‘I guess I'll have to wait until Mike is willing to reveal what
the magic was.’”
Broockman would also have to wait since, like most academics, he was constantly juggling a thousand different projects. During most of 2014, he was working on the question of how constituents react to communication from their lawmakers (http://stanford.edu/~dbroock/papers/broockman_butler_legislator_experiments_2014.pdf) , a critique of a prominent statistical method (http://stanford.edu/~dbroock/papers/broockman_approaches_to_studying_representation.pdf) , and research into how polarization affects political representation (http://stanford.edu/~dbroock/papers/ahler_broockman_ideological_innocence.pdf) . As an undergrad, Broockman had done some work with Joshua Kalla (http://polisci.berkeley.edu/people/person/joshua-kalla) , a Pittsburgh native who was a couple years below him — Kalla had been a research assistant on a study about housing discrimination Broockman worked on with Green. As Kalla started looking at grad schools, Broockman aggressively lobbied him to come out West.
These efforts were successful, and once Kalla arrived on campus in the fall of 2014, Broockman’s approach to the LaCour research changed: Now, he thought, he had the teammate he needed to finally build on LaCour’s promising canvassing work. Broockman and Kalla have a strong natural chemistry as research partners. In one class at Berkeley, Kalla, who is straight, highlighted the many similarities between himself and Broockman, who is gay, with a nerdy statistics joke about “matching” — the idea of finding two very similar people in a data set to test what effects emerge when you apply a treatment to one but not the other. “If you exact-match,” Kalla said, “you could use me and David to figure out the causal effects of being gay.”
Canvassing was a natural subject for two young researchers interested in the dynamics of persuasion. “It turns out that even if you’re not interested in canvassing per se, canvassing is a great medium through which to test other theories of how to persuade people,” Broockman says. Whereas traditional experiments involving opinion change tend to entail certain methodological difficulties — are the anonymous survey-takers really paying attention to the questions? Is the sleepy undergrad actually listening to the prompt you’re reading them? — with canvassing, you can say with relative confidence that the subject of the experiment is actively engaging with whatever argument you’re testing.
The 2014 election also helped focus Broockman and Kalla’s research agenda. They were convinced canvassing worked — at least to a point — and that politicians weren’t capitalizing on this fact nearly enough. After the election, they co-authored a November 2014 piece in Vox (http://www.vox.com/2014/11/13/7214339/campaign-ground-game) arguing as much. The pair wrote that “research has consistently found that authentic interpersonal exchanges usually have sizable impacts (http://www.bloomberg.com/politics/features/2014-10-06/how-do-you-change-someones-mind-about- abortion-tell-them-you-had-one) ,” linking to a positive preelection Bloomberg Politics cover story about LaCour and Green’s research.
Broockman would also have to wait since, like most academics, he was constantly juggling a thousand different projects. During most of 2014, he was working on the question of how constituents react to communication from their lawmakers (http://stanford.edu/~dbroock/papers/broockman_butler_legislator_experiments_2014.pdf) , a critique of a prominent statistical method (http://stanford.edu/~dbroock/papers/broockman_approaches_to_studying_representation.pdf) , and research into how polarization affects political representation (http://stanford.edu/~dbroock/papers/ahler_broockman_ideological_innocence.pdf) . As an undergrad, Broockman had done some work with Joshua Kalla (http://polisci.berkeley.edu/people/person/joshua-kalla) , a Pittsburgh native who was a couple years below him — Kalla had been a research assistant on a study about housing discrimination Broockman worked on with Green. As Kalla started looking at grad schools, Broockman aggressively lobbied him to come out West.
These efforts were successful, and once Kalla arrived on campus in the fall of 2014, Broockman’s approach to the LaCour research changed: Now, he thought, he had the teammate he needed to finally build on LaCour’s promising canvassing work. Broockman and Kalla have a strong natural chemistry as research partners. In one class at Berkeley, Kalla, who is straight, highlighted the many similarities between himself and Broockman, who is gay, with a nerdy statistics joke about “matching” — the idea of finding two very similar people in a data set to test what effects emerge when you apply a treatment to one but not the other. “If you exact-match,” Kalla said, “you could use me and David to figure out the causal effects of being gay.”
Canvassing was a natural subject for two young researchers interested in the dynamics of persuasion. “It turns out that even if you’re not interested in canvassing per se, canvassing is a great medium through which to test other theories of how to persuade people,” Broockman says. Whereas traditional experiments involving opinion change tend to entail certain methodological difficulties — are the anonymous survey-takers really paying attention to the questions? Is the sleepy undergrad actually listening to the prompt you’re reading them? — with canvassing, you can say with relative confidence that the subject of the experiment is actively engaging with whatever argument you’re testing.
The 2014 election also helped focus Broockman and Kalla’s research agenda. They were convinced canvassing worked — at least to a point — and that politicians weren’t capitalizing on this fact nearly enough. After the election, they co-authored a November 2014 piece in Vox (http://www.vox.com/2014/11/13/7214339/campaign-ground-game) arguing as much. The pair wrote that “research has consistently found that authentic interpersonal exchanges usually have sizable impacts (http://www.bloomberg.com/politics/features/2014-10-06/how-do-you-change-someones-mind-about- abortion-tell-them-you-had-one) ,” linking to a positive preelection Bloomberg Politics cover story about LaCour and Green’s research.
“When we wrote that piece, all of a sudden we received a ton of inbound interest in doing more studies, both
because people were persuaded by our point and because we kind of planted a flag in this,” says Broockman.
“And so practitioners who were interested in this decided to come talk to us about it.” It was clearly a good time
to hone in on canvassing. “That was the perfect storm for Now the time for this idea has come.”
This was also around the time Broockman first got hold of LaCour’s raw data (he’d read the Science paper when it was under review in late 2014). Certain irregularities quickly jumped out at him: The data was, in short, a bit too orderly given that it came from a big survey sample. In itself this didn’t constitute definitive proof that anything was amiss, but it definitely warranted further investigation. Whatever the excitement-suspicion ratio regarding LaCour’s findings had been in Broockman’s mind previously — maybe 90/10 when he first heard about the experiment — it was now closer to 50/50.
Broockman wasn’t sure what to do. He started to bring up his concerns with other friends and advisers (about a dozen of them), and they mostly told him one of two things: Either there was a reasonable explanation for the anomalies, in which case bringing attention to them would risk harming Green and especially the less established LaCour unnecessarily; or something really was fishy, in which case it still wouldn’t be in Broockman’s interest to risk being seen as challenging LaCour’s work. There was almost no encouragement for him to probe the hints of weirdness he’d uncovered. In fact, he quickly found himself nervous about openly discussing his reservations at all. “How much I said depended on how much I trust the person I was talking to and how inebriated I was at the time I had the conversation,” he explains.
On December 17, 2014, Broockman found himself a bit tipsy with someone he trusted: Neil Malhotra, a professor at Stanford’s business school. Broockman had just been offered a job there, and the two were dining at Oak and Rye, a pizza place in Los Gatos, partly so that Broockman could ask Malhotra for advice about the transition from grad school to the professional academic world. A few drinks in, Broockman shared his concerns about LaCour’s data. Malhotra recalled his response: “As someone in your early career stage, you don’t want to do this,” he told Broockman. “You don’t want to go public with this. Even if you have uncontroversial proof, you still shouldn’t do it. Because there’s just not much reward to someone doing this.” If Broockman thought there was wrongdoing behind the irregularities he’d discovered, Malhotra said, it would be a better bet for him to pass his concerns on to someone like Uri Simonsohn (http://opim.wharton.upenn.edu/~uws/) , a University of Pennsylvania researcher who already had established an identity as a debunker (eventually, Simonsohn gave Broockman some feedback on the data, but the exchange didn’t lead to any definitive findings).
This might seem like a strange, mafia-ish argument to a non-academic, but within the small world of political science — particularly within the even smaller world of younger, less job-secure political scientists — it makes sense for at least two reasons. The first is that the moment your name is associated with the questioning of someone else’s work, you could be in trouble. If the target is someone above you, like Green, you’re seen as envious, as shamelessly trying to take down a big name. If the target is someone at your level, you’re throwing elbows in an unseemly manner. In either case, you may end up having one of your papers reviewed by the target of your inquiries (or one of their friends) at some point — in theory, peer reviewers are “blinded” to the identity of the author or authors of a paper they’re reviewing, but between earlier versions of papers floating around the internet and the fact that everyone knows what everyone else is working on, the reality is quite different.
This was also around the time Broockman first got hold of LaCour’s raw data (he’d read the Science paper when it was under review in late 2014). Certain irregularities quickly jumped out at him: The data was, in short, a bit too orderly given that it came from a big survey sample. In itself this didn’t constitute definitive proof that anything was amiss, but it definitely warranted further investigation. Whatever the excitement-suspicion ratio regarding LaCour’s findings had been in Broockman’s mind previously — maybe 90/10 when he first heard about the experiment — it was now closer to 50/50.
Broockman wasn’t sure what to do. He started to bring up his concerns with other friends and advisers (about a dozen of them), and they mostly told him one of two things: Either there was a reasonable explanation for the anomalies, in which case bringing attention to them would risk harming Green and especially the less established LaCour unnecessarily; or something really was fishy, in which case it still wouldn’t be in Broockman’s interest to risk being seen as challenging LaCour’s work. There was almost no encouragement for him to probe the hints of weirdness he’d uncovered. In fact, he quickly found himself nervous about openly discussing his reservations at all. “How much I said depended on how much I trust the person I was talking to and how inebriated I was at the time I had the conversation,” he explains.
On December 17, 2014, Broockman found himself a bit tipsy with someone he trusted: Neil Malhotra, a professor at Stanford’s business school. Broockman had just been offered a job there, and the two were dining at Oak and Rye, a pizza place in Los Gatos, partly so that Broockman could ask Malhotra for advice about the transition from grad school to the professional academic world. A few drinks in, Broockman shared his concerns about LaCour’s data. Malhotra recalled his response: “As someone in your early career stage, you don’t want to do this,” he told Broockman. “You don’t want to go public with this. Even if you have uncontroversial proof, you still shouldn’t do it. Because there’s just not much reward to someone doing this.” If Broockman thought there was wrongdoing behind the irregularities he’d discovered, Malhotra said, it would be a better bet for him to pass his concerns on to someone like Uri Simonsohn (http://opim.wharton.upenn.edu/~uws/) , a University of Pennsylvania researcher who already had established an identity as a debunker (eventually, Simonsohn gave Broockman some feedback on the data, but the exchange didn’t lead to any definitive findings).
This might seem like a strange, mafia-ish argument to a non-academic, but within the small world of political science — particularly within the even smaller world of younger, less job-secure political scientists — it makes sense for at least two reasons. The first is that the moment your name is associated with the questioning of someone else’s work, you could be in trouble. If the target is someone above you, like Green, you’re seen as envious, as shamelessly trying to take down a big name. If the target is someone at your level, you’re throwing elbows in an unseemly manner. In either case, you may end up having one of your papers reviewed by the target of your inquiries (or one of their friends) at some point — in theory, peer reviewers are “blinded” to the identity of the author or authors of a paper they’re reviewing, but between earlier versions of papers floating around the internet and the fact that everyone knows what everyone else is working on, the reality is quite different.
Moreover, the very few plum jobs and big grants don’t go to people who investigate other researchers’ work —
they go to those who stake out their own research areas.
So Broockman decided he needed a way to get feedback on his suspicions without leaving a trace. He’d recently learned about a strange anonymous message board called poliscirumors.com, or PSR. “I believe I first learned about the board when I received a Google Alert with a page that had my last name on it, which proposed marriage to me,” he says. “So naturally that was a link that I clicked.”
Three different people independently described PSR to me as a “cesspool.” No one knows exactly who the site’s primary denizens are, because hardly anyone will admit to perusing it, but it seems to skew young — mostly political-science grad students and untenured professors. While the ostensible purpose of PSR is to provide information about job openings, posts on it have a tendency to devolve into attacks, rumor-mongering, and bitterness fueled by an apocalyptic academic job market. “It is essentially the 4chan of political science,” a political-science researcher told me via email.
It’s not, in short, necessarily the place where one goes for levelheaded debate about the results of statistical analysis. But Broockman didn’t know where else to turn, so on December 20, he posted about 25 lines of output from Stata, a statistical modeling program, along with some commentary, to a preexisting thread about the LaCour-Green study.
In the post, the author of which was dubbed “Reannon” by PSR’s random-name generator, Broockman highlighted two of the oddities in the data. One was that individual respondents’ attitudes toward gay marriage, measured on a five-point scale, and their attitudes toward gay people in general, measured on a 101-point “feelings thermometer,” didn’t vary that much from survey to survey (remember that respondents’ in LaCour and Green’s study were surveyed — or were supposed to have been surveyed — multiple times). One would expect their answers to jump around randomly a bit — that’s just the nature of attempting to translate a messy human opinion into a single number (feelings thermometers, in general, are known to have particularly low so-called “test-retest reliability”). Plus, when you have a big enough sample, which LaCour did, a few people are inevitably going to give zany results for some reason or another — sometimes, for example, they’ll appear to go from being Harvey Milk in the first wave of data collection to Jerry Falwell in the second. And yet the answers these participants gave were anchored in place over time. “That amount of intertemporal stability is not believable to me,” wrote Reannon.
The other concern was with the shape of the feelings-thermometer data over time. “The noise is weirdly normally distributed,” wrote Reannon, “with *NOT ONE* respondent out of 10,597 deviating by more than 25 in a negative direction or 38 in a positive direction.” In other words, the results, again, just didn’t include enough of the statistical randomness one would expect in a large sample of human opinion. They followed a pattern that looked artificial. “DEFINITELY not saying this nails anything,” concluded Broockman and/or Reannon, “just that digging deeper into this data probably should be done ...”
Broockman hoped that someone would pick up his observations and run with them. “I explicitly said, ‘Somebody should look into this,’” he says. “And I was hoping that somebody who wasn’t a vulnerable graduate student
So Broockman decided he needed a way to get feedback on his suspicions without leaving a trace. He’d recently learned about a strange anonymous message board called poliscirumors.com, or PSR. “I believe I first learned about the board when I received a Google Alert with a page that had my last name on it, which proposed marriage to me,” he says. “So naturally that was a link that I clicked.”
Three different people independently described PSR to me as a “cesspool.” No one knows exactly who the site’s primary denizens are, because hardly anyone will admit to perusing it, but it seems to skew young — mostly political-science grad students and untenured professors. While the ostensible purpose of PSR is to provide information about job openings, posts on it have a tendency to devolve into attacks, rumor-mongering, and bitterness fueled by an apocalyptic academic job market. “It is essentially the 4chan of political science,” a political-science researcher told me via email.
It’s not, in short, necessarily the place where one goes for levelheaded debate about the results of statistical analysis. But Broockman didn’t know where else to turn, so on December 20, he posted about 25 lines of output from Stata, a statistical modeling program, along with some commentary, to a preexisting thread about the LaCour-Green study.
In the post, the author of which was dubbed “Reannon” by PSR’s random-name generator, Broockman highlighted two of the oddities in the data. One was that individual respondents’ attitudes toward gay marriage, measured on a five-point scale, and their attitudes toward gay people in general, measured on a 101-point “feelings thermometer,” didn’t vary that much from survey to survey (remember that respondents’ in LaCour and Green’s study were surveyed — or were supposed to have been surveyed — multiple times). One would expect their answers to jump around randomly a bit — that’s just the nature of attempting to translate a messy human opinion into a single number (feelings thermometers, in general, are known to have particularly low so-called “test-retest reliability”). Plus, when you have a big enough sample, which LaCour did, a few people are inevitably going to give zany results for some reason or another — sometimes, for example, they’ll appear to go from being Harvey Milk in the first wave of data collection to Jerry Falwell in the second. And yet the answers these participants gave were anchored in place over time. “That amount of intertemporal stability is not believable to me,” wrote Reannon.
The other concern was with the shape of the feelings-thermometer data over time. “The noise is weirdly normally distributed,” wrote Reannon, “with *NOT ONE* respondent out of 10,597 deviating by more than 25 in a negative direction or 38 in a positive direction.” In other words, the results, again, just didn’t include enough of the statistical randomness one would expect in a large sample of human opinion. They followed a pattern that looked artificial. “DEFINITELY not saying this nails anything,” concluded Broockman and/or Reannon, “just that digging deeper into this data probably should be done ...”
Broockman hoped that someone would pick up his observations and run with them. “I explicitly said, ‘Somebody should look into this,’” he says. “And I was hoping that somebody who wasn’t a vulnerable graduate student
about to take their dream job would do it. For Christ's sake, I even gave them the code to do it! And nobody did.”
Worse yet, shortly after he posted the Stata code, it was deleted by Kirk, the site’s somewhat mysterious
moderator (Kirk confirmed this later, in an email to me and a PSR post
(http://www.poliscirumors.com/topic/kirk-i-deleted-the-gelmans-monkey-cage-post) ). “That was one more
signal to me that the concerns I have were considered totally out of bounds to have,” says Broockman. “So that
kind of shut me up for a while. So my view was, ‘Wow, if this thought is verboten even in this cesspool, I certainly
shouldn’t have it.’” (As a result of some combination of emails I sent to Kirk and pressure from PSR users, who
began talking about the deleted post after news of the scandal broke, he restored
(http://www.poliscirumors.com/topic/gelmans-monkey-cage-post/page/4#post-316530) the post
(http://www.poliscirumors.com/topic/gelmans-monkey-cage-post/page/2#post-240222) last Saturday.)
There was another near-miss around that time. Broockman ran a statistical test to see whether the changes in the respondents’ attitudes really did follow a perfectly normal distribution — which, again, would suggest the data had been cooked. The test replied that no, that wasn’t what was going on.
The problem was that Broockman failed to account for the fact that the feelings-thermometer data can’t go above 100 or below 0. So if a homophobe’s views on gay people were at a 3 the first time they were surveyed, and the pattern programmed in by a would-be data chef should have pushed this number to negative-5 in the second wave, in reality their view would only go down to 0, the lowest possible number. If he’d been comfortable enough to seek out collaborators more assertively, Broockman thinks he — or they — might have caught this error. “If I had been able to go to people and not say in hushed tones, I have concerns, but rather if I could have been frank about it and said, Here’s what I found and here’s one hypothesis that I’m concerned about, I think much more quality work would have been done earlier,” he says.
Instead, chastened by his deleted attempt to post his concerns anonymously and the lack of clear findings from his data analysis, Broockman pressed on with his academic work. Early in 2015, he and Kalla were recruited by Dave Fleischer (http://www.leadership-lab.org/team/) , the head of the Leadership LAB, for a project based in Miami (Broockman and Kalla worked on it from California) that dealt with transgender equality. That and finishing his dissertation kept Broockman busy. During that time, Broockman continued to speak with LaCour now and again — he says the two have averaged about one Google Hangout per month for the last two years — and when he did, he would often ask him questions about his survey methodology. LaCour’s answers, Broockman says, tended to be vague and unhelpful.
It was the Miami project that finally nudged Broockman and Kalla toward their first hard evidence that LaCour had done something wrong. A few months in, during a pilot study they were conducting, the duo were struggling to figure out how LaCour had been able to get such a high response rate — 12 percent in the first wave, which is a very solid number. Broockman and Kalla’s results, on the other hand, were frustratingly crappy. For a survey request they sent out via mail to 13,878 Miami-Dade County voters on May 8, only about 100 respondents had agreed to participate as of a week later — less than one percent. Broockman and Kalla were offering as much money, per survey, as LaCour had paid his respondents. Why were so many fewer people biting?
There was another near-miss around that time. Broockman ran a statistical test to see whether the changes in the respondents’ attitudes really did follow a perfectly normal distribution — which, again, would suggest the data had been cooked. The test replied that no, that wasn’t what was going on.
The problem was that Broockman failed to account for the fact that the feelings-thermometer data can’t go above 100 or below 0. So if a homophobe’s views on gay people were at a 3 the first time they were surveyed, and the pattern programmed in by a would-be data chef should have pushed this number to negative-5 in the second wave, in reality their view would only go down to 0, the lowest possible number. If he’d been comfortable enough to seek out collaborators more assertively, Broockman thinks he — or they — might have caught this error. “If I had been able to go to people and not say in hushed tones, I have concerns, but rather if I could have been frank about it and said, Here’s what I found and here’s one hypothesis that I’m concerned about, I think much more quality work would have been done earlier,” he says.
Instead, chastened by his deleted attempt to post his concerns anonymously and the lack of clear findings from his data analysis, Broockman pressed on with his academic work. Early in 2015, he and Kalla were recruited by Dave Fleischer (http://www.leadership-lab.org/team/) , the head of the Leadership LAB, for a project based in Miami (Broockman and Kalla worked on it from California) that dealt with transgender equality. That and finishing his dissertation kept Broockman busy. During that time, Broockman continued to speak with LaCour now and again — he says the two have averaged about one Google Hangout per month for the last two years — and when he did, he would often ask him questions about his survey methodology. LaCour’s answers, Broockman says, tended to be vague and unhelpful.
It was the Miami project that finally nudged Broockman and Kalla toward their first hard evidence that LaCour had done something wrong. A few months in, during a pilot study they were conducting, the duo were struggling to figure out how LaCour had been able to get such a high response rate — 12 percent in the first wave, which is a very solid number. Broockman and Kalla’s results, on the other hand, were frustratingly crappy. For a survey request they sent out via mail to 13,878 Miami-Dade County voters on May 8, only about 100 respondents had agreed to participate as of a week later — less than one percent. Broockman and Kalla were offering as much money, per survey, as LaCour had paid his respondents. Why were so many fewer people biting?
Broockman first emailed LaCour on the 13th about the issue; LaCour responded that he was headed to Princeton
and therefore couldn’t help until the following Monday. Two days later, Kalla came over to Broockman’s
apartment to try to figure out what was going on. Broockman, not wanting to wait for LaCour, got in touch with a
research partner of LaCour’s (who prefers to remain anonymous) and asked if he had any thoughts on how
LaCour had pulled off his impressive response rate. Eventually, this person sent Broockman an email from
LaCour that included a forwarded email from a uSamp employee named Jason Peterson.
Dave Flescher was one of the original recipients of this email, and he provided Science of Us with a copy (click here (http://img1.nymag.com/imgs/daily/science/2015/05/28/28-jason-peterson-email.w826.h1200.png) for a larger version):
Dave Flescher was one of the original recipients of this email, and he provided Science of Us with a copy (click here (http://img1.nymag.com/imgs/daily/science/2015/05/28/28-jason-peterson-email.w826.h1200.png) for a larger version):
Jason Peterson, then, was their man — he might be able to explain what was going on with their poor response
rates. Broockman asked Kalla to call him; Kalla went upstairs and did. He came back a little bit later. “Holy shit,”
he said. There was no employee named Jason Peterson at uSamp. An email to double check confirmed this: A
uSamp staffer responded later that day, saying, “There was never a Jason Peterson at uSamp at any time.” (Not
only had LaCour made up an employee, he’d promoted that made-up employee all the way up to vice-president
— and note the weird capitalization in LaCour’s name in the forwarded message, which may or may not mean
anything.) The firm also reiterated the message Broockman had received a year and a half prior, the one he
hadn’t realized the significance of at the time: The firm couldn’t even do the research LaCour had described. This
was now a far cry from vague concerns about artificial-looking data patterns and weirdly reliable thermometers.
Later that night, Broockman hosted a Hawaiian-themed graduation party at his apartment. About 70 people came — a mix of grad-school friends, family, and friends he had made in the Bay Area. All day, Broockman’s boyfriend had begged him to help get the place ready, but he was too fixated on the LaCour and Green paper. Yet even with the revelation about Jason Peterson, Broockman and Kalla still thought they lacked the smoking gun they needed. “We were in a state of panic, not sure if we should keep looking, what we should look for, and what to do with what we had found,” Broockman says. Right as the party started, Kalla tried to get Broockman to chill: “No more talking about this!” he told Broockman. “Your boyfriend made a nice party for you!”
But it was now almost impossible for Broockman to stop talking about the paper. He was surrounded by his academic advisers and imbibed more than a couple of sparkling rosés. He ended up talking to Malhotra some more. Yet again, Malhotra said: Be careful. Don’t expose yourself. But, in addition to the revelation about Peterson, Broockman was starting to feel moral concerns: “The Ireland referendum was coming up,” he says. “Big grants were about to be spent.” He hated the idea that people were being misled by “findings” that looked increasingly suspect.
Whatever internal conflict Broockman was grappling with, it didn’t matter: The next day, Saturday, May 16, Broockman and Kalla — and now Aronow, the Yale political scientist and survey-statistics specialist whom the pair brought onboard in the wake of the Peterson discovery — had their true smoking gun. “I found it!” Kalla exulted to Broockman on Gchat. "It," in this case, was the 2012 Cooperative Campaign Analysis Project, or CCAP, a big, well-known political-science data set. Because the first wave of LaCour’s data looked real enough — the weirdly uniform patterns only came in during subsequent waves — Broockman had wondered if maybe LaCour had taken real data from somewhere, manipulated it, and then presented the whole shebang as the “responses” to his “surveys.” CCAP was a likely candidate: It included the same gay-marriage and feeling-thermometer items LaCour had in his study. It wasn’t a publicly accessible dataset, but Kalla had figured out a way to download a copy. A couple simple statistical tests later, and there was no longer much doubt: This was the data LaCour was presenting as his own.
By the end of the next day, Kalla, Broockman, and Aronow had compiled their report and sent it to Green, and Green had quickly replied that unless LaCour could explain everything in it, he’d reach out to Science and request a retraction. (Broockman had decided the best plan was to take their concerns to Green instead of LaCour in order to reduce the chance that LaCour could scramble to contrive an explanation.)
After Green spoke to Vavreck, LaCour’s adviser, LaCour confessed to Green and Vavreck that he hadn’t conducted the surveys the way he had described them, though the precise nature of that conversation is
Later that night, Broockman hosted a Hawaiian-themed graduation party at his apartment. About 70 people came — a mix of grad-school friends, family, and friends he had made in the Bay Area. All day, Broockman’s boyfriend had begged him to help get the place ready, but he was too fixated on the LaCour and Green paper. Yet even with the revelation about Jason Peterson, Broockman and Kalla still thought they lacked the smoking gun they needed. “We were in a state of panic, not sure if we should keep looking, what we should look for, and what to do with what we had found,” Broockman says. Right as the party started, Kalla tried to get Broockman to chill: “No more talking about this!” he told Broockman. “Your boyfriend made a nice party for you!”
But it was now almost impossible for Broockman to stop talking about the paper. He was surrounded by his academic advisers and imbibed more than a couple of sparkling rosés. He ended up talking to Malhotra some more. Yet again, Malhotra said: Be careful. Don’t expose yourself. But, in addition to the revelation about Peterson, Broockman was starting to feel moral concerns: “The Ireland referendum was coming up,” he says. “Big grants were about to be spent.” He hated the idea that people were being misled by “findings” that looked increasingly suspect.
Whatever internal conflict Broockman was grappling with, it didn’t matter: The next day, Saturday, May 16, Broockman and Kalla — and now Aronow, the Yale political scientist and survey-statistics specialist whom the pair brought onboard in the wake of the Peterson discovery — had their true smoking gun. “I found it!” Kalla exulted to Broockman on Gchat. "It," in this case, was the 2012 Cooperative Campaign Analysis Project, or CCAP, a big, well-known political-science data set. Because the first wave of LaCour’s data looked real enough — the weirdly uniform patterns only came in during subsequent waves — Broockman had wondered if maybe LaCour had taken real data from somewhere, manipulated it, and then presented the whole shebang as the “responses” to his “surveys.” CCAP was a likely candidate: It included the same gay-marriage and feeling-thermometer items LaCour had in his study. It wasn’t a publicly accessible dataset, but Kalla had figured out a way to download a copy. A couple simple statistical tests later, and there was no longer much doubt: This was the data LaCour was presenting as his own.
By the end of the next day, Kalla, Broockman, and Aronow had compiled their report and sent it to Green, and Green had quickly replied that unless LaCour could explain everything in it, he’d reach out to Science and request a retraction. (Broockman had decided the best plan was to take their concerns to Green instead of LaCour in order to reduce the chance that LaCour could scramble to contrive an explanation.)
After Green spoke to Vavreck, LaCour’s adviser, LaCour confessed to Green and Vavreck that he hadn’t conducted the surveys the way he had described them, though the precise nature of that conversation is
unknown. Green posted his retraction request publicly on May 19, the same day Broockman, Kalla, and Aronow
posted their report. That was also the day Broockman graduated. “Between the morning brunch and
commencement, Josh and I kept leaving the ceremonies to work on the report,” Broockman wrote in an email.
His mom captured one such moment in a photograph.
Kalla and Broockman work on their final report during Broockman's graduation day Photo: Courtesy of David
Brookckman
On May 20, the well-read science blog Retraction Watch (http://retractionwatch.com/2015/05/20/author- retracts-study-of-changing-minds-on-same-sex-marriage-after-colleague-admits-data-were-faked/) broke the news of the scandal, which would eventually bring the site so much traffic that it crashed. That was it: The news was out, and David Broockman’s name was all over it.
***
Broockman is a thoughtful, fastidious guy — you don’t get a tenure-track Stanford professorship at 26 without being one. Before we spoke on the record for the first time, he sent me a seven-page document outlining all the thoughts that had rattled around in his head since the news broke. So it wasn’t surprising that he also prepared and emailed me a quote that, in his view, sums up the problems that plagued him during his sometimes-halting investigation of LaCour’s work. “I think my discipline needs to answer this question: How can concerns about dishonesty in published research be brought to light in a way that protects innocent researchers and the truth — especially when it’s less egregious?” he wrote. “I don’t think there’s an easy answer. But until we
On May 20, the well-read science blog Retraction Watch (http://retractionwatch.com/2015/05/20/author- retracts-study-of-changing-minds-on-same-sex-marriage-after-colleague-admits-data-were-faked/) broke the news of the scandal, which would eventually bring the site so much traffic that it crashed. That was it: The news was out, and David Broockman’s name was all over it.
***
Broockman is a thoughtful, fastidious guy — you don’t get a tenure-track Stanford professorship at 26 without being one. Before we spoke on the record for the first time, he sent me a seven-page document outlining all the thoughts that had rattled around in his head since the news broke. So it wasn’t surprising that he also prepared and emailed me a quote that, in his view, sums up the problems that plagued him during his sometimes-halting investigation of LaCour’s work. “I think my discipline needs to answer this question: How can concerns about dishonesty in published research be brought to light in a way that protects innocent researchers and the truth — especially when it’s less egregious?” he wrote. “I don’t think there’s an easy answer. But until we
have one, all of us who have had such concerns remain liars by omission.”
Broockman has ideas about how to reform things. He thinks so-called “post-publication peer review” — a system
which, to oversimplify a bit, makes it easier to evaluate the strength of previously published findings — has
promise. He’s also enamored, based in part on conversations he’s had with his software-engineer roommate,
with the concept of “responsible disclosure.”
“If you find out that there’s a security vulnerability in a website like Facebook, what you do is first you privately report that to Facebook, they fix it, and then after they fix it they retroactively open the conversation that you had,” he explains. After the fact, the bug-finder, in some cases, gets both recognition and a cash reward, but the key is that none of this is revealed publicly, in a way that would allow malicious hackers to take advantage of the vulnerability, until after the issue is resolved.
Broockman says he and his colleagues applied elements of this system to their approach to the LaCour case: They looked into LaCour’s data, but none of the details about what they found were posted publicly until it was clear that their suspicions were warranted and that some form of resolution was underway — that is, until Green had confronted LaCour, the latter had confessed to some degree of wrongdoing, and a UCLA investigation was imminent. But Broockman had to apply these principles in an ad-hoc way, and he says he hopes that “norms and institutions” — to political scientists, the two always go hand-in-hand — will emerge, which will make the process easier for people who find themselves in his position in the future.
Broockman repeatedly draws an important contrast: Before everything was public, part of him was terrified he’d be shunned if his suspicions were aired. But after his report went online, the reaction from his peers was “uniformly positive,” he says. “I literally couldn’t communicate for two days because my inbox was consistently overflowing with hundreds of people thanking me because the study had impacted their practical work or their academic work or career in some way,” he says.
“I think there’s an interesting metaphor between what I went through now and what I went through as a gay teenager,” Broockman says. “I felt trapped by this suspicion, it had weighed on my mind for a long time. You know, you try to act on it in small ways in hopes that it goes away, or you find confirmation that your suspicions are wrong. You’re worried about how people will react, so you proceed really cautiously. But finally, the truth is that when you come out about it, it’s really liberating.” He thinks part of the reason he was able to eventually debunk the study, in fact, was “because I’d gone through that same experience before in my life.”
“Part of the message that I wanted to send to potential disclosers of the future is that you have a duty to come out about this, you’ll be rewarded if you do so in a responsible way, and you don’t want to live with the regret that I do that I didn’t act on it [sooner],” he says. “This is sort of my It Gets Better project for people with suspicions about research.”
Copyright © 2015, New York Media LLC. All Rights Reserved. Vulture®, Grub Street® and The Cut® are registered trademarks of New York Media LLC.
“If you find out that there’s a security vulnerability in a website like Facebook, what you do is first you privately report that to Facebook, they fix it, and then after they fix it they retroactively open the conversation that you had,” he explains. After the fact, the bug-finder, in some cases, gets both recognition and a cash reward, but the key is that none of this is revealed publicly, in a way that would allow malicious hackers to take advantage of the vulnerability, until after the issue is resolved.
Broockman says he and his colleagues applied elements of this system to their approach to the LaCour case: They looked into LaCour’s data, but none of the details about what they found were posted publicly until it was clear that their suspicions were warranted and that some form of resolution was underway — that is, until Green had confronted LaCour, the latter had confessed to some degree of wrongdoing, and a UCLA investigation was imminent. But Broockman had to apply these principles in an ad-hoc way, and he says he hopes that “norms and institutions” — to political scientists, the two always go hand-in-hand — will emerge, which will make the process easier for people who find themselves in his position in the future.
Broockman repeatedly draws an important contrast: Before everything was public, part of him was terrified he’d be shunned if his suspicions were aired. But after his report went online, the reaction from his peers was “uniformly positive,” he says. “I literally couldn’t communicate for two days because my inbox was consistently overflowing with hundreds of people thanking me because the study had impacted their practical work or their academic work or career in some way,” he says.
“I think there’s an interesting metaphor between what I went through now and what I went through as a gay teenager,” Broockman says. “I felt trapped by this suspicion, it had weighed on my mind for a long time. You know, you try to act on it in small ways in hopes that it goes away, or you find confirmation that your suspicions are wrong. You’re worried about how people will react, so you proceed really cautiously. But finally, the truth is that when you come out about it, it’s really liberating.” He thinks part of the reason he was able to eventually debunk the study, in fact, was “because I’d gone through that same experience before in my life.”
“Part of the message that I wanted to send to potential disclosers of the future is that you have a duty to come out about this, you’ll be rewarded if you do so in a responsible way, and you don’t want to live with the regret that I do that I didn’t act on it [sooner],” he says. “This is sort of my It Gets Better project for people with suspicions about research.”
Copyright © 2015, New York Media LLC. All Rights Reserved. Vulture®, Grub Street® and The Cut® are registered trademarks of New York Media LLC.
No comments:
Post a Comment