Nate Silver has been researching the 2016 presidential election, starting with a comparison with Brexit earlier in 2016:

The U.S. presidential election, as I’ve argued here, was something of a similar case. No, the polls didn’t show a toss-up, as they had in Brexit. But the reporting was much more certain of Clinton’s chances than it should have been based on the polls. Much of The New York Times’s coverage, for instance, implied that Clinton’s odds were close to 100 percent. In an article on Oct. 17 — more than three weeks before Election Day — they portrayed the race as being effectively over, the only question being whether Clinton should seek a landslide or instead assist down-ballot Democrats:

Hillary Clinton’s campaign is planning its most ambitious push yet into traditionally right-leaning states, a new offensive aimed at extending her growing advantage over Donald J. Trump while bolstering down-ballot candidates in what party leaders increasingly suggest could be a sweeping victory for Democrats at every level. […]

The maneuvering speaks to the unexpected tension facing Mrs. Clinton as she hurtles toward what aides increasingly believe will be a decisive victory — a pleasant problem, for certain, but one that has nonetheless scrambled the campaign’s strategy weeks before Election Day: Should Mrs. Clinton maximize her own margin, aiming to flip as many red states as possible to run up an electoral landslide, or prioritize the party’s congressional fortunes, redirecting funds and energy down the ballot?

This is not to say the election was a toss-up in mid-October, which was one of the high-water marks of the campaign for Clinton. But while a Trump win was unlikely, it should hardly have been unthinkable. As we were fond of pointing out at the time, Trump’s chances in mid-October were around 1-in-6 according to betting markets and FiveThirtyEight’s forecast, about the same as a chance of being shot while playing a “game” of Russian roulette. And yet the Times, famous for its “to be sure” equivocations,² wasn’t even contemplating the possibility of a Trump victory​. It expressed nearly as much confidence in Clinton two weeks later even after the polls tightened substantially after FBI director James B. Comey’s letter to Congress, at which point Trump’s odds jumped to about 1-in-3 in our forecast.”

It’s hard to reread this coverage without recalling Sean Trende’s essay on “unthinkability bias,” which he wrote in the wake of the Brexit vote. Just as was the case in the U.S. presidential election, voting on the referendum had split strongly along class, education and regional lines, with voters outside of London and without advanced degrees being much more likely to vote to leave the EU. The reporters covering the Brexit campaign, on the other hand, were disproportionately well-educated and principally based in London. They tended to read ambiguous signs — anything from polls to the musings of taxi drivers — as portending a Remain win, and many of them never really processed the idea that Britain could vote to leave the EU until it actually happened.

So did journalists in Washington and London make the apocryphal Pauline Kael mistake, refusing to believe that Trump or Brexit could win because nobody they knew was voting for them? That’s not quite what Trende was arguing. Instead, it’s that political experts aren’t a very diverse group and tend to place a lot of faith in the opinions of other experts and other members of the political establishment. Once a consensus view is established, it tends to reinforce itself until and unless there’s very compelling evidence for the contrary position. Social media, especially Twitter, can amplify the groupthink further. It can be an echo chamber.

I recently reread James Surowiecki’s book “The Wisdom of Crowds” which, despite its name, spends as much time contemplating the shortcomings of such wisdom as it does celebrating its successes. Surowiecki argues⁵ that crowds usually make good predictions when they satisfy these four conditions:

1. Diversity of opinion. “Each person should have private information, even if it’s just an eccentric interpretation of the known facts.”
2. Independence. “People’s opinions are not determined by the opinions of those around them.”
3. Decentralization. “People are able to specialize and draw on local knowledge.”
4. Aggregation. “Some mechanism exists for turning private judgments into a collective decision.”

Political journalism scores highly on the fourth condition, aggregation. While Surowiecki usually has something like a financial or betting market in mind when he refers to “aggregation,” the broader idea is that there’s some way for individuals to exchange their opinions instead of keeping them to themselves. And my gosh, do political journalists have a lot of ways to share their opinions with one another, whether through their columns, at major events such as the political conventions or, especially, through Twitter.

But those other three conditions? Political journalism fails miserably along those dimensions.

Diversity of opinion? For starters, American newsrooms are not very diverse along racial or gender lines, and it’s not clear the situation is improving much.⁶ And in a country where educational attainment is an increasingly important predictor of cultural and political behavior, some 92 percent of journalists have college degrees. A degree didn’t used to be a de facto prerequisite<a class — “It’s not clear how many newsrooms require a college degree for their reporting positions — we usually don’t do so at FiveThirtyEight and a quick perusal of other job listings suggest that many other newsrooms also do not. But there may nonetheless be a lot of self-selection in which candidates seek out these jobs and who succeeds in the recruiting and hiring process for a reporting job; just 70 percent of journalists had college degrees in 1982 and only 58 percent did in 1971.

The political diversity of journalists is not very strong, either. As of 2013, only 7 percent of them identified as Republicans (although only 28 percent called themselves Democrats with the majority saying they were independents). And although it’s not a perfect approximation — in most newsrooms, the people who issue endorsements are not the same as the ones who do reporting — there’s reason to think that the industry was particularly out of sync with Trump. Of the major newspapers that endorsed either Clinton or Trump, only 3 percent (2 of 59) endorsed Trump. By comparison, 46 percent of newspapers to endorse either Barack Obama or Mitt Romney endorsed Romney in 2012. Furthermore, as the media has become less representative of right-of-center views — and as conservatives have rebelled against the political establishment — there’s been an increasing and perhaps self-reinforcing cleavage between conservative news and opinion outlets such as Breitbart and the rest of the media.

Although it’s harder to measure, I’d also argue that there’s a lack of diversity when it comes to skill sets and methods of thinking in political journalism. Publications such as Buzzfeed or (the now defunct) Gawker.com get a lot of shade from traditional journalists when they do things that challenge conventional journalistic paradigms. But a lot of traditional journalistic practices are done by rote or out of habit, such as routinely granting anonymity to staffers to discuss campaign strategy even when there isn’t much journalistic merit in it. Meanwhile, speaking from personal experience, I’ve found the reception of “data journalists” by traditional journalists to be unfriendly, although there have been exceptions.

Independence? This is just as much of a problem. Crowds can be wise when people do a lot of thinking for themselves before coming together to exchange their views. But since at least the days of “The Boys on the Bus,” political journalism has suffered from a pack mentality. Events such as conventions and debates literally gather thousands of journalists together in the same room; attend one of these events, and you can almost smell the conventional wisdom being manufactured in real time. (Consider how a consensus formed that Romney won the first debate in 2012 when it had barely even started, for instance.) Social media — Twitter in particular — can amplify these information cascades, with a single tweet receiving hundreds of thousands of impressions and shaping the way entire issues are framed. As a result, it can be largely arbitrary which storylines gain traction and which ones don’t. What seems like a multiplicity of perspectives might just be one or two, duplicated many times over.

Decentralization? Surowiecki writes about the benefit of local knowledge, but the political news industry has become increasingly consolidated in Washington and New York as local newspapers have suffered from a decade-long contraction. That doesn’t necessarily mean local reporters in Wisconsin or Michigan or Ohio should have picked up Trumpian vibrations on the ground in contradiction to the polls. But as we’ve argued, national reporters often flew into these states with pre-baked narratives — for instance, that they were “decreasingly representative of contemporary America” — and fit the facts to suit them, neglecting their importance to the Electoral College. A more geographically decentralized reporting pool might have asked more questions about why Clinton wasn’t campaigning in Wisconsin, for instance, or why it wasn’t more of a problem for her that she was struggling in polls of traditional bellwethers such as Ohio and Iowa. If local newspapers had been healthier economically, they might also have commissioned more high-quality state polls; the lack of good polling was a problem in Michigan and Wisconsin especially.

There was once a notion that whatever challenges the internet created for journalism’s business model, it might at least lead readers to a more geographically and philosophically diverse array of perspectives. But it’s not clear that’s happening, either. Instead, based on data from the news aggregation site Memeorandum, the top news sources (such as the Times, The Washington Post and Politico) have earned progressively more influence over the past decade:

The share of total exposure for the top five news sources⁹ climbed from roughly 25 percent a decade ago to around 35 percent last year, and has spiked to above 40 percent so far in 2017. While not a perfect measure¹⁰, this is one sign the digital age hasn’t necessarily democratized the news media. Instead, the most notable difference in Memeorandum sources between 2007 and 2017 is the decline of independent blogs; many of the most popular ones from the late ’aughts either folded or (like FiveThirtyEight) were bought by larger news organizations. Thus, blogs and local newspapers — two of the better checks on Northeast Corridor conventional wisdom run amok — have both had less of a say in the conversation.

All things considered, then, the conditions of political journalism are poor for crowd wisdom and ripe for groupthink. So … what to do about it, then?

Initiatives to increase decentralization would help, although they won’t necessarily be easy. Increased subscription revenues at newspapers such as The New York Times and The Washington Post is an encouraging sign for journalism, but a revival of local and regional newspapers — or a more sustainable business model for independent blogs — would do more to reduce groupthink in the industry.

Likewise, improving diversity is liable to be a challenge, especially because the sort of diversity that Surowiecki is concerned with will require making improvements on multiple fronts (demographic diversity, political diversity, diversity of skill sets). Still, the research Surowiecki cites is emphatic that there are diminishing returns to having too many of the same types of people in small groups or organizations. Teams that consist entirely of high-IQ people may underperform groups that contain a mix of high-IQ and medium-IQ participants, for example, because the high-IQ people are likely to have redundant strengths and similar blind spots.

That leaves independence. In some ways the best hope for a short-term fix might come from an attitudinal adjustment: Journalists should recalibrate themselves to be more skeptical of the consensus of their peers. That’s because a position that seems to have deep backing from the evidence may really just be a reflection from the echo chamber. You should be looking toward how much evidence there is for a particular position as opposed to how many people hold that position: Having 20 independent pieces of evidence that mostly point in the same direction might indeed reflect a powerful consensus, while having 20 like-minded people citing the same warmed-over evidence is much less powerful. Obviously this can be taken too far and in most fields, it’s foolish (and annoying) to constantly doubt the market or consensus view. But in a case like politics where the conventional wisdom can congeal so quickly — and yet has so often been wrong — a certain amount of contrarianism can go a long way.