The Facebook page of Ben Shapiro, the conservative personality who runs the Daily Wire, frequently makes the top ten list shared by Kevin Roose, a technology columnist for the New York Times. | Jessica Pons/The Washington Post via Getty Images
Untangling the influence of right-wing media is hard, especially with limited data.
A few years ago, New York Times columnist Kevin Roose decided he wanted to get a better look at what was happening on Facebook. Because Facebook does not offer much data about activity on its platform, Roose turned to a Facebook-owned tool called CrowdTangle, which lets journalists and researchers see which public posts are getting the highest levels of engagement. It seemed revealing at first glance.
“I started collecting this data without any kind of agenda. I was just fascinated by it personally,” Roose told Recode. “I was fascinated by how different the world that I was seeing on Twitter was from the one that the data showed was happening on Facebook.”
There, he says he discovered what he calls a “parallel media universe” where extreme right-wing pages reign supreme. Consistently, Roose found, conservative pages were beating out liberals’ in making it into the day’s top 10 Facebook posts with links in the United States, based on engagement, like the number of reactions, comments, and shares the posts receive. That seems to provide evidence against the notion that Facebook censors conservatives, a complaint often trotted out by Republicans despite lacking any significant data to support their claims of systemic bias. In fact, these numbers would make it seem that Facebook is almost entirely dominated by conservative voices. But the problem is, we don’t actually know if this is the case because engagement with public posts only measures one part of what users do on Facebook’s platform and can’t really reveal the extent of conservative influence there.
Critics have insisted that Roose’s numbers can’t tell the whole story. In late July, the head of Facebook’s News Feed, John Hegeman, chimed in, emphasizing that “these lists don’t represent what most people see” on Facebook, pointing to non-public data showing that the list of what’s most seen on Facebook tells a less partisan tale, with outlets like the Los Angeles Times and BuzzFeed capturing high amounts of attention. This prompted several journalists to ask why Facebook won’t just make the type of data Hegeman cited publicly available.
Meanwhile, Roose has doubled down on his analysis of what gets the most engagement on Facebook. In July, he created a standalone Twitter account, called Facebook’s Top 10, devoted to these calculations, which now has over 16,000 followers (and copycat accounts that do the same roundup for posts in Italy and in Sweden). While acknowledging its limitations, he’s floated that the engagement data can indicate what’s really “a rough gauge of what’s grabbing America’s attention” and serve as a “useful reality check for Democrats.” Essentially, Roose thinks this Facebook activity might be the “silent majority” that’s more supportive of Trump than liberals would like.
There’s now a running debate among academics, analytics experts, and observers like Roose around what we know about what’s happening on Facebook and why. Dartmouth political scientist Brendan Nyhan recently argued that “likes,” comments, and shares are just a small part of what people actually see on Facebook, and that it’s difficult to draw conclusions from these interactions alone or to know what they might mean for an election. CrowdTangle co-founder and chief executive Brandon Silverman conceded that there are limits to the data his tool provides, but he argues it’s still helpful for understanding how public pages behave.
So now, weeks before a pivotal presidential election, everyone is arguing about what’s happening on Facebook and what it means. Everyone seems to agree that focusing on the available engagement data has some serious limitations and that we need more data to understand the goings-on of the most politically significant social media network in the United States, if not the world. Trouble is, we don’t have any idea when, or if, Facebook will lift that curtain.
On Facebook, engagement is just one part of the story
Lots of stuff happens on Facebook, but what’s publicly available is not a perfect representative slice of the platform overall. Again, what CrowdTangle measures — and what Roose’s Top 10 lists catalog — is engagement with public posts on Facebook. But your interactions in the public spaces of Facebook represent a small chunk of what you do on the platform. Many posts, messages, groups, and pages are private, so what you do there doesn’t get captured by CrowdTangle.
Roose says he tries to focus his research by looking at US-based posts that include links from pages. That excludes, for instance, posts that are just status updates or posts that just include an image. With these search constraints in place, Roose says he’s finding a regular pattern of public right-wing pages capturing many more interactions than their left-leaning counterparts.
The top-performing link posts by U.S. Facebook pages in the last 24 hours are from:
1. Franklin Graham
2. Donald Trump 2020 Voters
3. Ben Shapiro
4. Ben Shapiro
5. David J. Harris Jr.
6. Fox News
7. Sarah Palin
8. Ben Shapiro
10. Robert Reich
— Facebook’s Top 10 (@FacebooksTop10) September 7, 2020
If you’re a liberal, it might worry you that, according to this metric, posts from pages belonging to Diamond and Silk or Ben Shapiro are regularly outperforming more progressive content like, say, the page belonging to Occupy Democrats. That counts posts like Ben Shapiro commenting on Nancy Pelosi’s recent hair salon visit (and linking to a Daily Wire article about it). What doesn’t get counted, as an example, is your friend posting “Vote for Joe Biden!” as a status only available for view by friends.
But again, the CrowdTangle engagement data doesn’t take into account much of what people do on Facebook. According to Aviv Ovadya, the founder of the Thoughtful Technology Project, CrowdTangle can be useful for understanding what public Facebook is up to, but there’s much more to Facebook than that. “There are better ways of getting at what the entire set of Facebook users are interacting with,” said Ovadya, pointing to Facebook data from another firm called NewsWhip.
The engagement that CrowdTangle measures involves users making a certain type of effort. Engaging with content on Facebook is not the same thing as seeing it or agreeing with it. It’s also something that not everybody does. Consider how often you actually take the step to like, comment on, or share content from a public page instead of just passively scrolling through your News Feed. Different people also engage in different ways.
“Lots of people use Facebook five minutes a day or 20 minutes a week,” said Nyhan, the Dartmouth professor. “There’s some people on Facebook eight or 10 hours a day. So you can get huge engagement numbers catering to that deeply devoted base of hyper-engaged Facebook users.”
And then there’s the hate-like and the hate-share. This is what happens when someone engages with a post from a page but doesn’t necessarily support what the post says.
“Reactions are both positive and negative,” said Kathy Qian, a data scientist who co-founded Code for Democracy. “Without directionality to the ranking of reactions, it’s hard to know if people are reacting positively because they agree or they’re hate-commenting or hate-angry-facing, basically.”
The fact that engagement data can be so misleading is why people like Hegeman, the Facebook News Feed boss, have suggested that a better mechanism for tracking activity on Facebook might be measuring reach: how many people actually see a particular post, link, or publisher. But even though the company has argued that reach data offers a more complete picture of what’s popular, Facebook seems shy about publicly sharing that information on a regular basis.
“I would give my left arm to track reach on Facebook,” Roose told Recode, conceding there’s a lot that CrowdTangle doesn’t capture. “If Facebook would like us to be tracking something other than interactions on public pages, groups, and verified profiles, it has a very easy way of making that possible, and it just hasn’t done it so far.”
Of course, data about engagement on Facebook isn’t useless. It does reflect what kinds of posts people are willing to devote effort to responding to, and it’s also one of the factors that influence recommendations throughout the Facebook platform.
But putting too much stock into the type of engagement data Roose has been sharing raises questions about how Facebook handles content posted by conservatives. Censorship on social media has been a hot-button issue for conservatives since at least 2016, when Gizmodo reported that some Facebook employees suppressed conservative voices in Facebook’s trending news section, which the company later removed from the site. Despite the protestations of certain Republican lawmakers, there’s no strong, empirical evidence that Facebook is systematically biased against conservatives. In some cases, the company has directly intervened to help right-wing voices. And again, the high levels of engagement on right-leaning Facebook pages would seem to suggest that conservative content is more popular than ever.
Still, high levels of engagement on Facebook is not quite proof that there’s a silent majority capable of swinging the election one way or the other. It might be more accurate to say that this heightened activity suggests that conservatives have found more concentrated and highly engaged audiences than liberals have with these kinds of posts.
“There is a thriving pages ecosystem on Facebook, where highly emotive content performs well,” Nyhan explained, “and many conservative publishers appear at the top of the highest-performing pages and URLs for generating engagement.” He emphasized that what people do on Facebook also doesn’t necessarily correlate with what they do in the voting booth.
Different data tells different stories about Facebook. More data could help.
One of the challenges in studying what’s politically going on for Facebook’s users is that different users use Facebook differently, and it’s very easy to end up essentially comparing digital apples to digital oranges.
When people do stuff on Facebook, they do it differently for different types of content. Silverman, the CrowdTangle chief, points out that people are more likely to interact with political content without actually clicking through to read it, while the exact reverse is true of celebrity gossip reports. Looking at the most-engaged public posts of all types of Facebook content — not just links but also photos, videos, and so on — reveals a snapshot of what’s popular on Facebook that’s somewhat different from what Roose is broadcasting, Silverman says.
If you look at data beyond the engagement on posts from public pages, the story of what’s popular on Facebook looks a bit different. NewsWhip, a social media analytics firm, calculates the most popular publications on Facebook by looking at reactions, shares, and comments to links that are shared not only publicly on the social network but also privately. Monthly rankings from NewsWhip show a mix of ideological leaning in outlets whose content is performing well on Facebook.
Of course, what the New York Times is trying to accomplish on Facebook might differ from the objectives of Fox News. Some of these actors, such as the Daily Wire, have their content boosted through systematic, coordinated networks, which might explain that site’s recent climb to huge levels of engagement. Whether conservatives appear to be getting more engagement overall, however, is up for debate. As the NewsWhip data showing engagement data from both public and private pages reveals, total engagement on Facebook is a bit more balanced than what Roose’s top 10 lists suggest.
But as those lists reveal, there is evidence of a trend in which certain kinds of conservative content — mostly emotion-driven, deeply partisan posts — attract large levels of engagement. NewsWhip on Tuesday reported that the most engaged articles on Facebook in the past 24 hours included a slew of conservative media articles.
“Anger travels really, really well on Facebook,” Roose told Recode. “I think the commentators that have sort of excelled at packaging grievances and presenting them to people in an attractive way have really gotten a lot of mileage.” Noting he doesn’t have supporting data, he hypothesizes that liberals, generally, trust a more fragmented set of sources of media than conservatives, which could contribute to why right-wing success makes it to the top of his list.
Facebook, for its part, did not comment on why conservative pages do well by this metric.
Facebook is important, even if we don’t know how the platform matters
Even if we were able to understand the full extent of activity on Facebook, the link between behavior on the platform and real-life political behavior isn’t clear. That’s not to say other mechanisms, like polls, are perfect at indicating voting behavior either, but the actual role of Facebook in political discourse is still difficult to truly understand.
That’s probably why Facebook announced in late August that it would bring on social scientists, including Nyhan, to help study the influence of Facebook’s systems on American democracy. But for those hoping to understand the platform in real time — or at least before the election — that won’t be so satisfying. For those watching Facebook’s CrowdTangle data in the meantime, don’t get your hopes up.
“There’s a lot of other data points we’d also love to add,” CrowdTangle’s Silverman told Recode. “And those are all conversations that are constantly happening for us internally.” But it wasn’t clear when, how, or if that would happen.
In the meantime, Roose insists that media outlets keep their eyes glued to the biggest social media platform and work with the data they do have. “I think people feel like I’m trying to prove a point by doing this, like I’m making an argument for this data,” he says. “I guess the argument that I feel like I’m making is that people should pay attention to what’s happening on Facebook.”
Open Sourced is made possible by Omidyar Network. All Open Sourced content is editorially independent and produced by our journalists.
Help keep Vox free for all
Millions turn to Vox each month to understand what’s happening in the news, from the coronavirus crisis to a racial reckoning to what is, quite possibly, the most consequential presidential election of our lifetimes. Our mission has never been more vital than it is in this moment: to empower you through understanding. But our distinctive brand of explanatory journalism takes resources — particularly during a pandemic and an economic downturn. Even when the economy and the news advertising market recovers, your support will be a critical part of sustaining our resource-intensive work, and helping everyone make sense of an increasingly chaotic world. Contribute today from as little as $3.