Posted on

An Exercise for Bias Detection

A great exercise to train your bias-detecting skills is to check on a high volume of outlets –say, eight to ten–across the political spectrum in the 6-12 hours right after a big political story breaks. I did this right after the release of the Nunes memo on Friday, Feb 2. This particular story provided an especially good occasion for comparison across sites for several reasons, including:

-It was a big political story, so nearly everyone covered it. It’s easier to compare bias when each source is covering the same story.

-The underlying story is fact-dense, meaning that a lot of stories about it are long:

-As a result, it is easier to tell when an article is omitting facts.

-It is also easier to compare how even highly factual stories (i.e., scores of “1” and “2” on the Veracity and Expression scales) characterize particular facts to create a slight partisan lean.

-There are both long and short stories on the subject. Comparison between longer and shorter stories lets you more easily find facts that are omitted in order to frame the issues one way or another.

-News outlets have had quite a while to prepare for this coming story, so those inclined to spin it one way or the other have had time to develop the spin. Several outlets had multiple fact, analysis, and opinion stories within the 12 hours following the story breaking. You could count the number of stories on each site and rate their bias and get a more complex view of the source’s bias.

I grabbed screenshots of several sources across the spectrum from the evening of Feb. 2 and morning of Feb. 3. These are from the Internet Wayback Machine https://web.archive.org (if you haven’t used it before, it’s a great tool that allows you to see what websites looked like at previous dates and times).  Screenshots from the following shots are below:

FoxNews.com

Breitbart.com

NationalReview.com

RedState.com

WashingtonPost.com

NYTimes.com

Huffpost.com

TheDailyBeast.com

Slate.com

BipartisanReport.com

 

You can get a good sense of bias from taking a look at the headlines, but you can get deeper insight from reading the articles themselves. For some sources, the headlines are a lot more dramatic than the articles themselves; for others, the articles are equally or more biased.

If you want to rank these articles (based on the articles themselves, or just on the headlines and pages below) on a blank version of the chart, I recommend placing the ones that seem most extremely biased first, then placing the ones that seem less biased. It’s easiest to identify the most extreme of a set, and then place the rest in relative positions. There’s not always a source or story that will land in whatever you consider “the middle,” but you can find some that are closer than others.

Going through this exercise is especially beneficial when big stories like this break. I know it is time-consuming to read so many sources and stories, so most people don’t read laterally like this very often, if ever (if you do, nice work!).  Doing so from time to time can help you remember that people are reading very different things from you, and increase your awareness of the range of biases across the spectrum. It can also help you identify how detect more subtle bias in the sources you regularly rely on.

Happy bias-detecting!

FOX NEWS

BREITBART

NATIONAL REVIEW

REDSTATE

 

WASHINGTON POST

NYTIMES

HUFFPOST

THE DAILY BEAST

SLATE

BIPARTISAN REPORT

 

Posted on

The Chart, 3.1 Minor Updates Based on Constructive Feedback

I’m a strong believer in changing one’s mind based on new information. That’s how we learn anyway, and I wish people would do it more often. I think it would lead to nicer online discussions and less polarization in our politics. Perhaps people don’t “change their minds based on new information” as much as they should because it is often framed more negatively as “admitting you are wrong.” I don’t particularly mind admitting I’m wrong

In any event, I’m making some minor updates, corrections, and improvements based on feedback I’ve gotten. I’ve been fortunate to hear from many of you thoughtful observers out there, and I’m so grateful that so many of you care about the subject of ranking quality and bias.

Here are the changes for version 3.1. I’m calling it 3.1 because they are mostly minor changes. I got quite a bit of feedback on these topics in particular.

  • The middle column now says “Neutral: Minimal Partisan Bias OR Balance of Biases.” I moved away from the term “Mainstream” because that term is so loaded as to be useless to some audiences. Also, there are some sources that are not really minimally biased or truly neutral; some have extreme stuff from both political sides.

 

  • The horizontal categories have been updated slightly. The “skew conservative” and “skew liberal” categories no longer have the parenthetical comment “(but still reputable),” mostly because the term “reputable” has more to do with quality on the vertical axis, and I’m doing my best not to conflate the two. The “hyper-partisan conservative” and “hyper-partisan liberal” categories no longer have the parenthetical comments “(expressly promotes views),” mostly because “promoting views” is not the only characteristic that makes something hyper-partisan. Finally, the outermost liberal and conservative “utter garbage/conspiracy theories” categories are now re-labeled “most extreme liberal/conservative.” This is, again, because the terms “utter garbage” and “conspiracy theories,” though often accurate for sources in those columns, has more to do with quality than partisanship.

 

I am writing a separate post that more specifically defines the horizontal axis and the criteria for ranking sources within them. It’s a pretty complex topic, and I’ll discuss many additional points frequently raised by those of you who have commented. I will likely have more revisions accompanying that post.

 

  • I have moved Natural News from the extreme left to slightly right. I know this may still cause some consternation among commentators that note correctly that they have a lot of extreme right wing political content. However, after categorizing dozens of articles over several sample days and counting how many fell in each category, the breakdown looked like this: About a third fell in the range of “skew liberal” to “extreme liberal” (in terms of promoting anti-corporate and popular liberal pseudo-science positions), another third were relatively politically neutral “health news,” and about a third fell into the extreme conservative bucket. There wasn’t much that fell into the “skew conservative” or “hyper-partisan conservative” categories. So even though the balance was 1/3, 1/3, 1/3, left, center, right, the 1/3 on the right was almost all “most extreme conservative,” so that pushed the overall source rank to the right. For those who are still unhappy and think it should be moved further right, take consolation in the fact that it is still at the bottom vertically, and to an extent, it doesn’t matter how partisan the junk news is as long as you still know it’s junk.

 

  • I removed US Uncut, because as some of you correctly pointed out, that site is now defunct.

 

  • I removed Al-Jazeera from the top middle, but not because I don’t think it’s a mostly reputable news source. I removed it for two reasons.

 

  1. First, many people are unclear on what I am referring to as Al-Jazeera. It is a very large international media organization based out of Qatar, (see https://en.wikipedia.org/wiki/Al_Jazeera), but it is not a very popular source for news to Americans. Americans who are familiar with it could assume that I am referring to Al-Jazeera English (a sister channel), or Al-Jazeera America (a short-lived US organization (2013-2016) which arguably leaned left), or AJ+ (a channel that provides explanatory videos on Facebook and also arguably leans left). I do think these are worth including in the chart, but I will differentiate them before including them in future versions. What I meant originally was the main Al-Jazeera site that is in English, which covers mostly international news, and which I consider a generally high quality and reputable source.
  2. Second , it is somewhat controversial because it is funded by the government of Qatar, and it has been accused of bias as it pertains to Middle East politics. This doesn’t necessarily mean that it is disreputable, or that its ownership results in stories that are biased to the left or right on the US political spectrum. However, I have only two other non-US sources on the chart—the BBC and DailyMail—which both have significant enough coverage of US politics that you can discern bias on the US spectrum. I don’t have any other internationally sources on the chart, and none that are primarily funded by a non-democratic government (the BBC is funded by the British public, NPR is publicly and privately funded in the US). Until I can specify which articles I have rated to form the basis for Al-Jazeerza’s placement, I’m going to leave it off.

Thanks for the comments so far, and please keep them coming. I appreciate your suggestions for how to make this work better and your requests for what you want to see in the future.