Posted on

An Exercise for Bias Detection

A great exercise to train your bias-detecting skills is to check on a high volume of outlets –say, eight to ten–across the political spectrum in the 6-12 hours right after a big political story breaks. I did this right after the release of the Nunes memo on Friday, Feb 2. This particular story provided an especially good occasion for comparison across sites for several reasons, including:

-It was a big political story, so nearly everyone covered it. It’s easier to compare bias when each source is covering the same story.

-The underlying story is fact-dense, meaning that a lot of stories about it are long:

-As a result, it is easier to tell when an article is omitting facts.

-It is also easier to compare how even highly factual stories (i.e., scores of “1” and “2” on the Veracity and Expression scales) characterize particular facts to create a slight partisan lean.

-There are both long and short stories on the subject. Comparison between longer and shorter stories lets you more easily find facts that are omitted in order to frame the issues one way or another.

-News outlets have had quite a while to prepare for this coming story, so those inclined to spin it one way or the other have had time to develop the spin. Several outlets had multiple fact, analysis, and opinion stories within the 12 hours following the story breaking. You could count the number of stories on each site and rate their bias and get a more complex view of the source’s bias.

I grabbed screenshots of several sources across the spectrum from the evening of Feb. 2 and morning of Feb. 3. These are from the Internet Wayback Machine (if you haven’t used it before, it’s a great tool that allows you to see what websites looked like at previous dates and times).  Screenshots from the following shots are below:


You can get a good sense of bias from taking a look at the headlines, but you can get deeper insight from reading the articles themselves. For some sources, the headlines are a lot more dramatic than the articles themselves; for others, the articles are equally or more biased.

If you want to rank these articles (based on the articles themselves, or just on the headlines and pages below) on a blank version of the chart, I recommend placing the ones that seem most extremely biased first, then placing the ones that seem less biased. It’s easiest to identify the most extreme of a set, and then place the rest in relative positions. There’s not always a source or story that will land in whatever you consider “the middle,” but you can find some that are closer than others.

Going through this exercise is especially beneficial when big stories like this break. I know it is time-consuming to read so many sources and stories, so most people don’t read laterally like this very often, if ever (if you do, nice work!).  Doing so from time to time can help you remember that people are reading very different things from you, and increase your awareness of the range of biases across the spectrum. It can also help you identify how detect more subtle bias in the sources you regularly rely on.

Happy bias-detecting!













Leave a Reply

17 Comments on "An Exercise for Bias Detection"

newest oldest most voted
Notify of
David G Shaw
Thank you Vanessa! Wonderful effort! Coincidental timing to say the least. I was trying to explain a chapter in a book I’m writing to a couple of close (politically biased) friends this morning where that explains media bias as being driven by formulated marketing efforts that more or less “flatter” the biased viewer by validating their opinion appealling to their ego, dopamine release, etc. for the sake of higher and deeper market penetration for increased advertising revenues; needless to say after two hours of reason and analogies I hadn’t gotten very far in gaining any acceptance of this as a… Read more »

Question, not a comment: during the time you have been a careful, methodical analyst of news media outlets, and focusing on the top middle tier of those outlets (AP, Reuters, NYT, etc.,), have you noticed a shift in the frequency and amount of opinion contained in their news articles?

Birrell Walsh

Thank you!

I am grateful when someone uses their education to benefit and clarify the public discourse. I also appreciate the hard work involved – so thank you a second time.

Ken Rhines
Too funny! I picked the same event to dig into, for the same reasons! I’m focusing on looking at omitted facts by horizontal category, and whether there are consistent patterns. This requires work: first reading and ingesting a lot of articles across a large sample of sites to create a complete fact “set”, and then doing all kinds of counts and summaries to see who reports –or doesn’t report– what. And then there’s the overlap of “news,” “analysis” and “opinion” just to make things interesting. But, I’ll let ya know how it goes when I’m done. Your point about how… Read more »

Hi Vanessa,

Any chance you could at The Young Turks to your media biased chart? It would be great to see where you rank them. It’s also a little funny they didn’t make the cut.


Thank you Vanessa for the chart. It is a great visual to help readers be more cognizant of bias in news reporting. I am finding it is generating lots of good discussion about media bias. One question: why did you list The Hill as “Skews Conservative.” I noticed I read some articles that skew liberal and other articles that skew conservative, it seems to swing back and forth depending on the writer, but I am not sure about overall count of articles though. I saw AllSides shows The Hill as lean right bias.

Matt Healy
Personally, I find the quickest way of getting a first read on a major story like this is to read what NY Times, Washington Post, Economist, NPR, Politico, National Review, and Wall Street Journal say about it. While The Economist is overall usually the closest of these to my own views, I think it’s important to check what a selected Conservative sources say just to make sure there aren’t any material facts that NYT and WAPO have missed or gotten wrong. I consider the WSJ Editorial policy execrable, and they often leave out facts that they find inconvenient, but they… Read more »

Thank you. Nice work. Scrolled down to the WaPo screengrab and saw the “laughable” included in a article title – after reading that, I check out. And thank you for your work on the graph, as well.

Brian B

Thanks for doing this! It is an exceptional piece of work.

A thought to help users interact would be to make a web-based version of your data with fully interactive display. That way a user could tease out (for the range over which individual sources wander. (you used ellipses) in an interactive version, you could actually show which sources are consistent, vs which have a broad range…
You could also (in the interactive version) allow the viewer to filter on media: newspapers, news-shows, …
or along the dimension of deep analysis vs timely-news.

Thanks again… keep it up.