Posted on

The Chart, Version 3.0: What, Exactly, Are We Reading?

 

Summary: What’s new in this chart:

  • I edited the categories on the vertical axis to more accurately describe the contents of the news sources ranked therein (long discussion below).
  • I stuffed as many sources (from both version 1.0 and 2.0, plus some new ones) on here as I could, in response to all the “what about ______ source” questions I got. Now the logos are pretty tiny. If you have a request for a ranking of a particular source, let me know in the comments.
  • I changed the subheading under “Hyper-Partisan” from “questionable journalistic value” to “expressly promotes views.” This is because “hyper-partisan” does not always mean that the facts reported in the stories are necessarily “questionable.” Some analysis sources in these columns do good fact-finding in support of their expressly partisan stances. I didn’t want anyone to think those sources were necessarily “bad” just because they hyper-partisan (though they could be “bad” for other reasons.
  • I added a key that indicates what the circles and ellipses mean. They mean that a source within a particular circle or ellipse can often have stories that fall within that circle/ellipse’s range. This is, of course, not true for all sources
  • Green/Yellow/Orange/Red Key. Within each square: Green is news, yellow is fair interpretations of the news, orange is unfair interpretations of the news, and red is nonsense damaging to public discourse.

Just read this one more thing: It’s best to think of the position of a source as a weighted average position of the stories within each source. That is, I rank some sources in a particular spot because most of its stories fall in that spot. However, I weight the ranking downward is if it has a significant number of stories (even if they are a minority) that fall in the orange or red areas. For example, if Daily Kos has 75% of its stories fall under yellow (e.g., “analysis,” and “opinion, fair”), but 25% fall under orange (selective, unfair, hyper-partisan), it is rated overall in the orange. I rank them like this is because, in my view, the orange and red-type content is damaging to the overall media landscape, and if a significant enough number of stories fall in that category, readers should rely on it less. This is a subjective judgment on my part, but I think it is defensible.

OK, you can go now unless you just really love reading about this media analysis stuff. News nerds, proceed for more discussion about ranking the news.

As I discussed in my post entitled “The Chart, Second Edition: What Makes a News Source Good?” the most accurate and helpful way to analyze a news source is to analyze its individual stories, and the most accurate way to analyze an individual story is to analyze its individual sentences. I recently started a blog series where I rank individual stories on this chart and provide a written analysis that scores the article itself on a sentence-by-sentence basis, and separately scores the title, graphics, lede, and other visual elements. See a couple of examples here. Categorizing and ranking the news is hard to do because there are so very many factors. But I’m convinced that the most accurate way to analyze and categorize news is to look as closely at it as possible, and measure everything about it that is measurable. I think we can improve our media landscape by doing this and coming up with novel and accurate ways to rank and score the news, and then teaching others how to do the same. If you like how I analyze articles in my blog series, and have a request for a particular article, let me know in the comments. I’m interested in talking about individual articles, and what makes them good and bad, with you.

As I’ve been analyzing articles on an element-by element, sentence-by-sentence basis, it became apparent to me that individual elements and sentences can be ranked or categorized in several ways, and that my chart needed some revisions for accuracy.

So far I have settled on at least three different dimensions, or metric, upon which an individual sentence can be ranked. These are 1) the Veracity metric, 2) the Expression metric, and 3) the Fairness metric

The primary way statements are currently evaluated in the news are on the basis of truthfulness, which is arguably the most important ranking metric. Several existing fact-checking sites, such as Politifact and Washington Post Fact Checker, use a scale to rate the veracity of statements; Politifact has six levels and Washington Post Fact Checker has four, reflecting that many statements are not entirely either true or false. I score each sentence on a similar “Veracity” metric, as follows:

  • True and Complete
  • Mostly True/ True but Incomplete
  • Mixed True and False
  • Mostly False or Misleading
  • False

Since there are many reputable organizations that do this type of fact-checking work, according to well-established industry standards, (see, e.g., Poynter International Fact Checking Network), I do not replicate this work myself but rather rely on these sources for fact checking.

It is valid and important to rate articles and statements for truthfulness. But it is apparent  that sentences can vary in quality in other ways. One way, which I discussed in my previous post (The Chart, Second Edition: What makes a News Source ‘Good’) is on what I call an “Expression” scale of fact-to-opinion. The Expression scale I use goes like this:

  • (Presented as) Fact
  • (Presented as) Fact/Analysis (or persuasively-worded fact)
  • (Presented as) Analysis (well-supported by fact, reasonable)
  • (Presented as) Analysis/Opinion (somewhat supported by fact)
  • (Presented as) Opinion (unsupported by facts or by highly disputed facts)

In ranking stories and sentences, I believe it is important to distinguish between fact, analysis, and opinion, and to value fact-reporting as more essential to news than either analysis or opinion. Opinion isn’t necessarily bad, but it’s important to distinguish that it is not news, which is why I rank it lower on the chart than analysis or fact reporting.

Note that the ranking here includes whether something is “presented as” fact, analysis, etc. This Expression scale focuses on the syntax and intent of the sentence, but not necessarily the absolute veracity. For example, a sentence could be presented as a fact but may be completely false or completely true. It wouldn’t be accurate to characterize a false statement, presented as fact, as an “opinion.” A sentence presented as opinion is one that provides a strong conclusion, but can’t truly be verified or debunked, because it is a conclusion based on too many individual things. I’ll write more on this metric separately, but for now, I submit that it is an important one because it is a second dimension of ranking that can be applied consistently to any sentence. Also, I submit that a false or misleading statement that is presented as a fact is more damaging to a sentence’s credibility than a false or misleading statement presented as mere opinion.

The need for another metric became apparent when asking the question “what is this sentence for?” of each and every sentence. Sometimes, a sentence that is completely true and presented as fact can strike a reader as biased for some reason. There are several ways in which a sentence can be “biased,” even if true. For example, sentences that are not relevant to the current story, or not timely, or that provide a quote out of context, can strike a reader as unfair because they appear to be inserted merely for the purpose of persuasion. It is true that readers can be persuaded by any kind of fact or opinion, but it seems “fair” to use certain facts and opinions to persuade while unfair to use other kinds.

I submit that the following characteristics of sentences can make them seem unfair:

-Not relevant to present story

-Not timely

-Ad hominem (personal) attacks

-Name-calling

-Other character attacks

-Quotes inserted to prove the truth of what the speaker is saying

-Sentences including persuasive facts but which omit facts that would tend to prove the opposite point

-Emotionally-charged adjectives

-Any fact, analysis, or opinion statement that is based on false, misleading, or highly disputed premises

This is not an exhaustive list of what makes a sentence unfair, and I suspect that the more articles I analyze, the more accurate and comprehensive I can make this list over time. I welcome feedback on what other characteristics make a sentence unfair, and I’ll write more on this metric in the future. Admittedly, many of these factors have a subjective component. Some of the standards I used to make a call on whether a sentence was “fair” or unfair” are the same ones in the Federal Rules of Evidence (i.e., the ones that judges use to rule on objections in court). These rules define complex concepts such as relevance and permissible character evidence, and determine what is fair for a jury to consider in court. I have a sense that a similar set of comprehensive rules for legal evidence could be developed for journalism fairness. For now, these initial identifiers of fairness metric helped me distinguish the presence of unfair sentences in articles. I now use a “Fairness” metric in addition to the Veracity scale and the Expression scale. This metric only has two measures, and therefore requires a call to be made between:

  • Fair
  • Unfair

By identifying a percentage of sentences that were unfair, I was able to gain an additional perspective on what an overall article was doing, which helped me create some more accurate descriptions of types of articles on the vertical quality axis. In my previous chart (second edition), the fact-to-opinion metric was the primary basis for the vertical ranking descriptions, so it looked like this:

In using all three metrics, 1) the Veracity scale, 2), the fact-to-opinion Expression scale, and 3) the Fairness scale, I came up with what I believe are more accurate descriptions of article types, which looks like this:

As shown, the top three categories are the same, but the lower ranked categories are more specifically described than in the previous version. The new categories are “Opinion; Fair Persuasion,” “Selective or Incomplete Story; Unfair Persuasion,” “Propaganda/Contains Misleading Facts,” and “Contains Inaccurate/ Fabricated Info.” If you look at the news sources that fall into these categories, I think you’ll find that these descriptions more accurately describe many of the stories within the sources.

Thanks for reading about my media categorizing endeavors. I believe it is possible (though difficult) to categorize the news, and that doing so accurately is a worthy endeavor. In future posts and chart editions I’ll dive into other metrics I’ve been using and refining, such as those pertaining to partisanship, topic focus (e.g., story selection bias), and news source ownership.

If you would like a blank version for education purposes, here you go:

Third Edition Blank

And here is a lower-resolution version for download on mobile devices:

Leave a Reply

335 Comments on "The Chart, Version 3.0: What, Exactly, Are We Reading?"

Notify of
avatar
Sort by:   newest | oldest | most voted
kayla
Guest

Where is Vice News?

Stacy
Guest

Yes, where is Vice News?
Thanks for this, it’s fantastic!!

chris
Guest

Hi Vanessa. Do I understand that this chart is based on articles off a particular news source website? This chart has nothing to do with news shows on TV?-right?

Mike de Martino
Guest

Please add Truthdig to the list they have been around as long as the Huffington Post
and you might look at TYT

Sarah Ellerbrock
Guest

I agree on the TYT

Christina DeJong
Guest

Thank you for updating, Vanessa!

Based on your first chart, I started following only the sources in the gray circle and the green oval and dropped everything else. One of the sources I started reading was The Hill, and I would argue that it skews more liberal than is demonstrated in your charts.

Having said that, I think your analysis is fantastic and I share it with my college students to help them evaluate the quality of media sources.

denice everham
Guest

it is funny i used to read the Hill and found it far right something must have happened ???same with Politico ???

Wendy
Guest

Where is Fox News? I don’t see it on the chart except for the Daily Wire.

Michael Turton
Guest

Wall Street Journal = minimal partisan bias.

No, seriously? It’s just more articulate version of Fox. It’s why stuff like this is so dangerously wrong. You’ve defined the center-right/rightist Establishment as “the center” and then located everything relative to the right-wing Establishment.

Dennis
Guest

I think you need to separate the WSJ reporting from the WSJ editorials/opinion pieces.

Carl
Guest
I agree. And though this just may betray bias, business interests need reliable information to make good choices about where to get the most bang for the buck (ie, where the largest profits can be extracted from production, which implies that labor, environmentalists and social movements must be rendered as powerless as possible, whether by death squads or the skillful adoption and application of laws passed by corporate-captured governments and well-bribed politicians). The editorial pages supply the idealogical justifications for the vast systems of exploitation, repression that results. It’s a bit like the hard sciences, which require the unrestricted use… Read more »
Bruce S Lindeke
Guest
The WSJ’s investigative reporting is still top notch. While the editorial content has always leaned conservative, it still had some integrity in my opinion. But they lost a lot of their better opinion writers like Bret Stephens and others once the Republican party became the Trump party. Over the last year their editorials have moved much further to the world of Trump and the far right. It’s sad. I’ve subscribed to the WSJ for over 30 years and even when I didn’t agree, most of the time I could respect their logic and view. But after William McGurn recently said… Read more »
ishi crew
Guest
I didn’t read this well but it looks similar to ‘content analyses’ and some older approaches (which did something like paragraph analyses of social science papers to see if the words used in a paragraph were related in a consistant manner , or instead just appeared to be some random collection strung together to appear to make an argument. An automatic computer program ‘postmodern generator’ can write academic papers automatically by combining phrases and words, and arernt too different from ‘real’ ones, and similar programs exist for computer sciences and art.) Lots of studies do content analyses of internet web… Read more »
Alex
Guest
Much better than the last version but a few fixes still needed. To say “David Wolfe” and “Natural News” are in any way “left” is insulting. Its pure quackery thats neither left or right but rather belongs underneath Enquirer. Daily Kos and Alternet belong where Huffpo is and Occupy Democrats and U.S. Uncut belong where they are, to compare to the bias/quality of the sources across from them on the right. Also, to be fair, putting AlJazeera in right in the middle is equally rediculous, who are we kidding, its editorial slant is clearly quite left, it belongs between WaPo… Read more »
PDM
Guest
I have to agree here. I was about to reply with mich of this same content. David Wolfe and Natural News have nothing to do with the left or with anything liberal, only with nutjobs. As mentioned, this belongs with The Enquirer and other tabloids. I disagree with where Daily Kos is located on the chart. I would move it up about between Mic and Mother Jones. And I’d move CNN, as well, up into the gray. Don’t let the far right’s propaganda skew your placement of CNN, which is what seems to have occurred here. There are individual programs… Read more »
PDM
Guest

Please excuse my typos. I’m typing on a smart phone with an injured hand and two cats on my lap. I clicked send a bit too quickly and missed my chance to proofread!

Treflip
Guest

Very interesting. I am always in question with each news source when hearing about a new story. People tend to say one source is pushing one view or the other. It is very positive to see some of the main stream outlets (abc, NBC, cbs, wallstreet, npr) in the fact/mainstream section. Thanks for taking the time to provide this type of information. I don’t have the time to do things like this!

Heather Campion
Guest

Where is Salon?

Jess
Guest

Awesome work! My aunt shared this on fb.
I get some news from independent.co.uk and would be interested in where that source falls within the metrics. Thank you.

Kay
Guest

I would urge you to look at TalkingPointsMemo. I _think_ they’ve got enough of a readership to be considered in a broad study like this. They have many active investigative journalists and have been active for over 15 years. I consider them left leaning but reliable. I’d be interested in seeing your take.

Thanks for this analysis. It’s really good to have a summarized graphic for reference when someone brings up a “news source”.

eyelessgame
Guest

That’s the first one I looked for too.

Bernardo Zapata
Guest

Great job Vanessa. I think you greatly improved the chart’s accuracy.

Anthony
Guest

Where are the Young Turks?

WP
Guest

this is so helpful and interesting. Did I miss GQ and New York magazine in the graphic?

Becca
Guest

Have you considered where specific shows might land on the grid? For example, I see where MSNBC is placed, but I’m curious where you might place Rachel Maddow in relation to that… or how her show contributes to the placement you ultimately decided on.

Jo Brennan
Guest

TYT news?

Caroline
Guest

Thank you for your work. Can you tell us what logo is under CNN?

Marie Walker
Guest

I don’t see the Wall Street Journal, which I would say is middle Conservative.

Marielena
Guest

Where would you rate ProPublica?

Mac Moye
Guest
I so very much appreciate this chart. Thank you for creating it. I have made it my business to spend a little time every day challenging people’s social-media use of those sources in the red box, even though I don’t know exactly how to categorize them. As it turns out, my sources are all in the green box, which makes me feel vindicated! I’m a little surprised to see Time clutching onto the yellow line, but also to see it represented as right-leaning. I’ve been a hard-copy subscriber since 1970, and I would have placed it about where NPR is.… Read more »
John
Guest

Is there a reason you did not list the Sinclair Broadcast Group?

Riley
Guest

Great stuff! Congrats and thanks! With just a little tweaking, this could also be a guide to just having a quality discussion intended to understand and explore, rather than dismiss and overcome.

Roxanne Griffin
Guest

Thanks for providing this well thought out and well presented information. Did I just over look “Salon” or was it not rated?

Tim
Guest

Clearly your opinion is biased. CNN reputable? And your other posts are clearly trying to manipulate Trump supporters. Trump is the best thing this country has seen all century. If you cannot understand why the country is thrilled Hillary lost it is futile to explain. Good luck and stop crying.

amanda
Guest

Vice news?

Carl
Guest
Thanks for all the good work you’ve put into this ongoing project. It’s a valuable service for those who are interested enough to benefit from it, to whatever depth they might use your chart and presentation. One thought regarding the amount of detail offered is that you could provide one large chart with all of the news sources you’ve considered, and then maybe two or three less detailed editions in which you’ve eliminated some or many of the sources to simplify the view for those who are less interested in the more minor sources. Or, perhaps you could provide one… Read more »
Sharon Alexander
Guest

I appreciate your comment about an infotainment scale. I recommend you look at mine about corporatocratic bias in the media. I expect there would be a great overlap in the two categories.

Scott
Guest

I cannot savec the image or read it on my mobile device. Please makr the image donwloadable.

ann moriarty
Guest
God love you, Vanessa for your efforts! While I am a professional librarian, I also serve as a university instructor. Might we need a new ‘blank chart’ that includes the Green/Yellow/Orange/Red rectangles? I teach a 3.0 credit course in the business school; information literacy & research. Evaluating resources is a critical skill addressed in competency standards in higher education. Standard 3: The information literate student evaluates information and its sources critically and incorporates selected information into his or her knowledge base and value system. http://www.ala.org/Template.cfm?Section=Home&template=/ContentManagement/ContentDisplay.cfm&ContentID=33553 I use your chart as an in-class activity. I pass out paper prints of news… Read more »
Josh
Guest

The only thing wrong is that infowars should have broken off the chart to the bottom and be starting its own chart that calls all other charts fake news. I mean, Louise Mensch is a weird conspiracy theorist, but not “JFK and Hitler are both alive on the secret UN moon base with the lizard people leaders of the Illuminati” level weird conspiracy.

Zeth
Guest

Forbes? I have the impression that it’s generally a bit to the right, but some articles are definitely outliers in that assessment.

Also interested in TYT, Jimmy Dore, and Independent.co.uk as others have requested.

Travis
Guest

Do you have a spreadsheet with the actual values given to each outlet or could you zoom in on the grey circle? It’s really messy and some items I don’t know if that is actually where they are or if its just best fit for the image. An interactive chart where you have the dots for locations and mouse over to see the outlet would be nice.

Ian
Guest

This is a brilliant chart Vanessa, thanks for the update.

Can you elaborate on why you chose to place MSNBC higher than Fox News? Maybe because I skew somewhat conservative it seems odd, but every time I turn on MSNBC it’s Rachel Maddow or that guy with the stupid glasses doing some really biased hit job. I’m not fan of Fox News but they seem like both sides of the same coin.

Gareth
Guest

The up-down axis seems solid, but I’m not sure about the left-right one. Why is Jacobin to the right of Occupy Democrats?

Tom
Guest

Wow! This chart should be in the national news!

Steve Peister
Guest

Vanessa, it’s good and useful work and I’m impressed. Being an amateur linguist, I can see some areas of congruence between your work and my fiddling around. You may have to break the chart down into segments, just to be as inclusive and detailed as possible. I wonder if or where you would consider putting satire and political humor articles or do they need to be on a different chart ? I mean, MAD magazine has been a major influence on adolescent thought for years.

Melanie
Guest

I love this! Great work! When you have time I’m curious where you’d put the Daily Skimm.

Craig Fluck
Guest

Please encourage C-Span to use this chart whenever they have a guest on.

Claudius
Guest

hey, did you consider ranking Business Insider, too?

Claudius
Guest

great map! really interesting! did you consider ranking Business Insider, too?

marcus
Guest

First, thank you so much for your work! I’m sure this must have taken a great deal of time! I saw it on one of my Facebook friends walls and shared it onto my page. In these times it is now more important than ever to know which news agencies we can count on to give us the truth so that we can make informed decisions. Thank you! What a wonderful public service you have done!❤️

marcus
Guest

First, thank you so much for your work! I’m sure this must have taken a great deal of time! I saw it on one of my Facebook friends walls and shared it onto my page. In these times it is now more important than ever to get to the news agencies that will give us the truth so that we can make informed decisions. Thank you! What a wonderful public service you have done!❤️

Gary
Guest

I’d like to request The Daily Show and Last Week Tonight. Personally, I think TDS would probably fall near Mother Jones, and LWT near The New Yorker. Both clearly have a somewhat liberal slant, but their primary target has always been absurdity.

martouf
Guest

Please add CBC (cbc.ca) CTV News (ctvnews.ca) and The Star (thestar.com)

Sean
Guest

I’m wondering if the size of the heading has any content/context. If not it could be used to represent the breadth of the two axes. Meaning, for example, the extent to which National Review skews between conservative and h.p. conservative. More work but maybe makes the tool even more interesting/informative.
Thanks!

Jeremy
Guest

Legalize marijuana. Legalize or regulate recreational drugs.
Legalize all adult consensual sexual activity. End government restrictions on abortion.
End police violence. End justice systems that prey on the poor, uneducated, and minorities.
Protect the 1st and 4th amendments. Restore rights to criminal defendants.
End wars. Reduce military spending–drastically.
Borders are arbitrary. Everyone should have the right to travel (immigrate) wherever they want.
Separate church and state. Give LGBT the same marriage rights as heterosexuals.

Q: Guess which organization this is.
Huffington Post?
Democracy Now?

A: Reason

Brian S.
Guest

Well said, Jeremy. Thank you.

E. Steev
Guest

Have you thought about making this a more interactive version? Something database-driven that can update automatically as you enter additional articles etc; that can be filtered by the viewer for specific networks/sources, and perhaps by timespan? I think this has potential to be a powerful tool if it can continue as such.

Florence Vincent
Guest

The owner of Natural News is a libertarian.

Achaessa James de Garibay
Guest
Achaessa James de Garibay

Fabulous work, Vanessa. Much appreciated. Keep the updates coming!

Steven B
Guest

I can comprehend why you placed MSNBC there, but not why NBC News isn’t right next to it, because they are the same operation. CNN’s place that low is questionable. And, it seems like the legacy of the Big 3 evening newscasts is influence their respective organizations place on the chart even though CBS News has a decidedly left leaning slant compared to ABC News or PBS Newshour for example. The categories and distinctions are good though.

Roy E Pardee
Guest

Thanks so much for your work on this–really good stuff.

I’m curious about how quantitative the metrics here are. Would you say this is closer to ordinal-scale information here (e.g., we know that CNN is to the left of The Economist, but can’t really tell if CNN is really *twice* as liberal-leaning as USA Today) or more like actual interval-level information? The neat symmetry is what makes me wonder–that seems too convenient to be real.

Thanks!

wpDiscuz