Posted on

The Chart, Second Edition: What Makes A News Source “Good?”

In my original news chart, I wrestled with the questions of what made news sources “good” and came up with some categories that generally resonated with people. I ranked sources on a vertical axis with those at the top ranked as “high quality” and those at the bottom as “low quality.” I characterized the sources, from top to bottom, in this order: Complex, Analytical, Meets High Standards, Basic, and Sensational/ Clickbait. This mostly works, because it results in sources regarded as high-brow or classy (e.g., The Atlantic, The Economist) being ranked high on the axis, and trashy sources (e.g., Addicting Info, Conservative Tribune) being ranked low, and most sophisticated news consumers agree with that. However, the vertical placements ended up causing me and others some consternation, because some of the placements relative to other outlets didn’t make sense. The most common questions I got were along these lines:

“Does FOX News really “meet high standards,” on par with something like the New York Times?” (I think no.)

“Is USA today really that bad?” (I think no.)

“Is Slate really “better” or “higher quality” than, say, AP or Reuters just because it is analytical?” (I think no.)

“Is CNN really that bad?” (I think yes.)

These questions and my instinctive responses to them made me want to reevaluate what makes news sources high or low quality.

I believe that answer to that question lies in what makes an individual article (or show/story/broadcast) high or low quality. Article quality can vary greatly even within the same news source. One should be able to rank an individual article on the chart in the same way one ranks a whole news source. So, what makes an article/story high or low quality? It’s hard to completely eliminate one’s own bias on that issue, but one way to try to do it consistently is to categorize and rate the actual sentences and words that make it up its headline and the article itself. In order to try to rank any article on the chart in a consistent, objective-as-possible manner, I started doing sentence-by-sentence analyses of different types of articles.

In analyzing what kind of sentences make up articles, it became apparent that most sentences fall into (or in-between) the categories of 1) fact, 2) analysis, or 3) opinion. Based on the percentages of these kinds of sentences in an article, articles themselves can be classified in categories of fact, analysis, and opinion as well. Helpfully, some print newspapers actually label articles as “analysis” or “opinion.” However, most news sources, especially on TV or the internet, do not. I set about analyzing stories that were not pre-labeled as “analysis” or “opinion” on a sentence-by-sentence basis. I discovered that my overall impression of the quality of an article was largely a function of the proportion of fact sentences to analysis sentences to opinion sentences. As a result, I classified stories into “fact-reporting,” “analysis,” and “opinion” stories. Ones with high proportions of “fact” sentences (e.g., 90% + fact statements) were what I refer to here as traditional “fact-reporting” news pieces. These are the kinds of stories that have historically been the basis of late 20th century-to-early-21st century journalism, and what people used to refer to exclusively as “news.” They are the “who,” “what,” “when,” and “where” pieces (not necessarily “why”). I classified ones with high proportions of “analysis” sentences (e.g., 30%-50% analytical statements) as “analysis” stories, which are the types of stories commonly found in publications like The Economist or websites such as Vox. I classified stories with high proportions of opinion sentences (e.g., 30%-50% opinion statements) as “opinion,” which are typically the types of stories found on websites such as Breitbart or Occupy Democrats.

(If you’ve made it this far, bless your heart for caring so much about the news you read.)

In the past, national evening news programs, local evening news programs, and the front pages of print newspapers were dominated by fact-reporting stories. Now, however, many sources people consider to be “news sources” are actually dominated by analysis and opinion pieces. This chart ranks media outlets that people consider to be, at some level “news sources,” even though many of them are comprised entirely of analysis and opinion pieces.

In my previous version of the chart, I had regarded analysis pieces as “higher quality” than the fact-reporting pieces because they took the facts and applied them to form well-supported conclusions. I like analytical writing, which is essentially critical thinking. However, analysis has a lot in common with opinion, and writing that is intended to be analytical often strays into opinion territory. (Note—I’m defining “analysis” as conclusions well supported by facts and “opinion” as conclusions poorly supported or unsupported by facts). Fact-reporting articles—true “scoops”—typically have the intent of just reporting the facts and typically have a very high percentage (e.g., 90%+) of fact-statement sentences, whereas both analysis and opinion articles have the intent of persuading an audience and often have a comparatively high percentage of analysis and opinion statement sentences (30%-50%). So, although I initially had the quality axis of “news” laid out top to bottom as:

That ranking is more reflective of the quality of writing rather than the quality of news sources. Good analysis is often written persuasively and well, fact-reporting is often written directly but well, and opinion writing is often (but not always) written poorly or is most easily discredited.  I submit that given the confusion caused by the overwhelming number of organizations proclaiming to be (or which are commonly confused with) “news sources,” it is more important to rank the quality of news sources than the quality of writing. I further submit, for reasons outlined below, that the percentage of fact reporting articles and stories should be used as the most determinative factor by which a news source is ranked in quality on this chart.

Therefore, I believe a more relevant ranking of the quality of news sources would be:

 

 

I assert that one of the biggest problems with our current news media landscape is that there is too much analysis and opinion available in relation to factual reporting.  New technologies have given more people more platforms to contribute analysis and opinion pieces, so many “news sources” have popped up to compete for readers’ attention. Unfortunately news consumers often do not recognize the difference between actual fact-reporting news and the analysis and opinion writing about that news.  This increase in “news sources” has not corresponded with an increase in actual journalists or news reporting, though. Many local and national print news organizations have reduced their numbers of journalists, while many of the biggest ones have merely maintained similar numbers of journalists over the past 10 years or so.
Furthermore, primarily analytical news sources also have several downsides. One downside is that they can alienate news consumers by making what people consider “news sources” so complex or partisan that it is tiring to consume any “news.”  For example, CNN, MSNBC and FOX News, which are primarily analysis and opinion-driven, can make news consumers too weary to pay attention to fact-based reporting from, say, AP or Reuters. Another problem with analysis and opinion-driven news sources is that it can be difficult for casual readers to differentiate between good analysis and pure opinion.
There are several good reasons why we should value fact-reporting sentences, fact-reporting articles, and fact-reporting news sources high on the quality scale of news, at least on this chart. For one, reported facts take a lot of work to obtain. They require journalists on the ground investigating and interviewing. Once a story is reported, dozens, hundreds, or thousands of other writers can chime in with their analysis or opinions of it. This is not to say analysis and opinion writing isn’t important. The critical thinking presented in analytical writing—especially good, complex analysis—is essential to public discourse. Our society’s best ideas are advanced by analytical articles. This piece you are reading now is analytical. But analysis in the news wouldn’t even exist without the underlying factual reporting.

For example, AP and Reuters have maintained around 2,000-2,500 journalists each over that time, while the New York Times and Washington Post have fluctuated in the 500-1000 range over the same period. The value of these organizations with large staffs of journalists, editors, and other newsroom employees is hard to overstate; not only do they provide a majority of the fact-reporting stories everyone else relies on, but they have the capacity to provide high-quality editorial review that stands up to industry scrutiny. In contrast, even some of the most popular analysis and opinion sites can be run with just a few dozen writers and staff; the number of these “news” websites, news aggregator websites, blogs, and podcasts has seemingly grown exponentially.

I believe improvements to our media landscape can be made if two things happen: 1) if news consumers start valuing factual reporting much more and analysis/opinion articles much less and 2) if news consumers become accustomed to differentiating articles in those categories.. Regarding point #2, I think it would be helpful if we narrowed the definition of “news” to only refer to fact reporting, and referred to everything else as “analysis” or “opinion.” It would be helpful if people could recognize the relative contributions of fact-reporting news organizations versus analysis and opinion sources. If people recognize just how much of what they read and watch is intended to persuade them, they may become more conscious and thoughtful about how much they allow themselves to be persuaded. One can hope.

To contribute to those goals, I’ve reordered the chart to value fact-reporting articles as the highest quality and everything else lower, even though there is some really excellent analysis out there. As a baseline, news consumers should understand when something is news (fact-reporting) and when it is not. On the new chart, the sources with the best analysis, but little reporting are at the top, but right under the sources that are comprised of high percentages of reporting articles. The most opinion-driven sources are at the bottom. There’s room for other things at the bottom below pure opinion, which can include sources that are sensationalist, clickbait, frequently factually incorrect, or which otherwise don’t meet recognized journalism standards.

On this version, I’ve included a number of different sources, mostly in the analysis and opinion categories, and left the most popular mainstream sources from the original chart, but have reordered some of them. Now, the rankings are more consistent with my initial answers to the example questions at the beginning of this post. Fox News is now ranked far lower than the New York Times for two main reasons; one, Fox News is dominated by opinion and analysis, and two, it has gotten precipitously worse in other measures (sensational chyrons, loss of experienced journalists, hyperbolic analysis by contributors, etc.) within the last six months. USA today, despite its basic nature, has been elevated because of its high percentage or fact-reporting stories. Slate, though it provides thoughtful, well-written analysis, is ranked lower than AP and Reuters, which better reflects their relative contributions to the news ecosystem. CNN still sucks, but it is clearer why now; CNN has the resources to provide twenty-four hours of news—it could provide Americans with a detailed global-to-local synopsis of the world—but instead it chooses to spend 5% of its time fact reporting a handful of stories, comprising mostly American political drama and maybe one violent leading world news story, and 95% on analysis and opinion ranging from the competent to the inane.

My analysis of news sources in the manner I’ve described herein has revealed that individual stories can and should be ranked on the chart in the same manner, and that individual stories can be placed in different places than the news sources in which they are published. I’ll be putting out individual story rankings and reasoning for those rankings from time to time for those that are interested. I’ll also take requests for rankings of sources and individual stories in the comments and on twitter. Thanks for reading and thinking.

*Update: a high-resolution PDF version is available here: Second Edition News Chart.V2

And a blank version is here: Second Edition Blank

 

32 thoughts on “The Chart, Second Edition: What Makes A News Source “Good?”

  1. I am currently reading the National Review for the conservative take on major news stories. I would be interested to know where that would fall on the chart.

    Great chart, and even better documentation. Thank you for this work!

  2. Will you rank TeenVogue and GQ? I’m seeing (unexpectedly!) interesting analysis/opinion articles from them online?

    1. I will in future editions. They do have good writers and articles, such as those by Lauren Duca as of late. They are a bit harder to categorize, though, as are other similar magazines such as Rolling Stone. This is because their focus is industry specific (i.e., teen fashion, men’s fashion, and music), and a low percentage of their articles are on news topics outside of those industries. I’m planning on putting out rankings of individual articles in the future for these instances.

  3. Disappointed that you did not include ‘The Guardian’

    1. It’s on the original version of the chart. Look back a couple of posts. 🙂

  4. Im having trouble with your “Mostly Noise” label. Is it referring to just that centre, bottom bubble? to all sources in blue bubbles? why is CNN mostly noise but FOX isnt? I feel like FOX is only “worse” than CNN because it has more political bias, and I see that in the vertical and horizontal placement of those two sources, but the “Mostly Noise” label is confusing.

    1. I’m not saying FOX News is any less opinion and analysis than CNN; you are right that it is only “worse” because of political bias. However, people distinguish political bias and associate it with low quality more easily. I characterize low quality, but less partisan sources as “mostly noise” just as a guidepost to why it is low quality. Not that FOX News isn’t noise–it’s just specifically right-wing kind of noise. There is a lot of commentary one could write about what is in each “bubble,” which I did not include. It would just be too much for a visual chart. Thanks for the feedback, though. I’ll be tweaking and refining future versions, as media sources change all the time.

  5. Trying to generalize “local TV news” as mainstream with minimum bias is ludicrous. There are over 1000 stations, most having some sort of news. Lumping them together does not do justice to their diversity.
    Ideological identification is often fairly clear between different stations.
    The growing consolidation becomes a problem. 5 companies own about 35% of all stations. And Sinclair, if their most recent buying proposal gains approval, will have about 15% of the stations. As such, the political bent of Sinclair matters — and there is a clear, demonstrated political bent.
    Finally, suggesting that local TV news has “complex analysis sources” is wishful thinking. For some in major markets, that may be true. In Denver, where I am now, some stations do have solid investigative and analytical resources. Some are a step or two up from “rip and read” coverage. And in smaller markets, the news teams are generally smaller, younger, and more transient — all factors that undercut the notion of “complex analysis sources.”

    1. All generalizations are false.

  6. […] Lawyer Does the News A Denver lawyer is making news with her analysis of news — she updated her news quality chart that evaluates a publication’s partisan […]

  7. Thanks for the monumental amount of work this took. This is meta-journalism and is journalism in its own right. It required a lot of fact gathering and independent research, and quite a bit of analysis. As you alluded to in your first post, where you encouraged readers to try this exercise on their own, opinion is also baked in as everyone will find sources to be more or less liberally biased.

    Hopefully everyone’s result would line up vertically fairly well as we are entitled to our own opinions but not to our own facts. I didn’t even get halfway through the exercise of creating my own chart because I simply do not read enough news or news sources to be aware of

    I have one question for you, about the more fact-based sources: Do you find or suspect any inherent biases about the way news feeds choose which stories to report, and which facts to report in those stories? “Bias” is actually a terrible choice of words – I am talking about more than just political bias. I’m talking about some sort of journalistic bias. Perhaps the better question is – Do you find that in the fact based news sources there is a consensus on what is important to report versus not to report?

    Thanks so much for doing this!

    1. Thanks for your comment. It certainly is a lot of work to try to categorize media enough to generate a fairly simple chart, but there is much, much more to research in this field. I have some ideas on how to further evaluate media sources on a more objective (or, I should say, objective-as-possible) basis, so I’ll be sharing that here in the future as I put that together.

      One of the aspects I’d like to spend more time analyzing is exactly the topic you mention. What news sources choose to report on, as a subject of a story, or the very facts they choose to report within a particular story, can, I think, be evaluated in some meaningful way. I don’t think there is currently a consensus, and unfortunately, I think the focus for most outlets has been on stories that get the most “engagement.” News organizations need clicks and subscriptions to function as a business, and a hyper-focus on engagement has led to what I believe are bad results. People are unfortunately “very engaged” by stories that either play to their worst instincts (fear, hyper-partisanship, addiction to drama), or which are really petty (such as criticisms of FLOTUS’ shoes or something relatively innocuous POTUS says). If you look at The Washington Post or New York Times, you’ll see every day that they list their top five “most read” or “most shared” articles, and typically all five are analysis or opinion pieces. I think the focus on the “engagement” metric has led to an overabundance of coverage of American political drama at the expense of coverage of “more important” stories.

      I think there are a couple of places to start to determine some kind of baseline metric for what news is actually “important,” and I think a key metric would be “how much does this impact people’s lives.” If news consumers and journalists valued this metric more, there would be more stories about, say, a candidate’s health care policies than, I don’t know, her emailz. Another key one would be “does this story provide information the reader didn’t know before?” If news consumers and journalists valued this metric more, there would probably be more stories about things happening across the world, like the civil war in Yemen or the famines in Africa. As I dig into more news sources and this topic of what you refer to as “journalistic bias” (I agree, we can come up with a better term–maybe “importance bias”), I’ll plan on generating some kind of ranking in the future of sources that tend to cover more “important” stories vs. ones that cover less “important” stories.

  8. Great infographic, my only question relates to the assessment of Reason.com’s analytical rigor.

    Politics aside, Vice and Reason are (imo) the only two news sources that apply a collegiate level of analysis in their journalism.

    1. Thanks for the comment. Are you saying you think Reason.com is ranked too high or too low? I have it up at the high end of analytical writing.

  9. Vanessa, I appreciate your thorough analysis. I think you’re spot on in saying that the vast majority of what passes for “news” is actually just opinion, or perhaps analysis. I’d like to challenge your thinking on whether you’ve done enough to adjust for your own admittedly left-leaning bias. From my perspective, I think your chart would be much more accurate if you left the logos in place, and picked up the grid and moved it one inch to the right. That is to say, ABC, NBC, and CBS all lean left, the NY Times and CNN are far left, and FOX is not so far right as you make them out to be.

    To be objective, here are some telltale signs that the sources you call mainstream are really left-leaning: 1) They all thought Hillary would win in a landslide, and they were all very wrong; that shows evidence of living in the liberal bubble. 2) I tracked CNN.com from Sep. 1 through the election. They posted 6 to 8 anti-Trump stories and 1 to 2 pro-Hillary stories Every. Single. Day. The bias was palpable. 3) They all present Climate Change stories as credible; in truth, no major prediction of the environmentalist movement has ever come true. Global cooling, overpopulation, global warming, and now the vague catch-all “Climate Change” — none of these doomsday scenarios have ever come close to reality, proving that it’s nothing more than a leftist anti-capitalist movement, which your “mainstream” sources nonetheless give credence to. 4) They all incessantly repeat the Ferguson narrative of “unarmed black murdered by racist white cop” even though Obama’s own DOJ report thoroughly disproved that. Ditto for Eric Garner, Sylville Smith, Alton Sterling, Philando Castile, Freddie Gray, etc. — these were all proven categorically false leftist narratives. No actual injustice occurred in any of these situations. 5) They routinely ignore any story that doesn’t fit their liberal narratives, such as the gay pedophilia epidemic, or science showing that life begins at conception.

    I could write a book on this stuff, but suffice to say that ABCNNBCBS generally cover identical stories, and story selection is the greatest form of bias. Granted, CNN is more egregious in their slant, but the others all clearly lean left as well. And I hate to say it, but that actually goes for AP as well — they simply don’t report facts that oppose the liberal agenda. So please reconsider where to place the center of your grid.

    1. Thanks for the response, Dave. I have given quite a bit of thought to what constitutes the “center” of the chart. Without getting into too much detail, I think that there is a “journalistic center” and that it differs from the “American political center” on a regional basis. In other words, that same journalistic center seems very left-learning in conservative areas, but right-leaning in liberal ones. I disagree with the arguments you cite as objective proof that the outlets I have in the middle are very liberally biased, but each is a separate broad topic, and their separate treatment in “the media” is broader still. I think these are better addressed on specifics, like on an article-by-article basis, so I’d prefer to talk about how an individual article or news source treats them.

      Besides, if I moved the center an inch to the right, what would remain in the right-most column? Cheers!

      1. Hmmm. To me, the center should be “fact-based truth” and the left / right should be opinions or analysis of what to do about that truth. So with the Ferguson example, the fact-based-truth is that all evidence points to Brown’s guilt and Officer Wilson’s innocence (autopsies, forensics / ballistics, the fact that all six eyewitnesses were black yet all said Michael Brown kept attacking the cop right up until the last shot was fired). Left-leaning opinion would question whether the white cop targeted Brown because of skin color; right-leaning opinion would be about how rampant fatherlessness among black youth (73%, including Brown) leads to disrespect for authority.

        So maybe where we think differently is that you are placing fact vs. opinion solely on the vertical axis, and the center of your horizontal axis is not truth, but simply the midpoint of the political spectrum? (Please correct me if I’m misreading you — I’m not trying to mischaracterize anything you’re saying!) On the other hand, I would put fact-based truth in the middle, and would make no attempt to have relatively equal numbers of opinion sources to either side. Of course, I believe that truth is both fixed and knowable, and I realize not everyone believes that.

        btw, given your deep interest in the topic of news reporting, I encourage you to read the DOJ report on the Michael Brown incident – it’s fascinating to compare the facts to the news reports.

        https://www.justice.gov/sites/default/files/opa/press-releases/attachments/2015/03/04/doj_report_on_shooting_of_michael_brown_1.pdf

  10. Do you have a blank chart of this second edition? I couldn’t find it.

    You really hit on something here, Vanessa. Thanks for all the work you put into this. I’m about to use this material in a community workshop on spotting fake news and stopping fake news. Your ideas are central. You give me hope.

    1. Thanks so much. I’ll post a blank version momentarily and e-mail it to you. Also, I just made a new reference tool for identifying conspiracy theories that you might find useful–it will be on my main page shortly.

  11. Do you have a link to this second chart? Using the image you have posted here is very low resolution. I’d love to have a link to a pdf. Thanks!

    1. Hi–I just added a link to a hi-res PDF at the bottom of the post. Hope that helps!

  12. Hi Vanessa,
    I’m a librarian who uses your chart for workshops on Fake News. I really like the new version and your logic behind it. Highlighting the difference between fact-reporting and analysis is important. Like Susan Witter, can you also send me a copy of the blank version?
    Thank you again.

  13. I’m afraid that any conservative who sees Al Jazeera and NPR as main-stream and factual will immediately dismiss your entire chart as “fake news” and perhaps even a left-wing communist plot 😉

    That being said, kudos on doing a yeoman’s job. In my years of reading I’d say you’re pretty much on the mark.

    1. You’re right…hopefully something will help turn the tide of misinformation and hyper-partisanship soon. Thanks!

  14. I would be interested in how they analyze 24-hour news programming versus the network news. In considering CBS, ABC, NBC; are they only considering their official Nightly News or are they also considering the Sunday morning shows as well as the daytime talk shows that have a strong political component.

    As for the Big 3, 24-hour news networks; does the analysis include only the news specific programs or the entire slate of programming.

    Also I’d be interested to know where they consider OAN in their charting. I didn’t see it.

    Finally, political bias is not only found in the opinion and Analysis portions of news programs. It can also be found in which ‘fact-based’ stories a news station/program chooses to report and which they choose to ignore.

    1. I’ll be coming out with more specific analysis of 24-hr news and network news shows in the near future. With all sources on the chart, the placement reflects an average across the various stories and news programs, which of course widely vary (e.g., news hours vs. opinion hours). I’ll have more detailed charts on individual networks, cable news, and their respective shows soon.

      I’ll include OAN on a future version as well. Agree with you on the “selection bias.”

  15. […] I discussed in my post entitled “The Chart, Second Edition: What Makes a News Source Good?” the most accurate and helpful way to analyze a news source is to analyze its individual stories, […]

  16. […] This revision came with an additional explanation: “The Chart, Second Edition: What Makes a News Source ‘Good?'” […]

  17. Hi Vanessa!
    I commented earlier but I don’t see my comment here. So I’ll ask again – what do you think about putting AJ smack in the middle of a bipartisan, mainstream, fact checking zone, when they are owned and managed by the Sheikhs of Qatar, a non democratic country that supports terror organizations? Thank you

    1. Hi Anat. That is a valid concern. My rankings are mostly based on analysis of individual stories, rather than on ownership. Ownership is a valid concern (see others’ comments about corporate ownership of various media outlets), but it’s impact on actual stories reported is not always direct–stories can be true, reliable, and corroborated by other outlets even if you find the ownership questionable. So, I’ll defend my initial placement because I considered Al-Jazeera America’s coverage of US news stories to be reputable. However, I hear yours and other’s concerns about Al-Jazeerza, and given that their news organization has shifted forms quite a bit in the last few years (giving them less of a reputational foothold compared to other organizations), I will likely leave them off the chart in the future.

Leave a Reply

Your email address will not be published. Required fields are marked *