Posted on

The Chart, Second Edition: What Makes A News Source “Good?”

In my original news chart, I wrestled with the questions of what made news sources “good” and came up with some categories that generally resonated with people. I ranked sources on a vertical axis with those at the top ranked as “high quality” and those at the bottom as “low quality.” I characterized the sources, from top to bottom, in this order: Complex, Analytical, Meets High Standards, Basic, and Sensational/ Clickbait. This mostly works, because it results in sources regarded as high-brow or classy (e.g., The Atlantic, The Economist) being ranked high on the axis, and trashy sources (e.g., Addicting Info, Conservative Tribune) being ranked low, and most sophisticated news consumers agree with that. However, the vertical placements ended up causing me and others some consternation, because some of the placements relative to other outlets didn’t make sense. The most common questions I got were along these lines:

“Does FOX News really “meet high standards,” on par with something like the New York Times?” (I think no.)

“Is USA today really that bad?” (I think no.)

“Is Slate really “better” or “higher quality” than, say, AP or Reuters just because it is analytical?” (I think no.)

“Is CNN really that bad?” (I think yes.)

These questions and my instinctive responses to them made me want to reevaluate what makes news sources high or low quality.

I believe that answer to that question lies in what makes an individual article (or show/story/broadcast) high or low quality. Article quality can vary greatly even within the same news source. One should be able to rank an individual article on the chart in the same way one ranks a whole news source. So, what makes an article/story high or low quality? It’s hard to completely eliminate one’s own bias on that issue, but one way to try to do it consistently is to categorize and rate the actual sentences and words that make it up its headline and the article itself. In order to try to rank any article on the chart in a consistent, objective-as-possible manner, I started doing sentence-by-sentence analyses of different types of articles.

In analyzing what kind of sentences make up articles, it became apparent that most sentences fall into (or in-between) the categories of 1) fact, 2) analysis, or 3) opinion. Based on the percentages of these kinds of sentences in an article, articles themselves can be classified in categories of fact, analysis, and opinion as well. Helpfully, some print newspapers actually label articles as “analysis” or “opinion.” However, most news sources, especially on TV or the internet, do not. I set about analyzing stories that were not pre-labeled as “analysis” or “opinion” on a sentence-by-sentence basis. I discovered that my overall impression of the quality of an article was largely a function of the proportion of fact sentences to analysis sentences to opinion sentences. As a result, I classified stories into “fact-reporting,” “analysis,” and “opinion” stories. Ones with high proportions of “fact” sentences (e.g., 90% + fact statements) were what I refer to here as traditional “fact-reporting” news pieces. These are the kinds of stories that have historically been the basis of late 20th century-to-early-21st century journalism, and what people used to refer to exclusively as “news.” They are the “who,” “what,” “when,” and “where” pieces (not necessarily “why”). I classified ones with high proportions of “analysis” sentences (e.g., 30%-50% analytical statements) as “analysis” stories, which are the types of stories commonly found in publications like The Economist or websites such as Vox. I classified stories with high proportions of opinion sentences (e.g., 30%-50% opinion statements) as “opinion,” which are typically the types of stories found on websites such as Breitbart or Occupy Democrats.

(If you’ve made it this far, bless your heart for caring so much about the news you read.)

In the past, national evening news programs, local evening news programs, and the front pages of print newspapers were dominated by fact-reporting stories. Now, however, many sources people consider to be “news sources” are actually dominated by analysis and opinion pieces. This chart ranks media outlets that people consider to be, at some level “news sources,” even though many of them are comprised entirely of analysis and opinion pieces.

In my previous version of the chart, I had regarded analysis pieces as “higher quality” than the fact-reporting pieces because they took the facts and applied them to form well-supported conclusions. I like analytical writing, which is essentially critical thinking. However, analysis has a lot in common with opinion, and writing that is intended to be analytical often strays into opinion territory. (Note—I’m defining “analysis” as conclusions well supported by facts and “opinion” as conclusions poorly supported or unsupported by facts). Fact-reporting articles—true “scoops”—typically have the intent of just reporting the facts and typically have a very high percentage (e.g., 90%+) of fact-statement sentences, whereas both analysis and opinion articles have the intent of persuading an audience and often have a comparatively high percentage of analysis and opinion statement sentences (30%-50%). So, although I initially had the quality axis of “news” laid out top to bottom as:

That ranking is more reflective of the quality of writing rather than the quality of news sources. Good analysis is often written persuasively and well, fact-reporting is often written directly but well, and opinion writing is often (but not always) written poorly or is most easily discredited.  I submit that given the confusion caused by the overwhelming number of organizations proclaiming to be (or which are commonly confused with) “news sources,” it is more important to rank the quality of news sources than the quality of writing. I further submit, for reasons outlined below, that the percentage of fact reporting articles and stories should be used as the most determinative factor by which a news source is ranked in quality on this chart.

Therefore, I believe a more relevant ranking of the quality of news sources would be:

 

 

I assert that one of the biggest problems with our current news media landscape is that there is too much analysis and opinion available in relation to factual reporting.  New technologies have given more people more platforms to contribute analysis and opinion pieces, so many “news sources” have popped up to compete for readers’ attention. Unfortunately news consumers often do not recognize the difference between actual fact-reporting news and the analysis and opinion writing about that news.  This increase in “news sources” has not corresponded with an increase in actual journalists or news reporting, though. Many local and national print news organizations have reduced their numbers of journalists, while many of the biggest ones have merely maintained similar numbers of journalists over the past 10 years or so.
Furthermore, primarily analytical news sources also have several downsides. One downside is that they can alienate news consumers by making what people consider “news sources” so complex or partisan that it is tiring to consume any “news.”  For example, CNN, MSNBC and FOX News, which are primarily analysis and opinion-driven, can make news consumers too weary to pay attention to fact-based reporting from, say, AP or Reuters. Another problem with analysis and opinion-driven news sources is that it can be difficult for casual readers to differentiate between good analysis and pure opinion.
There are several good reasons why we should value fact-reporting sentences, fact-reporting articles, and fact-reporting news sources high on the quality scale of news, at least on this chart. For one, reported facts take a lot of work to obtain. They require journalists on the ground investigating and interviewing. Once a story is reported, dozens, hundreds, or thousands of other writers can chime in with their analysis or opinions of it. This is not to say analysis and opinion writing isn’t important. The critical thinking presented in analytical writing—especially good, complex analysis—is essential to public discourse. Our society’s best ideas are advanced by analytical articles. This piece you are reading now is analytical. But analysis in the news wouldn’t even exist without the underlying factual reporting.

For example, AP and Reuters have maintained around 2,000-2,500 journalists each over that time, while the New York Times and Washington Post have fluctuated in the 500-1000 range over the same period. The value of these organizations with large staffs of journalists, editors, and other newsroom employees is hard to overstate; not only do they provide a majority of the fact-reporting stories everyone else relies on, but they have the capacity to provide high-quality editorial review that stands up to industry scrutiny. In contrast, even some of the most popular analysis and opinion sites can be run with just a few dozen writers and staff; the number of these “news” websites, news aggregator websites, blogs, and podcasts has seemingly grown exponentially.

I believe improvements to our media landscape can be made if two things happen: 1) if news consumers start valuing factual reporting much more and analysis/opinion articles much less and 2) if news consumers become accustomed to differentiating articles in those categories.. Regarding point #2, I think it would be helpful if we narrowed the definition of “news” to only refer to fact reporting, and referred to everything else as “analysis” or “opinion.” It would be helpful if people could recognize the relative contributions of fact-reporting news organizations versus analysis and opinion sources. If people recognize just how much of what they read and watch is intended to persuade them, they may become more conscious and thoughtful about how much they allow themselves to be persuaded. One can hope.

To contribute to those goals, I’ve reordered the chart to value fact-reporting articles as the highest quality and everything else lower, even though there is some really excellent analysis out there. As a baseline, news consumers should understand when something is news (fact-reporting) and when it is not. On the new chart, the sources with the best analysis, but little reporting are at the top, but right under the sources that are comprised of high percentages of reporting articles. The most opinion-driven sources are at the bottom. There’s room for other things at the bottom below pure opinion, which can include sources that are sensationalist, clickbait, frequently factually incorrect, or which otherwise don’t meet recognized journalism standards.

On this version, I’ve included a number of different sources, mostly in the analysis and opinion categories, and left the most popular mainstream sources from the original chart, but have reordered some of them. Now, the rankings are more consistent with my initial answers to the example questions at the beginning of this post. Fox News is now ranked far lower than the New York Times for two main reasons; one, Fox News is dominated by opinion and analysis, and two, it has gotten precipitously worse in other measures (sensational chyrons, loss of experienced journalists, hyperbolic analysis by contributors, etc.) within the last six months. USA today, despite its basic nature, has been elevated because of its high percentage or fact-reporting stories. Slate, though it provides thoughtful, well-written analysis, is ranked lower than AP and Reuters, which better reflects their relative contributions to the news ecosystem. CNN still sucks, but it is clearer why now; CNN has the resources to provide twenty-four hours of news—it could provide Americans with a detailed global-to-local synopsis of the world—but instead it chooses to spend 5% of its time fact reporting a handful of stories, comprising mostly American political drama and maybe one violent leading world news story, and 95% on analysis and opinion ranging from the competent to the inane.

My analysis of news sources in the manner I’ve described herein has revealed that individual stories can and should be ranked on the chart in the same manner, and that individual stories can be placed in different places than the news sources in which they are published. I’ll be putting out individual story rankings and reasoning for those rankings from time to time for those that are interested. I’ll also take requests for rankings of sources and individual stories in the comments and on twitter. Thanks for reading and thinking.

 

 

 

14 thoughts on “The Chart, Second Edition: What Makes A News Source “Good?”

  1. I am currently reading the National Review for the conservative take on major news stories. I would be interested to know where that would fall on the chart.

    Great chart, and even better documentation. Thank you for this work!

  2. Will you rank TeenVogue and GQ? I’m seeing (unexpectedly!) interesting analysis/opinion articles from them online?

    1. I will in future editions. They do have good writers and articles, such as those by Lauren Duca as of late. They are a bit harder to categorize, though, as are other similar magazines such as Rolling Stone. This is because their focus is industry specific (i.e., teen fashion, men’s fashion, and music), and a low percentage of their articles are on news topics outside of those industries. I’m planning on putting out rankings of individual articles in the future for these instances.

  3. Disappointed that you did not include ‘The Guardian’

    1. It’s on the original version of the chart. Look back a couple of posts. 🙂

  4. Im having trouble with your “Mostly Noise” label. Is it referring to just that centre, bottom bubble? to all sources in blue bubbles? why is CNN mostly noise but FOX isnt? I feel like FOX is only “worse” than CNN because it has more political bias, and I see that in the vertical and horizontal placement of those two sources, but the “Mostly Noise” label is confusing.

    1. I’m not saying FOX News is any less opinion and analysis than CNN; you are right that it is only “worse” because of political bias. However, people distinguish political bias and associate it with low quality more easily. I characterize low quality, but less partisan sources as “mostly noise” just as a guidepost to why it is low quality. Not that FOX News isn’t noise–it’s just specifically right-wing kind of noise. There is a lot of commentary one could write about what is in each “bubble,” which I did not include. It would just be too much for a visual chart. Thanks for the feedback, though. I’ll be tweaking and refining future versions, as media sources change all the time.

  5. Trying to generalize “local TV news” as mainstream with minimum bias is ludicrous. There are over 1000 stations, most having some sort of news. Lumping them together does not do justice to their diversity.
    Ideological identification is often fairly clear between different stations.
    The growing consolidation becomes a problem. 5 companies own about 35% of all stations. And Sinclair, if their most recent buying proposal gains approval, will have about 15% of the stations. As such, the political bent of Sinclair matters — and there is a clear, demonstrated political bent.
    Finally, suggesting that local TV news has “complex analysis sources” is wishful thinking. For some in major markets, that may be true. In Denver, where I am now, some stations do have solid investigative and analytical resources. Some are a step or two up from “rip and read” coverage. And in smaller markets, the news teams are generally smaller, younger, and more transient — all factors that undercut the notion of “complex analysis sources.”

    1. All generalizations are false.

  6. […] Lawyer Does the News A Denver lawyer is making news with her analysis of news — she updated her news quality chart that evaluates a publication’s partisan […]

  7. Thanks for the monumental amount of work this took. This is meta-journalism and is journalism in its own right. It required a lot of fact gathering and independent research, and quite a bit of analysis. As you alluded to in your first post, where you encouraged readers to try this exercise on their own, opinion is also baked in as everyone will find sources to be more or less liberally biased.

    Hopefully everyone’s result would line up vertically fairly well as we are entitled to our own opinions but not to our own facts. I didn’t even get halfway through the exercise of creating my own chart because I simply do not read enough news or news sources to be aware of

    I have one question for you, about the more fact-based sources: Do you find or suspect any inherent biases about the way news feeds choose which stories to report, and which facts to report in those stories? “Bias” is actually a terrible choice of words – I am talking about more than just political bias. I’m talking about some sort of journalistic bias. Perhaps the better question is – Do you find that in the fact based news sources there is a consensus on what is important to report versus not to report?

    Thanks so much for doing this!

    1. Thanks for your comment. It certainly is a lot of work to try to categorize media enough to generate a fairly simple chart, but there is much, much more to research in this field. I have some ideas on how to further evaluate media sources on a more objective (or, I should say, objective-as-possible) basis, so I’ll be sharing that here in the future as I put that together.

      One of the aspects I’d like to spend more time analyzing is exactly the topic you mention. What news sources choose to report on, as a subject of a story, or the very facts they choose to report within a particular story, can, I think, be evaluated in some meaningful way. I don’t think there is currently a consensus, and unfortunately, I think the focus for most outlets has been on stories that get the most “engagement.” News organizations need clicks and subscriptions to function as a business, and a hyper-focus on engagement has led to what I believe are bad results. People are unfortunately “very engaged” by stories that either play to their worst instincts (fear, hyper-partisanship, addiction to drama), or which are really petty (such as criticisms of FLOTUS’ shoes or something relatively innocuous POTUS says). If you look at The Washington Post or New York Times, you’ll see every day that they list their top five “most read” or “most shared” articles, and typically all five are analysis or opinion pieces. I think the focus on the “engagement” metric has led to an overabundance of coverage of American political drama at the expense of coverage of “more important” stories.

      I think there are a couple of places to start to determine some kind of baseline metric for what news is actually “important,” and I think a key metric would be “how much does this impact people’s lives.” If news consumers and journalists valued this metric more, there would be more stories about, say, a candidate’s health care policies than, I don’t know, her emailz. Another key one would be “does this story provide information the reader didn’t know before?” If news consumers and journalists valued this metric more, there would probably be more stories about things happening across the world, like the civil war in Yemen or the famines in Africa. As I dig into more news sources and this topic of what you refer to as “journalistic bias” (I agree, we can come up with a better term–maybe “importance bias”), I’ll plan on generating some kind of ranking in the future of sources that tend to cover more “important” stories vs. ones that cover less “important” stories.

Leave a Reply

Your email address will not be published. Required fields are marked *