How controversial are today's newspapers?

How to measure controversy?

Controversy score with sentiment analysis

For each selected topic we want to calculate a sentiment score to assess how controversial it is. A sentiment analysis can be done on all quotes linked to the sub-topics defined in the previous step.

The sentiment analysis allows us to define if a quote is addressed in a positive or negative manner. We ended up with a repartition of positive and negative sentiments for a given word.

From this distribution, we defined a controversy score as follows: a subject with an equal repartition of positive/negative sentiment should be interpreted as strongly controversial.

The following formula was implemented to define the score: \(2 \cdot \frac{min(\#pos, \#neg)}{\#pos + \#neg}\).

As we can see from the figure below the theme “Politic” is more controversial (controversy_score = 0.98) whereas “Team” does not reflect a strong controversy (controversy_score = 0.54).

Signature score

To assess the correctness of our controversy score, we have compared it to methods available from literature. In particular we compared it to a method based on the variance of sentiments from the paper “Quantifying controversy in social media”.

We also considered a method that compares sentiment distributions within quotations. Additionally, this “sentiment signature” based method aims at providing a relative controversy score to attempt to validate our model.

To validate or discard our model, we applied a principle of majority vote. Our expectation was that the two methods that “agreed” the most often should be more accurate and closer to the ground truth. In the figure above, we have the similarity scores of the three different methods we considered. Essentially, it can be seen from the cells colors how similar one method is to another for a given topic. Doing this pairwise comparison, we got that the mean similarity score (across all topics) is highest (i.e. 0.52) between our method and the sentiment signature based one. One can remark that this analysis provides us with arguments against the variance-based method.

As a conclusion, we will rely on our method for the aforementioned reasons and also for the fact that it is simpler and provides an absolute measure of controversy.

Controversy score for each topic

Finally, after checking our model, it was time to evaluate the controversy score of the selected topics.

The controversy scores was calculated by summing all the positives and negatives sentiments over all sub-topics. Here, it was done over a 5-year period, from 2015 to 2020, see graphic below.

First, it can be observed that our scores are consistent over the years and specific to each topic (they are not all packed together). In addition, some topics, such as “Politics”, “Investing” or even “Countries”, seem strongly controversial as they are usually perceived within popular opinion.

In opposition to the terms like “Sports”, “Datetime” and “Games” do not seem to represent controversial subjects, as sport journalists are supposed to stay neutral in their report.

How to select newspapers?

Newspapers in the dataset

Newspapers’ sales data

The sales data for the aforementioned newspapers were not available. So instead, we checked for which newspapers the sales data could be obtained and then investigated those newspapers presence in our dataset.

Several English newspapers, present in our dataset, had accessible sales data. Therefore, we focused our analysis on those journals. You can see below the evolution of their sales data over the years.

Analysing controversy through quotations

Finding the quotes within selected newspapers

Subsequently, we extracted all quotes cited in those newspapers from the subset of 100’000 quotations for each year.

Secondly, to link a topic to a quote we looked if the quote contained one of the sub-topics. This resulted in a matrix where we could find the number of occurrence of each topic for the different newspapers.

The distributions of the quotes over the topics were normalized by the total number of quotes by newspaper. From the heatmap below, one can observe that all newspapers are focused on mainstream topics such as “Sports” or “Teamwork”. It can also be noted that the topic “Investing” has a higher chance to appear in one article of the newspaper Financial Times than in another one, this makes sense as the newspaper is specialized in this field.

Measuring newspapers controversy

Finally, by multiplying the topic-newspapers matrix with the controversy score vector for each topic, we got the controversy score of each journal.

The process was applied for each year of the period individually, this resulted in the controversy score evolution of each newspaper over time. The resulting graphic is presented below.

Looking at the sales table one can observe that they are decreasing over the years, however the controversy score seems to react in an opposite manner. Is it a reaction of the newspaper companies, which are willing to keep their numbers as high as possible and are willing to modify their usual topics to do so? In 2020, all newspapers seem to have talked less on controversial subjects, one can explain this behaviour by the pandemic which was taking all the spotlight.

One newspaper clearly stands out from the others, FT which stands for Financial Times. It is the most controversial newspaper and its controversy score is continuously increasing until it suddenly dropped in 2020. We, first, tried to explain this behaviour with the fact that the pandemic completely stopped the global economy in 2020. It generated a consensus regarding the financial market. However, as good ADAventurers that we are, we took a closer look at the data and quickly realized that for this year an abnormally small amount of quotations were selected in the dataset. This value was neglected for subsequent calculations.

On the graphics below you can see how newspapers sellings are related to their controversy score. On the left the results for all newspapers and on the right you can scroll through the graphics to have a better view of each newspaper separately.