Intrusion of software robots into journalism (2017)

Clippy attends a press conference
All your pays are belong to us

Good journalism is exciting, inspiring, and informative. But there’s also a lot of mundane reporting that simply isn’t very exciting to produce. One might be tempted to automate these tasks. Are articles written by robots robust enough to withstand the watchful eyes of journalists and the general public? Jung et al. think that may be the case!

Why it matters

Let’s start with a simple question. Are algorithms able to generate trivial articlesi.e. articles that can be generated automatically from structured data using pre-determined sentences that are good enough?

I’ll just spoil this one right away, because it’s not part of the main study: the quality of articles written by algorithms can be indistinguishable from that of articles that were written by journalists, if one leaves out the bylinei.e. the part that shows the author’s name.

But what happens when we do add that byline?

Trust in mainstream media – and consequentially journalists who work for those media – has slowly decreased among the general public in the recent years. It’s therefore probable that the general public will be more favourable towards articles that are written by algorithms.

Journalists on the other hand, might be worried about the possibility that robots will take over their jobs and therefore prefer articles that have been written by a human peer.

These are assumptions that can be tested.

How the study was conducted

The authors performed two studies, one for each target group:

In each study participants were asked to read an article that was written by either a robot or journalist, and had a byline of either a robot or journalist, regardless of who wrote the article. Afterwards, participants were asked to answer a series of questions about the article on a 5-point Likert scale; was it well-written, clear, professional, and credible?

What discoveries were made

The table below shows the average ratings given by respondents from the general public (higher is better):

Written by robot Written by human Avg.
Robot in byline 3.39 3.41 3.40
Human in byline 3.21 3.31 3.26
Average 3.30 3.36 3.33

Respondents think that the quality of the hand-written article is slightly higher than that of the algorithmically generated article. However, we can also see that articles that have a byline that claims that it’s written by a robot are perceived to be of higher quality – regardless of who actually wrote it!

How did the journalists rate the articles then?

Written by robot Written by human Avg.
Robot in byline 3.23 3.33 3.28
Human in byline 2.85 3.13 2.99
Average 3.04 3.23 3.14

The same pattern is visible here as well: putting a robot in the byline appears to have a positive effect on the perception of the quality of an articleIn both cases the best-rated article is hand-written, but attributed to a robot. One might be tempted to exploit this – don’t..

It’s not clear why this happens. Maybe journalists don’t feel that much (or any) hostility towards algorithms; maybe Koreans are more receptive of technological innovations; or maybe people simply have lower expectations of algorithmically generated articles.

Well, that’s something for another study!

The important bits

  1. Articles that are claimed to have been written by an algorithm are perceived to be of higher quality, regardless of the actual author