How Facebook’s algorithm led a test user in India to fake news, gore

In February 2019, Facebook Inc. set up a test account in India to determine how its own algorithms affect what people see in one of its fastest growing and most important overseas markets. The results stunned the company’s own staff.

Within three weeks, the new user’s feed turned into a maelstrom of fake news and incendiary images. There were graphic photos of beheadings, doctored images of India air strikes against Pakistan and jingoistic scenes of violence. One group for “things that make you laugh” included fake news of 300 terrorists who died in a bombing in Pakistan.

“I’ve seen more images of dead people in the past 3 weeks than I’ve seen in my entire life total,” one staffer wrote, according to a 46-page research note that’s among the trove of documents released by Facebook whistleblower Frances Haugen.

The test proved telling because it was designed to focus exclusively on Facebook’s role in recommending content. The trial account used the profile of a 21-year-old woman living in the western India city of Jaipur and hailing from Hyderabad. The user only followed pages or groups recommended by Facebook or encountered through those recommendations. The experience was termed an “integrity nightmare,” by the author of the research note.

ALSO READ: Facebook’s independent oversight board seeks more transparency

While Haugen’s disclosures have painted a damning picture of Facebook’s role in spreading harmful content in the U.S., the India experiment suggests that the company’s influence globally could be even worse. Most of the money Facebook spends on content moderation is focused on English-language media in countries like the U.S.

But the company’s growth largely comes from countries like India, Indonesia and Brazil, where it has struggled to hire people with the language skills to impose even basic oversight. The challenge is particularly acute in India, a country of 1.3 billion people with 22 official languages. Facebook has tended to outsource oversight for content on its platform to contractors from companies like Accenture.

“We’ve invested significantly in technology to find hate speech in various languages, including Hindi and Bengali,” a Facebook spokeswoman said. “As a result, we’ve reduced the amount of hate speech that people see by half this year. Today, it’s down to 0.05 percent. Hate speech against marginalized groups, including Muslims, is on the rise globally. So we are improving enforcement and are committed to updating our policies as hate speech evolves online.”

The new user test account was created on Feb. 4, 2019 during a research team’s trip to India, according to the report. Facebook is a “pretty empty place” without friends, the researchers wrote, with only the company’s Watch and Live tabs suggesting things to look at.

“The quality of this content is… not ideal,” the report said. When the video service Watch doesn’t know what a user wants, “it seems to recommend a bunch of softcore porn,” followed by a frowning emoticon.

The experiment began to turn dark on Feb. 11, as the test user started to explore content recommended by Facebook, including posts that were popular across the social network. She began with benign sites, including the official page of Prime Minister Narendra Modi’s ruling Bharatiya Janata Party and BBC News India.

Then on Feb. 14, a terror attack in Pulwama in the politically sensitive Kashmir state killed 40 Indian security personnel and injured dozens more. The Indian government attributed the strike to a Pakistan terrorist group. Soon the tester’s feed turned into a barrage of anti-Pakistan hate speech, including images of a beheading and a graphic showing preparations to incinerate a group of Pakistanis.

There were also nationalist messages, exaggerated claims about India’s air strikes in Pakistan, fake photos of bomb explosions and a doctored photo that purported to show a newly-married army man killed in the attack who’d been preparing to return to his family.

ALSO READ: Facebook to rebrand itself as ‘metaverse company’, get new name: Report

Many of the hate-filled posts were in Hindi, the country’s national language, escaping the regular content moderation controls at the social network. In India, people use a dozen or more regional variations of Hindi alone. Many people use a blend of English and Indian languages, making it almost impossible for an algorithm to sift through the colloquial jumble. A human content moderator would need to speak several languages to sieve out toxic content.

“After 12 days, 12 planes attacked Pakistan,” one post exulted. Another, again in Hindi, claimed as “Hot News” the death of 300 terrorists in a bomb explosion in Pakistan. The name of the group sharing the news was “Laughing and things that make you laugh.” Some posts containing fake photos of a napalm bomb claimed to be India’s air attack on Pakistan reveled, “300 dogs died. Now say long live India, death to Pakistan.”

The report–entitled “An Indian test user’s descent into a sea of polarizing, nationalist messages”–makes clear how little control Facebook has in one of its most important markets. The Menlo Park, California-based technology giant has anointed India as a key growth market, and used it as a test bed for new products. Last year, Facebook spent nearly $6 billion on a partnership with Mukesh Ambani, the richest man in Asia, who leads the Reliance conglomerate.

“This exploratory effort of one hypothetical test account inspired deeper, more rigorous analysis of our recommendation systems, and contributed to product changes to improve them,” the Facebook spokeswoman said. “Our work on curbing hate speech continues and we have further strengthened our hate classifiers, to include four Indian languages.”

But the company has also repeatedly tangled with the Indian government over its practices there. New regulations require that Facebook and other social media companies identify individuals responsible for their online content — making them accountable to the government. Facebook and Twitter Inc. have fought back against the rules. On Facebook’s WhatsApp platform, viral fake messages circulated about child kidnapping gangs, leading to dozens of lynchings across the country beginning in the summer of 2017, further enraging users, the courts and the government.

The Facebook report ends by acknowledging its own recommendations led the test user account to become “filled with polarizing and graphic content, hate speech and misinformation.” It sounded a hopeful note that the experience “can serve as a starting point for conversations around understanding and mitigating integrity harms” from its recommendations in markets beyond the U.S.

“Could we as a company have an extra responsibility for preventing integrity harms that result from recommended content?,” the tester asked.