Could Personal Data Be Put To Positive Use?

By James O’Malley
01 March 2019
China

China's Social Credit Scheme appears the stuff of nightmares, but a gamified rating system isn't anything new

Crossing the road in China can be challenging. The streets of the largest cities, such as Beijing and Shanghai, are an incoherent mass of cars, trucks, scooters and bicycles. The horn and the hazard lights are a standard part of the Chinese motorist’s toolkit and, for every type of road user, red lights are taken not as instructions, but as gentle suggestions to be easily ignored.

This is except for one city. Shenzhen, China’s southern tech hub might just have figured out the trick to making roads slightly less chaotic, and it’s all thanks to the liberal application of CCTV and some vigorous public shaming. Teaming up with a company called Intellifusion to observe jaywalkers in the city, if you cross the road at the wrong time, or in the wrong place, you risk being identified by cameras hooked up to facial recognition software that, in turn, is connected to a police database. Once identified, offenders could soon find their mugshots appearing not only on a government blacklist, but on electronic billboards around the city, publically mocked with the razzmatazz usually reserved for a brand new Netflix series. If you’re unlucky enough to get caught by the cameras, that might not be your only punishment. The same technology could also be used to automatically send the rule-breaker a text or a message on a Chinese messaging app akin to WhatsApp, giving them a telling off – as well as issuing a fine. No human intervention required.

What’s perhaps most surprising is that the system, seemingly pulled direct from the brains of messrs Orwell and Brooker, appears to be getting results. According to the South China Morning Post, in its first year of operation around 13,930 people were recorded breaking the rules at just one junction in the city’s Futian District, and authorities believe that the system is already significantly reducing the number of citizens ready to step out in the wrong place all over the city.

In 2014, China’s state council fired the starting gun on such an enterprise with a document snappily entitled “Planning Outline for the Construction of a Social Credit System”. The idea was that the state would aggregate public and private records of behaviour to create a single number of good conduct. An algorithmic indication that would have real life consequences. According to the document, the main aim of the idea was to monitor and penalise unscrupulous businesses and corrupt local politicians. So far so good, but here’s where the waters become a little muddied: the same principles were also set to apply to the normal, everyday citizen in China, the idea being that by attaching a scoring system to meaningful outcomes – say, access to certain privileges in society or the denial of services or facilities – people would have a direct incentive to make sure that they behaved in a way that was amenable to what city planners would see as the greater good. Well-behaved citizens could be directly rewarded, and poorly behaved citizens could be directly punished.

“People always cite the ‘Nosedive’ episode of the television series Black Mirror as an example of this type of gamification,” explains Gianluca Sgueo, a New York University professor and policy analyst for the European Parliament. “This [the episode] is a world where people rate each personal interaction on a scale, and an individual’s average rating has significant influence on status. A positive rating is accompanied with a high-spirited warble, whereas a negative score is accented with a gloomy tune. Life is a constant struggle for higher scores. Scaling up means cooler friends, healthier food, superior services – in summary: a better life.”

China’s district of Rongchang is perhaps the best example of this in real-time. Here, citizens are reportedly given 1000 points to start with, and gain or lose them depending on how virtuous or disappointingly they behave. Driving offences can reportedly hit your social credit score, whereas donating to charity or volunteering can boost it. And in a nice twist on the Shenzhen formula, in Rongchang, the electronic billboards are used to celebrate the top-performing citizens, rather than punish the worst offenders.

But the system isn’t simply geared to punishing those who break the law. Purchasing patterns will also be used to indicate your lifestyle, attitude and, ultimately, your perceived personality. Buy too many video games and – regardless of your record – you can figure on a laziness index. Opt for a particular brand of tech and you’ve just indicated your social class, stock up on nappies and baby food and well, you’re clearly a parent so, responsibility bonus achieved – power-up! Ultimately, the same algorithms that help curate ‘for you’ categories on shopping websites will now be used to paint a picture that could determine the rest of your life.

There are currently around 40 regional pilots of social credit in action. In Jinan, dog owners have a social credit score for their pet – and if too many complaints are made, say, for walking it without a leash or for too much barking, you risk having your dog taken away. In Beijing, according to reports on China’s Twitter-like social network Weibo, one top-performing student was denied entry to a Beijing University because his father was on the Supreme People’s Court blacklist of debtors, which is linked to the social credit system.

“It provides incentives for maximising citizen value through politically and commercially aligned social behaviour”, write Nicholas Loubere and Stefan Brehm, on social credit, in the Made in China journal. They sum up the benefit – and ominous sides – of social credit succinctly, saying that, “Using algorithms to render citizens and organisations compliant with the visions and rationales of the ruling regime reduces the State’s information and monitoring costs dramatically.”

In other words, if your life can be affected by your reputation score falling below a certain mark, suddenly you have a strong incentive to do whatever it takes to keep your numbers high – including falling into line with government scorekeeping. Suddenly you’ve achieved social order without so much an arrest made. At least that’s the current theory.

It turns out that the arbitrary nature of applying numbers to complex, human behaviour across a wide range of fields might be harder than you think. According to a paper written by Professor Xin Dai from Beijing’s Ocean University, one trial in the province of Qingzhen demonstrated how hard social credit is to enumerate: Hiring a child worker would cost an employer five points – but selling coal products that don’t meet regulatory standards would cost them 10. Good luck trying to figure out why one is exactly twice as bad as the other. This is why arbitrating behaviour is usually left to a court, where a judge can make a qualitative decision on the merits of the individual case, and impose fines or other punishments with more discretion.

This problem isn’t even hypothetical. One of the earliest social credit trials in the city of Suining crashed because of the arbitrary scoring system.
At the moment, there isn’t a single national social credit system up and running in China. Instead, authorities and government agencies have been building their own local systems, such as the one described in Shenzhen. The different local systems are all varied in scope, punishments and rewards. Not all of them even have a scoring system – some are instead much cruder blacklists that an individual is either listed on, or not. But together they demonstrate some of the possibilities of what a social credit system could achieve… not to mention what a not too distant future in China could look like under a constant cloud of data armed with real-world consequences.

Not that Big Brother hasn’t been watching the good folk of the PRC for quite some time. Dang’an (your personal record) and the hukou (a household record) have been part of a life weighed and measured by the ruling elite for decades. But what has changed is the tech that has become a tightly woven part of our daily existence. From social media to CCTV to online purchasing to the smartphone you might be reading this article on right now, our every move can be monitored and aggregated with ease. The privacy issues here are nothing new, with everyone from Facebook to health insurance providers to even our digital television all holding an astonishing amount of information about us. And in a neo-authoritarian world, data is king, and social credit could make full use of the figures.

Of course there can certainly be positives. Live a certain lifestyle and the doors are open to smarter education, better vacations and a thoroughly decent life.
But it’s not only the potential to be shaped by one particular number that’s the problem in China. Data sharing means that information from different regional governments and agencies will be more accessible to others, meaning that a mistake in one area could quickly haunt you in many others.

There are reportedly already instances of this happening. For example, people on the Court blacklist are reportedly banned from buying luxury goods on Tmall. This online shopping service, operated by one of China’s largest tech firms, has hooked up its own social credit system – Sesame Credit – in conjunction with the government’s blacklist. Seen in this light, the links with what you probably know as a credit score are clear.

In the PRC, numbers are crunched to determine one essential human trait: Trustworthiness. Because trust equals order. In urban conurbations elsewhere, much daily order is an uncontrived result of millions of people sharing an environment. This is particularly true of bustling cities where eyes are always at street level – whether via resident, tourist or simple passer-by. This level of organic, unprejudiced surveillance is what unofficially governs social order. But a culture left to chance is not something that particularly flies with the governing bodies of the PRC. The way they see it, public order leads to stability and stability is the primary motivator of continued financial growth.

There’s an argument that states that Social Credit is simply getting China up to speed with the rest of the world when it comes to personal finance. Until very recently, the country’s banking sector was not consumer focused. In fact, only around 25 percent of Chinese people today even have a traditional bank account. With no credit history to go on, mortgage lenders have no data to determine who they offer cash to.

“The system is really meant as a device to make the marketplace work better” explains Rogier Creemers from Leiden University, describing the more benevolent side of social credit. By examining blacklists you feature on, as well as other data held about you, the idea is that financial institutions will be able to tell if you’re trustworthy – and loanworthy – at a glance.

Of course, there is another way to view this whole story. Only this one is a little more altruistic in nature. With so much data available to explain our every move, why not use it to establish a better way of living instead?

“We already see [examples of] traffic sensors at junctions that feed congestion data back to controllers, and infrared and stereo cameras to monitor pedestrians and cyclists, and these will all increase in popularity and in effectiveness”, explains Laurie Winkless, author of Science and the City. “We’re beginning to see buses and trams being given priority over other traffic based on how many passengers they are carrying. Self-driving vehicles will require huge networks of sensors that can continuously communicate amongst themselves and with the vehicles – this again will help with monitoring live traffic flow.”

Think of all of those data-points already being collected that could be used to optimise our world. Especially when we can  take a more humane approach to design.

“The design of a city has a huge influence on behaviour,” explains Winkless. “Want people to leave their cars at home? Provide low-cost, reliable mass transit options. Want to encourage people to walk? Widen footpaths, improve lighting and signage. Improve air quality? Plant more trees, dedicate more space to parks, make it easier and more attractive to cycle, or use an electric vehicle to get around.”

Whether the notion of social credit can ever really work should be clear by 2020, when the Chinese government hopes the national system will be up and running, but the reality could lie in application over idea.

Today, in 2019, our lives are tracked, monitored, enumerated and scored. Whenever we summon an Uber, the driver uses our score to determine whether they want to pick us up – and we use theirs to decide whether we’re willing to get into their car. When we look up a restaurant on Yelp, we’re using the review scores from previous diners. And when we post on Twitter, our social standing is defined by the number of followers we have and the number of retweets and likes we receive. But that shouldn’t have to mean that the things we take for granted as part of daily life might ultimately be reliant on those scores.

While a life so entrenched in technology cannot avoid aggregation, an dystopian future isn’t necessarily a given. The fact is that success lies in one simple thing: how well we use the data. 

Illustration: Kevin Hong