Nosimpler

by Yohan J. John

This is the era of metrics: it seems that if we are to hack a path through the information jungle of the 21st century, we must be armed with an arsenal of scores, quantities, indices, factors, grades, and ratings. Our corporate and governmental overlords seem most comfortable parlaying in the seemingly objective language of numbers.

But can complex social and biological conditions be boiled down to scores? To GDP-per-capita, or a happiness index, or a body mass index? Social and biological metrics are attempt to quantify things that often seem unquantifiable: the overall health of a country or of a person, the ability of a school to educate its pupils, the quality of a consumer product, and even the aesthetic value of a movie, TV show, or musical album.

I've always been uncomfortable with this process of quantification: on the one hand  reducing any phenomenon to a single number seems like a major oversimplification, and on the other, the procedures for generating such numbers are often opaque. How exactly is inflation calculated? Or the cost of living? How do Nielson ratings work? Or the Netflix recommendation system? My discomfort with metrics began to crystallize and expand when I was introduced to a somewhat obscure "law" that should perhaps be more widely known outside of the dismal science that originated it.

Goodhart's Law was originally formulated as follows: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." The word 'statistical' probably doesn't excite most people. But if we cut to the essence of what is being said, we find a rule of thumb (it's not a real law of nature) that might have implications well beyond the world of economics:

"When a measure becomes a target, it ceases to be a good measure."

Let's unpack this idea with a few examples. We can start with academic testing. Written examinations are a time-honored way to assess whether a student has learned something. But when tests scores become the metric by which to judge the performance of schools and teachers, then the connection between the test score and the quality of education often breaks down. This is because teachers "teach to the test", or even cheat in order to raise scores. By making test scores the target, rather than one among many factors that go into the assessment process, the people involved are incentivized to find the path of least resistance that leads to the highly specific targeted outcome. This drive to find the easiest route to higher test scores is what breaks the correlation between test scores and the more general goal of quality education. You can do well on a test because you have a degree of mastery of the subject, or because you were trained in a mechanical fashion to do the specific test, with no care given to your ability to apply what is learned in other potentially important contexts. [1]

Good examples of Goodhart's Law crop up because of search engine optimization. Early search engines ranked webpages based in part on their content. This was easily manipulated by dodgy websites that contained vast repetitive stretches of text specifically tailored to web-crawling algorithms. Google's PageRank system was a major innovation because it used links between websites, rather than content alone, as the metric for importance. The logic was that useful pages tend to receive lots of incoming links. But this too can be gamed: webmasters can engage in link trading to move their websites higher up in Google's search results. [2]

If there is more than one way to produce the desired metric, then the agents being measured — the schools, the markets, the websites — will game the system whenever possible to minimize their effort and discomfort. The net effect is a kind of social quantum mechanics: the act of observing a complex social system can disrupt it, and also break down the link between the metric and the system. For this breakdown to occur, specific conditions need to be met. First, the metric must be loose enough to confer a degree of play to the system: there must be more than on causal pathway leading from the system to the metric. High test scores can result from a solid understanding of the subject, or from "teaching to the test", or from outright cheating. Second, the metric must be used for some 'payout' purpose: the system being measured must feel the consequences of the metric. Websites that get more traffic get more ad revenue.

If a metric is connected to a payout, this creates a form of feedback in the system. This feedback in turn creates a selective pressure, or a drive towards competition, if the payout is a limited resource (like money or attention). The metric is a way for the 'measurer' (the school board, a company, a regulator) to declare its desires to the system, and therefore for the system to adapt to these desires in order to get paid. The goal might actually be high quality education, but because this is very difficult to assess in all its complexity, the easily-gamed metric of average test scores steps in as substitute for the elusive ideal. Some sort of 'objective' guide for getting to one's goal must be better than none at all, right? Perhaps, but be careful what you wish for, because it might not be what you were really wishing for.

In HG Wells's short story "The Truth About Pyecraft" (which you can read for free online here if you are worried about spoilers), a man named Pyecraft asks the narrator for a magical Oriental recipe to help him lose weight. But the recipe turns out to make him weightless, leaving his size and shape intact. After consuming the magical elixir, Pyecraft floats up to the ceiling like a helium balloon. The narrator concludes that Pyecraft has committed the "Sin of Euphemism": asking for a way to lose "weight" when what he really wanted to do was lose fat. One's weight normally has a fair correlation with the amount of fat in one's body, but in the universe Pyecraft inhabits, this relationship can break down. (Pyecraft has to weigh himself down with lead underwear in order to live a normal life from then onwards.) Euphemism is the use of a pleasant or neutral word to talk about something unpleasant. The "euphemistic" nature of Goodhart's Law is that we use an easy-to-measure number to target something hard to get a handle on. Unless we are very lucky indeed, most of the social metrics we come up with will give the system far too much leeway.

If Goodhart's Law were no more than a specific example of the more general tendency of humans to mess with institutions, then perhaps I would not be that interested in it. But I think that the conditions for Goodhart's Law — a multiplicity of causal pathways, feedback, and competition for limited resources — are not simply hallmarks of social networks. You can also discern these dynamics in neural networks. But in order to understand this, we will need to set up an extended metaphor for how the brain works.

The Neural Citadel

Nowadays we routinely encounter descriptions of the brain as a computer, especially in the pop science world. Just like computers, brains accept inputs (sensations from the world) and produce outputs (speech, action, and influence on internal organs). Within the world of neuroscience there is a widespread belief that the computer metaphor becomes unhelpful very quickly, and that new analogies must be sought. So you can also come across conceptions of the brain as a dynamical system, or as a network. One of the purposes of a metaphor is to link things we understand (like computers) with thing we are still stymied by (like brains). Since the educated public has plenty of experience with computers, but at best nebulous conceptions of dynamical systems and networks, it makes sense that the computer metaphor is the most popular one. In fact, outside of a relatively small group of mathematically-minded thinkers, even scientists often feel most comfortable thinking of the brain as a elaborate biological computer. [3]

However, there is another metaphor for the brain that most human beings will be able to relate to. The brain can be thought of as an economy: as a biological social network, in which the manufacturers, marketers, consumers, government officials and investors are neurons. Before going any further, let me declare up front that this analogy has a fundamental flaw. The purpose of metaphor is to understand the unknown — in this case the brain — in terms of the known. But with all due respect to economists and other social scientists, we still don't actually understand socio-economic networks all that well. Not nearly as well as computer scientists understand computers. Nevertheless, we are all embedded in economies and social networks, and therefore have intuitions, suspicions, ideologies, and conspiracy theories about how they work.

Because of its fundamental flaw, the brain-as-economy metaphor isn't really going to make my fellow neuroscientists' jobs any easier, which is why I am writing about it on 3 Quarks Daily rather than in a peer-reviewed academic journal. What the brain-as-economy metaphor does do is allow us to translate neural or mental phenomena into the language of social cooperation and competition, and vice versa. Even though brains and economies seem equally mysterious and unpredictable, perhaps in attempting to bridge the two domains something can be gained in translation. If nothing else, we can expect some amusing raw material for armchair philosophizing about life, the universe, and everything. [4]

So let's paint a picture of the neural economy. Imagine that the brain is a city — the capital of the vast country that is the body. The neural citadel is a fortress; the blood-brain barrier serves as its defensive wall, protecting it from the goings-on in the countryside, and only allowing certain raw materials through its heavily guarded gates — oxygen and nutrients, for the most part. Fuel for the crucial work carried out by the city's residents: the neurons and their helper cells. The citadel needs all this fuel to deal with its main task: the industrial scale transformation of raw data into refined information. The unprocessed data pours into the citadel through the various axonal highways.  The trucks carrying the data are dispatched by the nervous system's network of spies and informants. Their job is to inform the citadel of the goings-on outside its walls. The external sense organs — the eyes, ears, nose, tongue and skin — are the body's border patrols, coast guards, observatories, and foreign intelligence agencies. The muscles and internal organs, meanwhile, are monitored by the home ministry's police and bureaucrats, always on the look-out for any domestic turbulence. (The stomach, for instance, is known to be a hotbed of labor unrest.)

The neural citadel enables an information economy — a marketplace of ideas, as it were. Most of this information is manufactured within the brain and internally traded, but some of it — perhaps the most important information — is exported from the brain in the form of executive orders, requests and the occasional plaintive plea from the citadel to the sense organs, muscles, glands and viscera. The purpose of the brain is definitely subject to debate — even within the citadel — but one thing most people can agree on is that it must serve as an effective and just ruler of the body: a government that marries a harmonious domestic policy — unstressed stomach cells, unblackened lung cells, radiant skin cells and resilient muscle cells — with a peaceful and profitable foreign policy. (The country is frustratingly dependent on foreign countries, over which it has limited control, for its energy and construction material.)

The citadel is divided into various neighborhoods, according to the types of information being processed. There are neighborhoods subject to strict zoning requirements that process only one sort of information: visions, sounds, smells, tastes, or textures. Then there are mixed use neighborhoods where different kinds of information are assembled into more complex packages, endlessly remixed and recontextualized. These neighborhoods are not arranged in a strict hierarchy. Allegiances can form and dissolve. Each is trying to do something useful with the information that is fed to it: to use older information to predict future trends, or to stay on the look-out for a particular pattern that might arise in the body, the outside world, or some other part of the citadel.  Each neighborhood has an assortment of manufacturing strategies, polling systems, research groups, and experimental start-up incubators. Though they are all working for the welfare of the country, they sometimes compete for the privilege of contributing to governmental policies. These policies seem to be formulated at the centers of planning and coordination in the prefrontal cortex — an ivory tower (or a corporate skyscraper, if you prefer merchant princes to philosopher kings) that has a panoramic view of the citadel. The prefrontal tower then dispatches its decisions to the motor control areas of the citadel, which notify the body of governmental marching orders.

Let us briefly step away from the citadel metaphor to imagine a concrete situation where different parts of the brain might be saying different things, and therefore having to compete for access to the decision-making tower. Imagine you are in the jungle, and you are somewhat hungry, but also aware that there is a man-eating tiger on the loose. Let's say you find a source of food. Should you start eating, and therefore lower your guard slightly? Or should you stay vigilant, and wait to find a source of food in a safer location? It depends on a variety of factors: on metrics that the brain has either inherited or invented. If you stay hungry for too long you'll become weak, and then it will be much harder to deal with the tiger. But if you distract yourself through the process of eating, you might get eaten yourself.

The brain's metrics might be signals like the volume of animal noise in the jungle (which some say is an indicator of a tiger's proximity), or some estimate of the body's energy levels. Whenever you have two mutually exclusive courses of action, only one of them can win out. And the way the brain decides on one action versus another is by weighing those internal metrics. The brain areas and sub-areas that have proven consistently useful become increasingly able to communicate with the decision-making. In neuroscience we explicitly use the term 'credit assignment' to refer to how a neural network can pick out the neurons with most useful piece of information available, and allow them to have a stronger effect on decisions and actions. One of the mechanisms for this is called synaptic learning: the connections (synapses) between neurons can be strengthened or weakened depending on a variety of factors.

So going back to the citadel, we can make guesses about how payments work in the neural economy. Neurons and groups of neurons receive synaptic credit, which grants them more access to places up and down the organizational hierarchy. Good workers gain the trust of their bosses, co-workers and underlings, and can therefore more easily make their opinions heard. They also gain access to the best quality raw material — data arriving from elsewhere. Meanwhile, workers who aren't saying anything particularly useful or relevant to the context get less attention and less access. Success in the neural economy is much like success in a social network: it all boils down to the number and strength of connections. (Just having lots of connections is not the point, however. Quality and relevance matter. These connections must serve the distinctive goals of the neuron and the networks it is part of.) Whether neural connections are strengthened or weakened is a complex matter, and depends on wider forces including the neurochemical weather conditions.

A troubled neighborhood

Celebrity neurochemicals like dopamine, serotonin, and oxytocin can be thought of as indirect indicators of weather conditions — the temperature, the pressure, the humidity, the cloud cover — and they can have all kinds of effects on the neural citizenry, which tries to adapt to the ever-changing conditions. They can change the way a neuron talks, or modify the rules by which its connections with other neurons get regulated. Just as a meteorologist's instruments serve as a window on what the weather is and how it will develop in the near future, neurochemicals and hormones are used in the brain as metrics for signaling and assessing various situations in the brain and body.

Let's focus on one particular neural weathervane: dopamine. I've mentioned in a previous 3QD essay that dopamine has been mischaracterized as 'the pleasure chemical'. Dopamine doesn't just get released in response to pleasurable activities: it can also be released by negative experiences or by situations that are unpredictable. But dopamine does seem to have a powerful ability to change the decisions made by the brain as a whole. Drugs of abuse often affect the dopamine system. So researchers initially thought that drugs like cocaine or methamphetamine were just so pleasurable that the addicts simply couldn't resist them. The story turned out to be much more complicated.

Dopamine is not itself the chemical that engenders the subjective experience of pleasure. There may not be any single chemical that operates in such a simplistic way. What it might be is a kind of metric that the brain uses to represent some (still debated) class of potentially useful information. Dopamine has a strong ability to change the synapses between neurons. The chemical and informational highways that lead up to the dopamine cell neighborhood seem to allow dopamine cells to notify the rest of the brain of situations that are potentially worth paying attention to, and perhaps remembering for later.

So if you experience something pleasurable, like a taste of honey, you will probably want to know how you can repeat this experience. In order to do this you need to find out which pieces of information preceded the taste of honey. Perhaps it was the sounds of bees. Or the sight of a particular tree. A burst of dopamine seems to help form a link between certain kinds of information and their consequences. Once these links are made, the next time your circumstances resemble the happy honey discovery episode, you can begin to respond, perhaps ensuring that you get more honey than last time, or paying closer attention, so you can discover the precise source of the honey. More proactively, you may also be able to re-create those circumstances — by getting into beekeeping, for instance.

Remember that dopamine is not a direct mapping from honey to the brain. It is embedded in a complex causal web woven by the blind spider of natural selection. Evolutionary forces are under no compulsion to construct signaling pathways that convey one and only one thing. Natural selection is a satisficing process, so it creates situations that are good enough, but not necessarily the most efficient. Dopamine signaling was evidently good enough as a signal of potentially useful information for evolutionary forces to allow it to change synapses and bias our behavior. The raw data generated when honey arrives at the tongue and later the stomach travels along multiple roads that lead into the brain. One of those roads leads to the dopamine neighborhood. But the full subjective experience does seem to involve the consequences of all those other data streams.

When a drug starts to make trouble on the streets of the dopamine zone, it is very likely to be doing so through causal pathways that are distinct from the ones linking honey with dopamine. Because there is more than one way to get a dopamine cell to sound the alarm, dopamine's correlation with usefulness (or pleasure, or unpredictability) can easily break down. Drugs may have a pleasurable component, but addicts often talk about how the pleasure can fade away at some point, leaving only the compulsion to feed the addiction. It is at this point that we can import Goodhart's Law into our picture of the neural economy. Perhaps dopamine is a metric that the brain can use to assign credit to useful information, giving the neural sources of that information more access to the halls of power. But if dopamine stimulation becomes a target, it ceases to have a strong relationship with usefulness. It nevertheless retains its more mechanical ability to restructure neural networks through synaptic change. Direct dopamine stimulation might cause the brain's motivation system to skip the 'liking' stage and go straight to the 'wanting' stage.

For drug addicts, it seems that the feedback and competition in their neural networks has created a Goodhart-like scenario, in which behavior becomes focused on the self-reinforcing chemical stimulation of dopamine cells, rather than on a more holistic stewardship of brain, body and society. Circumstances that prevent such a holistic approach may actually plant the seeds of addiction. The famous Rat Park experiments suggest that rats will not become addicted to drugs of abuse — even if they do consume them occasionally — if they are placed in a stimulating environment with sufficient food and opportunity for social interaction. Social isolation may contribute to the overcast neurochemical skies that are most conducive to drug abuse. It may create a situation where chemical stimulation is the only signal that is loud and clear, with nothing healthy to compete with it for access to the decision-making centers.

The head and the heart

Addiction to drugs is just an extreme version of a 'neural Goodhart' situation that can happen to anyone. One of the things the human brain allows for is rational theorizing about the state of the world. Whatever else these theories are for, at the very least they should occasionally help a person navigate in the world. But if rationalizing per se becomes the focus of the theorizing, rather than usefulness or agreement with reality, then we might end up delusional, or paranoid, or in thrall to a conspiracy theory. In such situations apparent internal consistency becomes the only metric for the worth of an idea. Rationalizing can become less useful as a guide to behavior if it becomes the only target of behavior. A group of rogue neurons that specialize in rationalizing might hijack one of the brain's credit assignment centers, giving it the ability to pay other groups to adopt its dubious standards.

We might come up with more examples if we replace 'rationalizing' with other neurochemical signals, ideas, theories, or patterns of behavior. Almost anything can become a 'totalitarian' obsession and lead to the detriment of other modes of thought and forms of life. We can propose a neural Goodhart's Law, which we might as well call Good-head's Law: a neural or conceptual metric might cease to correlate with mental and bodily well-being if it becomes the dominant prism through which experience is viewed.

In allocating power based solely on some reductive measure of goodness or truth, a totalitarian government destroys diversity and create stereotypical internal yes-men. We might go on to define a fundamentalist as a would-be totalitarian: someone who takes very seriously some narrow and rigidly defined set of metrics of personal or social health, and attempts to remake the self or society in slavish accordance with this idealized image. On the personal level, the quantified self movement might confront such issues head on. Metrics for personal health — cholesterol levels, body mass indices, blood sugar levels — capture very broad and often poorly understood statistical correlations. Imagine a person whose goal is to be healthy, and rigidly defines health in terms of these numbers. This person might conceivably find specific drugs or exercise regimens that bring these numbers into their allegedly optimal levels. But there are always mental and social consequences to any change in behavior. If the drug and exercise regimen leads to anxiety or social isolation, it may have various adverse effects on health.

A person who follows a Mediterranean diet might have health indices in the right places, but the good health that accompanies these numbers may also depend on specific causal chains, such as the psychological, social and environmental conditions that frame the person's life. A person might be able to manipulate his or her health indices into the supposedly healthy zone, but in a more forced or anxiety-inducing way that could be counterproductive. Along with Good-head's Law we can propose a Good-heart's Law: health metrics might cease to correlate with good health if they become lifestyle goals.

I hope it's clear that almost everything I've written here is wildly speculative. Not all metrics are going to break down when they become targets — whether they're implicit metrics used by the brain or explicit metrics invented by governments and private companies. Some of them have a very close relationship with the things they are measuring. If you are interested in reducing the number of atoms in your body, then your mass in kilograms is an appropriate metric. If you are interested in controlling how hot a room is, then temperature is the ideal metric.

The Goodhart scenario arises in very specific situations. First, the metric must be trying to capture something that is hard to define — there must be some gap between what the measurer is actually interested in, and what is directly measured. Second, there must be more than one causal pathway that can give you the targeted value for the metric. In other words, there must be no necessary or one-to-one relationship between the metric and the phenomenon being studied. (This is almost always the case in complex networks.) Third, the metric must be used to give feedback in a competitive situation: to allocate limited resources —money, credit, energy, attention — in a neural or social network.  [5]

The letter of the law and the spirit of the law

Goodhart's Law and its (admittedly speculative) biological corollaries serve as a warning for us as we concede more and more power to the algorithmic management of our lives. People working in government, in the business world, and in academic research are increasingly confronting problems that have to do with network level phenomena: at the highest levels we are grappling with complex social, economic and ecological networks, and at the lowest level we are dealing with equally complex networks of neurons and proteins and genes. New metrics will increasingly be deployed to make sense of the ever expanding pool of Big Data. Already on Wall Street there are anonymous trading algorithms doing odd things for unclear reasons. Perhaps in the near future governments and large companies will scan social networks (or the bio-sensors that some people will voluntarily place on themselves) looking for signs of incipient public outrage. Their metrics and algorithms will be marketed to us as 'scientific' and 'objective' approaches to concepts that used to be described in the wishy-washy terms used by artists, humanities professors, and social workers.

Goodhart's Law tells us that if we are not careful about what we target in a complex network, we might end up with something that is not at all what we actually wanted. Perhaps it is simply an amusing quirk of networks — something that future researchers will do away with using better metrics. On the other hand, Goodhart's Law could be a warning for how technocrats might inadvertently lead us into an Orwellian world by convincing society to dispense with those old wishy-washy terms for what we actually want. "Liberty, equality, fraternity, peace… what do they mean? Give me a number!" If everyone is forced to adopt the simultaneously oversimplified and opaque language of  metrics, then we may gradually lose the ability to articulate what is missing.

Perhaps the lesson we can learn from Goodhart's Law is that there must be a difference between the spirit of the law and the letter of the law, and between the spirit of measurement and a specific metric [6]. In a democratic culture, a specific law is a concrete expression of some goal for the self and society. In a scientific culture a metric is a concrete quantification of some multifaceted state of a self or a society. Behind a specific metric there is often a vague awareness of a more complex reality: after all, this awareness is usually what motivates the creation of the metric in the first place. I think it is important not to abandon this awareness, however vague it is, once a metric arises to dispel it. The post-Enlightenment legal systems that structure our lives also arose from a vague awareness: a spirit or sensibility that drives us to become fitter, happier, more productive, and more perfect personalities and polities. We might call these forms of awareness the scientific spirit and the humanistic spirit. They both seem to arise from a common source: the impulse to transcend the material, social, political, and conceptual boundaries of the present moment. This drive towards transcendental novelty also gives rise to art, literature, poetry and music — those wellsprings that seem to provide us with our most resonant expressions of what a self or a society ought to look like. Perhaps our allegiance must be with this mysterious spirit that animates our arts, our laws, and our measurements, rather than with any particular — and therefore limited — manifestation of it.


______

Notes

[1] In India, the prestige of the Indian Institutes of Technology has lead vast numbers of students to prepare for the entrance exam in intensive coaching camps. These coaching centers train students to get into the IITs, but do not prepare them at all for the qualitatively different (and more difficult) academic challenge of the coursework that awaits them. Many of them quickly go from elation at having made the grade to depression and sometimes even suicide.

[2] Several examples that illustrate Goodhart's Law, including an amusing one from the Soviet Union, can be found here.

[3] For an excellent historical discussion of the various metaphors that have been used to describe the mind and brain, see John G. Daugman's essay Brain Metaphor and Brain Theory [pdf].

[4] The trick of seeing the human society as a macrocosmic body is used to great effect in a section of the song 'Maya' by the Incredible String Band.

The great man, the great man, historians his memory
Artists his senses, thinkers his brain
Labourers his growth
Explorers his limbs
And soldiers his death each second
And mystics his rebirth each second
Businessmen his nervous system
No-hustle men his stomach
Astrologers his balance
Lovers his loins
His skin it is all patchy
But soon will reach one glowing hue
God is his soul
Infinity his goal
The mystery his source
And civilization he leaves behind
Opinions are his fingernails

[5] Economic indicators often check off all three boxes, which is presumably why Goodhart's Law turned up in economics.

[6] The Indian mythologist Devdutt Pattanaik makes an interesting point about the spirit of the law and the letter of the law in a television presentation on the differences between the two great Indian epics, the Ramayana and the Mahabharata. In his opinion the Ramayana, which takes place in an innocent age, depicts a society that still remembers the spirit of the law. The Mahabharata, by contrast, describes events at a later and more corrupt age, when society has forgotten the spirit of the law, and clings to the letter of the law.

 

[Image from Wikipedia: Seventeenth-century plan of the fortified city of Casale Monferrato. The citadel is the star-shaped structure on the left.]

Last week I met Pennsylvania State University research technologist Phillip Galinsky at Transhumanism Summer Camp at Juniata College. Galinsky gave a terrific talk on the recent advances in neurotechology. Galinsky later showed me his video And The Deaf Hear where he melds the power of art with the brilliance of technology. I asked him if it would be OK for me to post the video at Reason? He kindly consented.

Galinsky explains why he put together this one minute video containing footage of activation of cochlear implants and auditory brainstem implants. From Galinsky:

• This is just a small example of the low hanging fruit that science has produced for modern artists, but which modern artists are not often using as subject matter for their work.
• My intention is to use the emotional and glorifying power of music to bring the viewer into an emotional state that more closely resembles the emotional state of a deaf person gaining hearing.
• Modern art culture must rediscover what Richard Wagner referred to as the “glorifying power of art”.  Art has been used for glorification of phenomena since its birth.  Art was the first neurotechnology and good art is still just as functional as a tool for behavioral change as it was when it began – our brains simply have not changed enough as of yet to resist the emotionally persuasive power of art.
• Most artists today are used for advertising, and art has always been a tool for behavioral change.  Completely free modern artists tend to focus on negative topics, and it’s important to make negative art to bring attention to important negative issues.  However, we need to also embrace a more Roman attitude towards art, and glorify those subjects that are worthy of glorification, such as restoring hearing to the deaf.

Enjoy your weekend.

During the Spanish revolution of 1936, anarcho-syndicalists seized the movie studios and labs based in Barcelona and Madrid. In the ensuing years, they made radical documentaries, they made politically charged dramas, and, of course, they made a children's musical. Wait, what?

Yes, a children's musical. The anarcho-studios still needed to pay the bills, and that meant producing commercially viable entertainment along with the agitprop. Or, in this case, combining their commercially viable entertainment with their agitprop. In Nosotros somos asi, young Spaniards meet regularly for a sort of informal variety show at a boy's house, watching other kids sing, dance, recite a poem about a kitten, and even do a Busby Berkeley–style production number. These performances are intercut with scenes linked to the ongoing social revolution. There is talk of barricades. A boy's father is arrested, and then some children persuade an official to destroy the evidence against him. The rich kid sheds his class prejudices and develops a proletarian consciousness. Eventually the children have a big revolutionary meeting where the girls demand equal rights with the boys and a sign calls for less study time, more recess, and the abolition of arithmetic.

Unfortunately, the only subtitled version of the film that I could find was translated into French, not English. (The summary in the previous paragraph comes via my bad college Spanish, my worse high school French, and the judicious use of Google Translate.) But even if you can't understand a word of it, you should at least watch the introductory sequence, with children in different poses declaiming directly to the camera. And then, if you want to see the Busby Berkeley knockoff, you can skip to the 12:17 mark:

(For past editions of the Friday A/V Club, go here.)

While the dangerous breadth of modern state surveillance has been rightfully exposed by whistleblowers like Edward Snowden, many of the forces that allow this underhanded Internet spying have gone remarkably unnoticed. In fact, an unexplored world of private sellers surreptitiously collaborates with intelligence agencies to help maintain their expansive snooping apparatuses. These security agents for our digital panopticon receive virtually no scrutiny thanks to their privileged, yet nuanced, relationships with powerful groups (and subsequent lack of mainstream-media coverage). But this month, the shadowy world of mercenary exploit sales finally had its huge Snowden moment.

In early July, an activist hacker known as "PhineasFisher" effortlessly infiltrated the systems of a notorious Italian zero-day exploit seller, called "Hacking Team." ("Zero day" refers to security vulnerabilities that are unknown to vendors, which "exploit sellers" often make available to the highest bidder.) PhineasFisher dumped 400 Gigabytes of documentation online for the world to browse. The trove confirmed what many in the security community had long suspected, including bombshell revelations that Hacking Team maintained business relationships with almost 40 different governments including the United States and Russia, sold spyware to brutal dictatorial regimes, and sold products that directly targeted journalists, software developers, and activists for surveillance and monitoring.

The transparency imposed on the rogue Hacking Team was incredibly valuable on its own; in fact, one of the company’s own vendors has called it a "blessing in disguise" to shed light on the industry and begin a discussion of zero-day sales reform. But the Hacking Team hack also provides important lessons about the broader security ecosystem and the thinning line between private and public entities as we adapt to the age of hacking without borders.

The Hacking Team was typical of an above-ground business operating legally in the exploit market. Like Germany’s Trovicor, France’s Amesys, the UK’s BlueCoat, and previous PhineasFisher target Gamma International, Hacking Team profits by selling exploits of popular computer software to powerful groups under the guise of "cybersecurity." When firms offer to look for and report any vulnerabilities so the firm that hired them can patch up and improve their software, this can be a wholly legitimate and beneficial trade. Often, however, these groups merely sell governments different ways to spy on or manipulate political enemies and even innocent citizens.

Indeed, the difference between these kinds of groups and the more stereotypical, hoodie-wearing, lone wolf hacker-for-hire is often one of style rather than ethical substance. Both of these groups make money by discovering or purchasing unknown computer bugs and selling them to governments, political parties, or even terrorist groups for a healthy mark-up.

Zero-day vulnerabilities are incredibly useful to parties wishing to unknowingly manipulate other people online. They are a bit like having a monopoly on a secret entrance to a popular computer program that only you know about. Zero-days can be exploited to remotely inserted malware or spyware that will activate anytime a user sends an online payment or updates iOS or runs Adobe Flash Player. (Incidentally, it might be a good idea to uninstall Flash for now, since we now know Hacking Team sold not one but two Flash exploits.) Other times, exploit merchants use vulnerabilities that are already known and target people running older, unpatched versions of popular software instead. This type of exploit service constituted the bulk of Hacking Team’s portfolio.

The trade in software exploits to further government surveillance is troubling enough from a privacy perspective. Activist groups such as the Electronic Frontier Foundation (EFF) and Reporters Without Borders have long criticized such practices for violating human rights and expanding the global net of digital surveillance.

But there are grave security implications as well. Selfishly hoarding zero-day vulnerabilities intentionally ensures that the Internet will remain systemically insecure. Going a further step and exploiting any kind of vulnerability for political surveillance or oppression could potentially introduce catastrophic weaknesses beyond the scope that the initial exploiters ever anticipated. A responsible netizen finds a zero-day and reports it to the public so that we can all be more secure. An unscrupulous sociopath sells it to Ethiopia for $1 million to crack down on U.S. journalists and wreck huge parts of the Internet in the process.

Security researchers pored through the Hacking Team document-dump on Wikileaks to determine which software vulnerabilities Hacking Team was selling so they could warn the public about which products needed to be updated or uninstalled. They have found three zero-days so far: the two Flash bugs and another for the Windows kernel. While those who practice good cyber hygiene will be able to inoculate themselves against these revealed exploits, the vast majority of less sophisticated Internet users may still be vulnerable to attack as prepackaged "exploit kits" of all three bugs are being sold to newbie hackers.

It is clear that "security" was far from the top priority for Hacking Team because their own security sucked. Hacking Team was not a sophisticated cybercastle whose alligator-filled moat nonetheless failed, it was an inflatable bounce-house with a paperclip lock. Their password was "P4ssword"—when it wasn’t "wolverine," "universe," or "Pssw0rd," that is. In the middle of a sensitive email exchange with an outside associate, Hacking Team COO Giancarlo Russo suddenly remembers to ask, "Do you have PGP [email encryption] by the way? We really do need to encrypt these emails." This one moment of late foresight is far outweighed by the firm’s incomprehensive encryption and poor user operational security.

By not-so-secretly stockpiling destructive exploits and engaging in ample public boasting, Hacking Team was more or less begging to be attacked. Their one-stop-shop arsenal of poorly-protected cyberweapons proved too tempting a target for rival hackers. Really, Hacking Team CEO David Vincenzetti should have known better. A veteran of the anti-authoritarian, pro-privacy Cypherpunk hacking movement, Vincenzetti cut his radical teeth developing a "file tampering detector" that would identify and repel intruders like Hacking Team from computer systems in 1992.

But Vincenzetti has changed quite a bit since the days when he participated in the same listservs as Wikileaks founder Julian Assange and EFF co-founder John Gilmore. His security chops have certainly suffered. Despite being an early promoter of email-encryption software, emails show that Hacking Team hardly used PGP at all.

More fundamentally, the "freedom hacker turned government tool" angle of the Hacking Team story reveals the unfortunate incentive structure presented to the tiny elite of hackers capable of building—or breaking—the global surveillance network that tracks our every online move. They can choose to fight or expose the system, risking media demonization, foreign asylum, and even lifelong prison sentences for the heinous crime of defending our freedoms online. Or they can sell out and enjoy fat retirements as cyberweapons dealers of choice for the world’s repressive states. Either way, this episode is an important reminder that the enemies of an open Internet are not limited to the state. 

Bolivian President Evo Morales gave Pope Francis a crucifix fashioned in the form of a Communist hammer-and-sickle during Francis' stop there. The Argentinian pope is on his first trip to South America since assuming the papacy, and did not appear pleased, murmuring "no está bien eso" and shaking his head.

You can watch the gift exchange below:

The Vatican tried to distance the pope from his objection. The hammer-and-sickle crucifix was a replica of one created by a Spanish Jesuit killed by the Bolivian government in 1980. The Catholic News Agency reports:

At a July 9 press briefing the Holy See press officer, Fr. Federico Lombardi, noted the lack of clarity in the audio of the exchange, and remarked that Pope Francis had been unaware the crucifix was a replica of Fr. Espinal's.

He also claimed that Fr. Espinal's use of it was not ideological but expressed a hope for dialogue between communism and the Church, adding that Pope Francis' remark likely expresed a sentiment of "I didnt' know", rather than "This is not right."

Other Catholics ween't as accommodating:

Catholics from various Hispanophone countries rejected Morales' gesture, considering it offensive to the numerous victims of terrorist groups in Latin America and of the historical totalitarian communist regimes.

Bishop Jose Munilla Aguirre of San Sebastián, a Spaniard, tweeted: "The height of arrogance is to manipulate God in the service of atheistic ideologies … Today, once again: #ChristCrucified".

Around the world, the left's become excited about the pope's latest encyclical, specifically and exclusively portions about climate change and environmental stewardship. The pope curried that kind of selective support by tapping anti-capitalist Naomi Klein for upcoming climate talks. But, as A. Barton Hinkle wrote earlier this week, the left's embrace of religious justification for public policy only applies when it leads to bigger government.