The Unaccountability Machine summary

When YouTube executives realized in 2017 that bizarre cartoon parodies were being recommended to children, they found themselves confronting a faceless chain of decisions no one person had truly intended. How did this happen? The company had defined “engagement” as its core priority, and this algorithmic fixation took on a life of its own. No one had personally chosen to frighten kids – it simply happened because the system had been aimed at maximizing watch time, whatever the content.

Many argue that corporations sometimes act like rudimentary artificial intelligences, using rules and incentives that run on autopilot. Even an influential executive may not fully grasp the web of decisions happening under their watch. In complex setups, procedures can become so elaborate that the system’s overall behavior no longer matches any individual’s intent. Philosophers have compared this to the “Chinese room” thought experiment, in which a person inside a sealed room mechanically answers questions written in Chinese by following a set of instructions – without understanding Chinese at all. The written responses appear coherent to outsiders, but there’s no genuine comprehension behind them. The black-box nature of an organization can similarly shield it from accountability, since it’s tricky to pinpoint where true responsibility lies.

In 1948, mathematician Norbert Wiener proposed that the feedback loops keeping an automated gun sight locked on a moving airplane might also shape how major organizations process information. He noticed that excessive feedback could trigger wild swings in behavior, while too little would leave targets slipping away. This insight launched cybernetics – which later influenced Ross Ashby, a British psychologist investigating how neural networks might stabilize the human mind, and Stafford Beer, a management consultant who applied these principles to real-world businesses.

Their message was that you don’t have to dissect a giant institution’s every inner mechanism. Instead, you can treat each part as a “black box” with defined inputs and outputs, then arrange clear feedback loops so no one piece gets overloaded. An example often used is a simple squirrel cage: the cage’s temperature, light, and other factors can each be regulated by small, self-contained controls, letting the squirrels stay healthy without constant human tinkering. Ashby referred to the number of possible states a system can occupy as its variety. If the outside world changes in many ways, you need enough adaptability to match it – otherwise, chaos results.

In 1881, Joseph Wharton founded a business school at the University of Pennsylvania to elevate the status of management – yet, even decades later, many economists graduate without ever learning how to interpret a balance sheet. This gap suggests that running a profitable enterprise doesn’t guarantee intellectual rigor. Instead, leaders can mask shaky decisions under sophisticated models and financial reports, giving the illusion of competence. Economists, trained in abstract theories, often miss the immediate realities of production and overhead costs. Meanwhile, managers who rely too heavily on standardized accounting can end up with a distorted picture of which products are actually profitable.

For example, a skincare firm might conclude that high-priced lotions outperform cheaper bulk lines because the entire marketing budget is listed as a “period cost” – an expense recognized in a single timeframe, rather than assigned to each product – while cheaper items carry all their direct manufacturing costs. Outsourcing the cheaper line can then seem like a smart move, but hidden logistics or quality problems may end up erasing any short-term savings. In the end, consultants and middle managers often scramble to reinterpret the data, further blurring what truly went wrong.

In 1997, a deal worth over $2 billion briefly padded General Electric’s earnings, after which the company abruptly closed factories and laid off more than a thousand employees to avoid distorting its quarterly numbers. Though this move kept the metrics neat, it undermined the overall resilience of the organization, stripping away the middle management tiers that once handled real-time information. Over the next decade, prioritizing short-term financial gains – what some called the shareholder value revolution – became widespread. It rewarded a few high-profile CEOs and their allies but piled debt and insecurity onto everyone else.

As more companies rushed to cut costs, outsource operations, and shuffle debts through financial engineering, they gradually lost the internal feedback loops that once flagged potential trouble. Corporate decision-making came to hinge on meeting quarterly targets, with executives presiding over “rank and yank” policies – assessing employees and firing the bottom 10 percent, regardless of their performance. Although such strategies appeared efficient on paper, they hid the fact that many firms could no longer adapt if economic shocks arrived unexpectedly. Eventually, entire communities and industries frayed as wages stagnated and local economies became reliant on temporary fixes.

Brian Eno, known for layering musical elements into shifting, adaptive compositions, once found common ground with Stafford Beer, the management consultant who championed cybernetics. They both saw that large systems – whether creative or corporate – work best when they juggle multiple inputs instead of obsessing over a single goal. Eno’s generative music evolves through ongoing feedback, and Beer believed institutions should do the same, preserving accountability in the face of complexity. Ignoring vital signals in favor of quarterly profits or rigid targets only leads to short-term wins at the cost of long-term resilience.

One central idea is that corporations shouldn’t behave like single-minded robots maximizing just one metric. When a firm focuses solely on cutting costs or raising quarterly profits, it amplifies certain signals while ignoring everything else. This leads to short-term wins but destabilizes both the organization and the broader economy. One proposed remedy is ending the practice of using unlimited debt and limited liability to fund buyouts. If investors had to shoulder more of a company’s obligations, they would curb reckless borrowing and encourage managers to weigh long-term effects.