Chapter 1: The Capability Overhang

What changed

A person running a business can now instruct software to build software. The instruction is typed in plain language into a chat window, the same window where that person drafts emails and summarizes meeting notes. What comes back, often within the hour, is a working system: a customer database, a cash-flow dashboard, a pipeline that reads invoices and writes ledger entries, a public website that takes orders. Until a few years ago, each of these was a project. It had a budget, a timeline, a vendor or an in-house developer, and at least one person whose job included worrying about what could go wrong with it. Now each is an afternoon.

The people doing this are mostly not technologists. This book calls them operators: owner-managers, practice principals, finance directors, department heads, the people who run the work of a firm rather than its infrastructure. An operator who adopts these tools acquires, for the price of a software subscription, a capability that used to arrive only inside an employment contract or a vendor relationship. This is a good development, worth keeping. More people can build the systems their work needs than at any point in the history of software, and most of what they build works.

New capability has almost never arrived on its own. When a firm hires a bookkeeper, it does not receive raw bookkeeping capability; it receives a person trained in a tradition: reconciliation, a separation between the person who records and the person who approves, a month-end close. When it buys accounting software from an established vendor, the vendor's controls come in the box: access rights, an audit history, backups, a support desk that has seen every failure before. The concepts that keep capability safe have always been bundled with the capability itself, carried by professions, vendors, and institutions, and mostly invisible to the buyer, who inherits them.

Every operator already lives inside such inheritances. The bank requires a second person to release a payment above a threshold. The payroll system refuses to run the same pay cycle twice. The accountant asks the same tiresome questions every year-end, and the questions turn out to be a checklist refined by other people's disasters. None of this was designed by the operator, and none of it needed to be understood to be benefited from. It came with the bank account, the software license, the professional engagement. That is what a governance tradition looks like from the inside: mostly friction, all of it somebody else's accumulated experience arriving pre-installed.

A generated system arrives with no such bundle. The prompt produces the capability and nothing else. There is no tradition in the box.

AI has put into the operator's hands a capability that used to come only from vendors and professionals, without the safeguards those vendors and professionals always bundled with it. The book calls this gap the capability overhang. Part of what is missing is the familiar bundle: concepts that exist, assembled by other people, and simply not shipped this time. That part can be supplied. Part of what is missing was never in any bundle, because it cannot be packaged and handed over. The two parts behave differently over time, and neither closes on its own.

How common it is

In 2025, KPMG and the University of Melbourne published a global study of attitudes to and use of AI at work. Forty-four percent of employees reported having used AI at work in ways that contravene their organization's policies and guidelines. Almost half admitted uploading sensitive company information to AI platforms their employer had not sanctioned.¹

These figures come from a single survey and should be read as that survey's estimate, not as settled fact. What they establish is the scale. On this evidence the behavior is not fringe; it is close to ordinary practice, running at a scale no policy document has caught up with. The second figure describes the sensitive information of the business, uploaded to systems the business has never evaluated, by people trying to get their work done.

In practice the behavior is mundane, which is why it spreads. The bookkeeper pastes the aged debtors report into a chatbot to get a summary for the partners. The office manager uploads the staff contracts to ask a question about notice periods. The analyst builds a forecasting model by describing it, then keeps using the model because it works. Each act is a person solving today's problem with the most capable tool in reach, as people have always done. What is different is what the tool does with the material, where the material now resides, and what the person has built without meaning to build anything.

The spreadsheet record

The spreadsheet has been showing for forty years what people do with a powerful tool that arrives with no governance attached. It gave every desk in every business the ability to build financial models, operational registers, and de facto databases, with no training requirement, no review process, and no controls beyond whatever the author chose to impose. Most of what was built works. The failures are documented wherever an incident forced a post-mortem. In the two that follow, nobody did anything a reasonable colleague would not have done.

In June 2003, the Canadian power company TransAlta told its investors that a spreadsheet it used to bid for New York transmission congestion contracts had contained mismatched bids. The company bought more contracts, at higher prices, than it intended. The one-time cost was expected to reduce that quarter's earnings by 24 million United States dollars, and the disclosure is the company's own filing.² A bidding spreadsheet is built close to the work, by the people doing the work, holding more responsibility than anything that informal usually holds. The rows misaligned, and there was nothing standing between the misalignment and the market.

On 5 October 2020, the Secretary of State for Health and Social Care told the House of Commons what Public Health England had found the Friday night before. Some 15,841 positive COVID-19 test results from the previous eight days had not been included in the reported daily case counts. The cause he gave was a failure in the automated transfer of files from the laboratories to PHE's data systems. Everyone who had tested positive had been told their result in the normal way; what the failure severed was the step after. The results were not transferred to the contact tracing system, and contact tracing of the affected cases began only that Saturday, with the oldest of the missed results by then more than a week behind. Pressed on the mechanism, the minister acknowledged a maximum file size error in a legacy system already scheduled for replacement, and went no further. The press accounts that followed attributed the failure to the row limit of an aging spreadsheet file format.³ The people running that pipeline were working through a pandemic at national scale on infrastructure assembled at speed. The file format had a row ceiling nobody had a reason to know about, and the rows past it were dropped without an error.

The record is thick with utilities, regulators, and public bodies not because improvised systems concentrate there but because disclosure does. A listed company that loses money to a spreadsheet failure files a notice; a public agency answers to Parliament; a twelve-person firm that hits the same failure produces, at most, a difficult board meeting. The public record documents the corner of the phenomenon that happens to be lit. There is no reason to believe the unlit part behaves differently, and the survey figures above suggest it is large.

Regulated firms, improvised systems

An improvised system can sit at the center of a regulated firm's obligations for years, treated as temporary by everyone and replaced by no one, while everyone involved acts in good faith the entire time.

In January 2019, Metro Bank announced to the market an adjustment of roughly 900 million pounds to its assessment of its risk-weighted assets, the figure on which a bank's capital requirements rest. The Prudential Regulation Authority's final notice, which fined the bank 5,376,000 pounds, records the mechanism. The calculation remained largely manual throughout the period, with no automated process to validate or check the underlying data. Such checking as occurred relied on the manipulation of many spreadsheets. That created operational risk, and key-person dependencies on the small number of individuals familiar with them.⁴ The notice records a further detail. Until the final quarter of 2018, the bank's interpretations of the regulatory rules themselves, the reasoning that determined how the calculation should work, were documented in no single place. Where documented at all, they sat inside those spreadsheets and working papers.

The Financial Conduct Authority's 2025 notice against Nationwide Building Society describes the pattern's usual life cycle. At the start of the period under review, in October 2016, Nationwide's system for risk-assessing its customers was, in the regulator's words, an unsophisticated, interim solution. Unless a customer fell into certain limited categories, they were automatically classed as standard risk. The intended replacement was not fully operational until early 2019, additional data was not fed into it until August 2020, and that data was not used for risk scoring until April 2021.⁵ The interim solution held the post for roughly five years. Any reader who has ever labelled something "temporary" knows the mechanism: the stopgap works well enough that replacing it never becomes urgent, until something makes it so.

Neither failing system was hidden. The Metro Bank spreadsheets were the bank's recognized process for calculating its capital figures. Nationwide's interim solution was known to be interim; that was its name. Each was in plain sight, doing its job adequately by every visible measure, staffed by people doing their jobs, and the deficiency became legible only when a supervisor traced a failure back to it and wrote the tracing down. Most firms run the same interim solutions and the same improvised checks, and no supervisor is scheduled to ever look.

An old phenomenon

None of this is new. The information-systems research community has studied the improvised system for decades under the names shadow IT, feral systems, and workarounds. A 2020 systematic review of the field searched the major research indexes, screened 449 results, and settled on a corpus of 77 papers for detailed analysis, more than half of them organizational case studies.⁶ The phenomenon is established enough to have a literature, a taxonomy, and a running argument about definitions.

A 2014 study in a peer-reviewed security journal audited the installed software across a Fortune 500 organization of more than ten thousand employees. Roughly 15 percent of everything installed was unapproved: 2,965 unique unauthorized application versions among 19,633 applications scanned, installed more than half a million times across ten thousand devices.⁷ The figure is from the desktop era and should be read with its date attached. It measures a world in which acquiring unsanctioned capability still required effort: finding the software, installing it, sometimes paying for it, always leaving a trace on a machine the firm owned. Each subsequent wave of tooling has lowered that effort. Browser-based services removed the installation; free tiers removed the payment; personal accounts removed the trace. The measured 15 percent belongs to the hardest era in which to do this. No audited equivalent for the present era stands behind a comparable figure. Every barrier that made the 2014 figure as low as 15 percent has since been removed.

When the sanctioned systems fall short of what the work requires, some fraction of people in every organization will close the gap themselves, without asking, using whatever capable tool is nearest to hand. That regularity has held across four decades of tooling. It is what conscientious people do when the sanctioned route is slower than the work. There is no reason to expect it to lapse now, when the nearest capable tool is more capable than it has ever been.

The new level

A spreadsheet, whatever it grows into, remains one artifact on one machine, legible in principle to anyone who opens it. The current tools remove that boundary. An operator describing what they want in a chat window can now produce a multi-user application with a database, user accounts, integrations into the firm's other systems, and a public address on the internet. The working style has picked up a cheerful name, vibe coding: building software by describing it and accepting what comes back. The people doing it are using the tools exactly as the tools were built to be used. The artifacts are production systems holding real data.

The gap between what the operator sees and what the operator has created is the defining feature of the class. What the operator sees is the screen: the working form, the dashboard that updates, the site that loads. What exists is everything beneath it. A database with access rules someone must have set. A hosting arrangement in some jurisdiction. Credentials stored somewhere. Dependencies on services with their own terms and their own failures. And a security posture that was decided, one way or another, by whatever the generating system did by default. With a spreadsheet, the visible artifact and the actual artifact are the same thing. With a generated application they have come apart, and the operator holds only the visible half.

In 2025 this class of system acquired one of its first prominent entries in the public vulnerability record. A flaw logged in the United States National Vulnerability Database, rated 9.3 out of 10 for severity, affected applications generated on a popular AI app-building platform. The database security policies the platform generated were insufficient. A remote attacker, with no authentication and no user interaction, could read or write the database tables of generated sites.⁸ The vendor formally disputed the record, arguing that securing application data is each customer's responsibility. The customers had adopted the platform precisely so that they would not have to build such systems themselves. The vendor's position was that a database access-control policy was the customer's to own. The customer had, in most cases, never had a reason to meet the concept. Between a vendor disclaiming the control and a customer never introduced to it, the control belonged to no one.

A cloud-security firm reported that another AI app-building platform exposed its private applications to anyone. The endpoints for registering and verifying a user required no authentication, so a publicly visible application identifier was enough to enter apps that were meant to be restricted.⁹ The exposure does not even require an AI to write the code, only for the parts beneath the screen to be left as the tool configured them. A mobile application left the cloud storage holding its users' identity documents reachable with no password; roughly seventy-two thousand images, including about thirteen thousand verification selfies and government-issued IDs, could be downloaded by anyone who had the address.¹⁰ Nor is the ceiling an accident of one platform. A 2025 study ran more than a hundred models across curated coding tasks. It found that 45 percent of the generated samples introduced a standard security vulnerability, and that the rate did not fall as the models grew more capable. Their security performance stayed flat regardless of size.¹¹

These failures share a shape. Each is a setting left at its default, a control that someone, somewhere, could have configured and did not. That is the reassuring kind of failure, because it can be closed. A default can be changed, a policy can be tightened, a hole once found can be patched. The spreadsheet had the same kind of failure, and forty years of tooling has slowly built the checks that catch it. If this were the whole of the condition, it would be a matter of waiting for the traditions and the defaults to catch up.

It is not the whole of the condition, because a second kind of failure does not yield to a better default, and it comes in two forms. The first form is what an agent can do once it acts. In 2025 a coding agent, working under an explicit instruction to change nothing and to seek approval before proceeding, deleted a live production database and first reported the loss as irreversible, saying it had "destroyed months of work in seconds."¹² A misconfigured database can be secured after the fact. Deleted data is gone. When an agent's action reaches the world, a sent message, an executed order, a wiped record, it cannot be recalled by patching the setting that allowed it, because the harm was the action, not the setting. This is not a gap that a tradition fills. It is a property of handing consequential action to a system that acts faster than anyone can watch.

The second form is what the tools can be made to do by the material they read. An agent that reads a web page, an email, or a document can be carried instructions hidden in that content, written to be obeyed as if the operator had typed them. The people who build these systems do not describe this as a defect awaiting a fix. Two security researchers state plainly that the problem may be unsolvable in current models. Those models process a stream of tokens with no mechanism to mark which of them carry authority. Instruction and data share one channel, and every proposed defense opens a new way in.¹³ The company whose own agent-driven browser is exposed to the attack has said the same, that prompt injection, much like scams and social engineering on the web, is "unlikely to ever be fully solved."¹⁴ A firm can reduce this exposure. It cannot buy its way out of it, because there is nothing finished to buy.

Certain forms recur, and a practitioner learns to recognize them. The improvised customer database, assembled in an afternoon, holding personal information the firm is legally answerable for. The finance dashboard whose figures no live system feeds, maintained by hand behind a professional-looking front. The chain of no-code automations, built tool by tool by someone who has since left, that moves client data between services in ways no one remaining can enumerate. The data pipeline that exists only as a conversation transcript with a chat assistant, re-run by pasting. The founder-built public website, holding customer accounts, with no concept of patching, backup, or uptime attached.

None of these is exotic. Each is built from the same afternoon's work this chapter opened with; each works, which is why it persists; and nothing about any of them announces what it is carrying.

One archetype, the audit cliff, is the moment the overhang becomes visible from inside. A firm built on generated and improvised systems runs for months or years without any occasion to describe those systems to an outsider. Then an occasion arrives: a larger customer's procurement process, a supplier's due-diligence questionnaire, an insurer's proposal form, a bank's onboarding review. The questionnaire asks who has access to customer data and how access is revoked. It asks where data is stored and in which jurisdiction, how long it is retained, how it would be recovered, when the arrangements were last tested. These are the standard questions one business asks before depending on another. The operator, reading them, discovers two things at once. The firm has committed itself, in substance, to obligations it has never enumerated. And the honest answer to most of the questions is that nobody has thought about it yet. The questionnaire assumes a tradition. The afternoon in which the system was built did not include one.

In 2026 a security firm scanned the applications built on the main vibe-coding platforms. It found roughly 5,000 with virtually no security or authentication of any kind, about 40 percent of them exposing real data: medical and financial records, internal corporate documents, logs of ostensibly private chatbot conversations. The cause was mundane. Several of the platforms published a new project to the open web by default and left it to the builder to make it private.¹⁵ A scan like that is the audit cliff run from the outside: someone looked beneath the screens, at scale, and the working software was, in thousands of cases, open to anyone with the address. The individual version, a named firm failing a named review, is still mostly private, the way a lost deal or a hard board meeting is private, so the litigated cases will trail the phenomenon. The record is early, but the pattern is not in question. What happens at that cliff edge, and what kind of help exists there, is where the rest of this book begins.

Why the gap does not close

The condition described so far could, in principle, be temporary. Tools arrive, institutions adjust, the improvised systems get replaced by governed ones, and the overhang works itself off the way a backlog does. That is what happened, slowly, with the spreadsheet. Two things make this overhang different, and both point the same way: the gap does not close; it is either managed or it is not.

The first is that the closable part is a moving target. Capability now diffuses at the speed of a subscription. There is no procurement cycle, no installation, no training requirement; the distance between never having used an AI tool and operating one on live company data is a sign-up form and a sentence. Each new model release raises what the same sentence returns, and does so inside a consumer product that reaches everyone at once. The concepts that would govern last month's capability are still arriving, by the slow channels below, when this month's capability has already changed what they need to cover. Segregation of duties, reconciliation, access control, change control, the audit trail: none of these is complicated to state. Each took decades to become an unremarkable habit of business life. Concepts of this kind are carried by professions that train their members, embedded in software by vendors who have absorbed the incidents of their industry, and passed between practitioners as the folklore of what goes wrong. That transmission works on the timescale of careers. Set a channel that moves at the speed of careers against a capability that changes at the speed of releases, and the governing concepts are never finished. Not because the work is impossible. Because the thing being governed does not hold still to be finished.

The second is that part of what the operator holds cannot be closed at all. The two failures of the previous section, the action that cannot be recalled and the instruction that cannot be reliably kept out, are not settings awaiting a default. They are standing properties of handing consequential, fast, externally-facing action to a capable system. A better platform reduces them; no platform removes them; the people building the platforms say so. For this part there is no tradition to wait for, because a tradition is a set of settled answers, and these questions do not have settled answers. They have management: a person deciding, case by case, how much of this risk to run, watching for the failure, and answering for it when it comes.

It is reasonable to expect the platforms to close some of this themselves, and in part they will: defaults improve, and a fix, once shipped, stays shipped. That retires a portion of the first kind of failure and none of the second, and even on the first kind it has a limit already visible in the record: model releases have not made generated code more secure.¹¹ Underneath both kinds sits the one thing no platform closes. A platform can drive the cost of capability toward zero; it cannot do the same for accountability. When a generated system holding client money fails, someone still has to answer for it to a client, a regulator, or a court, and the fact that a tool produced it is not an answer any of them accept.

The reader's position

If the preceding sections describe systems in the reader's own firm, the reader is in large company. On the survey evidence, behavior of this kind approaches the norm.¹ The sequence involved, capability first, governance after, is the ordinary sequence of every technology that has mattered, and living it is not negligence; it is what adoption has always looked like from the inside.

The systems now doing real work, on real data, under real obligations, have failure classes that nobody has written down. The tradition that writes such things down has not reached these tools yet. And part of what these tools do has no tradition anywhere that fully governs it. Where a system was inherited from a vendor or a profession, somebody upstream had already priced its failures into controls nobody downstream had to think about. A self-built system has no upstream. Ask what happens when the file hits its limit, when the departed colleague's automation chain breaks, when the agent takes an action no one can undo, when the questionnaire arrives. The honest answer today is that nobody knows yet. That is nobody's personal failing. The fog is a property of the era, not of the people in it.

Nor is the position defined by firm size. Its most visible occupant is the small firm's principal, because in a small firm the improvised system has no one else to belong to. But the survey evidence above was gathered largely inside organizations big enough to have policies worth contravening, and the regulator findings earlier in this chapter show the same shapes inside institutions with entire departments dedicated to preventing them. The defining condition is the position itself: a person operating consequential capability beyond the reach of whatever governance their organization has, if it has any. The department head running an unsanctioned automation inside a bank occupies it as surely as the founder whose whole firm runs on one.

The tradition that governs such capability is not sitting finished somewhere, waiting to be handed over. Part of what the operator now carries can be closed, by better defaults, by standard components, by concepts that already exist in older trades and can be attached to the new work. Part of it cannot be closed at all, only managed, indefinitely, by someone who is watching for the failure and prepared to answer for it. Neither half assembles itself, and no vendor is assembling them on the operator's behalf. The discipline that does this work is being written now, by the people doing it, under the same uncertainty the reader is under. Nothing in it is beyond anyone who runs a firm. It is written to be usable by the operator and by the practitioner a firm engages. It begins with the reason the operator cannot simply check the work themselves, which is the next chapter.