“And however much potential they see in the technology, bosses know they have brands to protect, which means minimising the risk that a bot will make a damaging mistake or expose them to privacy violations or data breaches.” - The Economist, May 21st 2025
When you search for “What is an AI agent” on Google Search, the front page features the management consultants’ answers - including from McKinsey and BCG. These stalwarts of transformation make the case for - and market themselves as the natural partners around - agentic AI innovation. Naturally I asked AI to help me assess their views:
“BCG: 52/100
McKinsey: 66/100
..both pieces have significant shortcomings. Common weaknesses:
Overly optimistic tone without sufficient discussion of current limitations
Lack of technical depth on how agents actually work under the hood
Insufficient discussion of failure modes and realistic implementation challenges
Marketing-heavy language that obscures practical realities
Both are high-level marketing pieces that oversell the current state of AI agents. A truly excellent explainer would spend more time on what agents can't do reliably today, common failure patterns, and the significant technical and organizational challenges most companies face in implementation… BCG essentially says "hire AI employees," while McKinsey is saying "rebuild your company around AI systems." (per Claude)
Moving Beyond Coupons and Concert Tickets
When I was a research analyst, the canonical made-up consumer mobile commerce example in every conference talk was the “so you’re walking by a Starbucks and they send you a mobile coupon…” or some variation thereof. These days with AI agents, it’s the concert ticket example - discover, plan and coordinate calendar/friends and pay for tickets for a permissioned credit card. Funnily enough, this was the same example Microsoft pitched me back in the time (2002!!) of “Hailstorm” aka “.NET My Services”, a centralized system where users could store and access their personal information and delegate permissions to automated services to act on their behalf.
The concert ticket task requires your agent to be able to read browser cookies, unlock your password manager, scrape emails for confirmation codes, and edit your calendar. To work, these agents demand sweeping, continuous (what Meredith Whittaker of Signal (1) calls) “root-like permission” that tears the “blood-brain barrier” between apps and operating systems, turning every personal device into a single point of surveillance and failure.
This shifts privacy from being a largely regulatory and theoretical topic, to instead become a pre-condition for the autonomy, security, and legitimacy we’ll need for agentic AI to function.
Privacy-Safety Dilemma at Scale
Agents multiply both the number of attack surfaces and the relative scale of each. Agents that can execute thousands of actions per second may exfiltrate terabytes before anyone notices (2), which is why Mustafa Suleyman warns that any serious plan to contain autonomous systems will rely on audits, logs, and licensing that risk becoming an all-seeing dragnet (3).
This creates what we might call the "verification paradox": the more powerful an agent becomes, the more comprehensive our monitoring must be to ensure it remains aligned with human intentions. But comprehensive monitoring of autonomous systems that interact with every aspect of our digital lives means comprehensive monitoring of us. If your agent handles your email, calendar, banking, health records, and social interactions, then auditing the agent requires visibility into all of these domains.
The technical requirements make this worse. Unlike traditional software that processes discrete inputs, agents maintain persistent context across interactions. To verify that an agent isn't being manipulated or developing concerning patterns, auditors need access to its memory, reasoning chains, and decision trees - essentially a complete record of how it models your preferences, relationships, and behavioral patterns. This isn't just surveillance of your actions but surveillance of an AI's increasingly sophisticated model of your inner life.
Without deep inspection you cannot verify what an agent is doing, but deep inspection can erase privacy altogether absent mathematically-provable controls AND legal oversight. The usual privacy-security tradeoffs become exponentially more severe when the system being secured is designed to act autonomously on your behalf across every domain of your existence.
This paradox helps explain why current agent deployments remain so limited. Companies aren't just worried about technical failures - they're grappling with the impossibility of providing both meaningful autonomy and meaningful accountability without creating exactly the kind of total information awareness that privacy advocates have long warned against.
The Laissez-Faire Counter-Narrative
Not everyone accepts new controls. Techno-optimists like Marc Andreessen insist that "permissionless innovation" is the true safeguard of progress, arguing that regulation poses a bigger threat than data misuse (4). This view has merit: historically, breakthrough technologies have emerged from experimentation at the edges, not from committee-designed specifications. The internet itself grew through loose protocols and emergent standards rather than top-down architecture.
Open-source advocates extend this logic, claiming that transparency and massive developer communities catch privacy flaws faster than closed audits ever could. They point to projects like Signal and Tor as proof that distributed scrutiny creates more robust privacy than corporate compliance theater. Think-tank studies argue for "tiered openness" rather than blanket licensing, claiming that hybrid disclosure can preserve both security and innovation (5).
The crypto-maximalist position goes further: Balaji Srinivasan argues that self-sovereign identity and network-state governance can embed privacy in decentralized ledgers, eliminating the need for central gatekeepers entirely (6). From this view, the solution isn't better regulation but better architecture - cryptographic guarantees that make surveillance structurally impossible rather than merely illegal.
Free-market groups such as the Competitive Enterprise Institute demand proof of specific market failures before regulators intervene, warning against a "ready, fire, aim" approach to AI rules (7). They argue that premature regulation could lock in today's technical limitations while preventing tomorrow's breakthroughs.
These arguments aren't easily dismissed. Markets have indeed produced privacy-enhancing innovations faster than legislators can understand them. But they assume that individual choice and technical solutions can address systemic power imbalances; a faith that feels increasingly strained as AI agents demand unprecedented access to personal data and decision-making authority.
Privacy-Preserving Containment
The debate’s extremes misunderstand what privacy must become. Reactive monitoring is too slow, yet blanket transparency threatens intimacy and dissent. The path forward is privacy-preserving containment:
Least-privilege design. Agents should receive granular, time-boxed scopes, not root keys.
On-device planning. Keep memory and reasoning local; send only encrypted execution tokens to the cloud.
Cryptographic proof-of-intent. Every tool call or memory write is signed before execution, turning policy compliance into math rather than trust.
Immutable audits visible to the user. Dashboards and kill-switches restore individual agency while giving regulators verifiable evidence.
Privacy-enhancing technologies (PETs). Techniques like homomorphic encryption, differential privacy, and federated learning can make data useless to anyone but its owner, reducing the need to inspect content at all (8).
These measures borrow as much from cryptographers as from policymakers. They also partially satisfy the open-source and free-market camps by favouring transparency of proofs over transparency of personal data.
The debate isn't whether we should choose privacy over progress, or security over openness. It's about architecting systems where privacy becomes the foundation upon which genuine autonomy and innovation can thrive. As we enter an age of autonomous systems acting on our behalf, privacy transforms from a regulatory checkbox into the essential infrastructure of human agency: not an obstacle to progress, but its prerequisite.
Notes:
Image/title quote from Anchorman: The Legend of Ron Burgundy
Meredith Whittaker on agentic AI and “root permission.” https://www.linkedin.com/posts/nichalley_hands-down-the-best-overview-of-the-intrinsic-activity-7343816273593569280-MDeO
“Why Reactive Security Has Reached Its Limits: The Emerging Threat Landscape in Agentic AI.” MACAW Security, 8 June 2025. https://www.macawsecurity.com/blog/why-reactive-security-has-reached-its-limits-the-emerging-threat-landscape-in-agentic-ai
Klover.ai summary of Mustafa Suleyman’s The Coming Wave. https://www.klover.ai/the-coming-wave-ai-containment-mustafa-suleymans-risk-framework/
Marc Andreessen, “The Techno-Optimist Manifesto.” https://a16z.com/the-techno-optimist-manifesto/
R Street Institute, “Mapping the Open-Source AI Debate: Cybersecurity Implications.” https://www.rstreet.org/?p=85817&post_type=research
Balaji Srinivasan, TOKEN2049 talk: “Network States of the Internet.” https://www.youtube.com/watch?v=cTacROXRbwQ
Competitive Enterprise Institute, “We Need to Avoid a ‘Ready, Fire, Aim!’ Approach to AI Regulation.” https://cei.org/opeds_articles/we-need-to-avoid-a-ready-fire-aim-approach-to-ai-regulation/
OECD, Emerging Privacy-Enhancing Technologies (2024). https://www.oecd.org/en/publications/emerging-privacy-enhancing-technologies_bf121be4-en.html