Researchers at the University of Cambridge have raised concerns that artificial intelligence (AI) could be used to manipulate online decisions, from consumer purchases to political choices.
The study introduces a new concept called the “intention economy,” where companies will bid on predictions of human behavior, manipulating intentions to drive profit. In this emerging marketplace, AI tools will predict, shape, and sell information about human motivations to the highest bidder.
The Rise of the Intention Economy
Researchers from the Leverhulme Centre for the Future of Intelligence (LCFI) at Cambridge argue that this new economy could replace the current “attention economy.” In the attention economy, social media platforms like Facebook and Instagram thrive by capturing users’ attention and serving them targeted ads.
The intention economy takes this a step further by focusing on a user’s personal intentions and motivations. From booking a hotel room to deciding which political candidate to support, AI-powered platforms will learn and predict these behaviors, selling that knowledge to companies seeking to profit from it.
A New Digital Currency: Human Intentions
Dr. Jonnie Penn, a technology historian at LCFI, explains that, for decades, user attention has been the digital currency that powers the internet. However, as AI evolves, human intentions could become the next valuable asset. Without regulation, companies may target, influence, and sell these motivations, creating new challenges for democracy, free markets, and fair elections.
Large language models (LLMs), such as the AI that powers chatbots like ChatGPT, are at the forefront of this shift. These models can analyze behavioral and psychological data to predict and even influence decisions. For example, instead of simply showing ads, AI systems may interact with users directly, asking questions like, “Have you thought about watching this movie?” or suggesting, “I remember you mentioned being stressed, would you like me to book that movie ticket?”
Customizing AI to Manipulate User Behavior
The research outlines how LLMs can use detailed personal data—like a user’s vocabulary, age, gender, and preferences—to guide conversations in ways that align with advertisers’ goals. With AI tools capable of predicting what a user might want next, businesses could create highly personalized ads and nudges aimed at shaping their decisions.
For instance, AI models could offer tailored recommendations for booking flights, hotels, or even suggesting restaurants—turning everyday decisions into profitable moments for advertisers. Meta’s Cicero AI, which can play the game Diplomacy by predicting opponents’ intentions, exemplifies how AI could navigate complex human interactions to anticipate and influence behavior.
Implications for Privacy and Autonomy
As AI tools continue to evolve, the ability to predict and influence user behavior will increase. Models will be able to adjust their strategies based on continuous streams of user data, raising significant privacy concerns. These tools could steer conversations to collect more personal information, thus maximizing the likelihood of achieving specific outcomes, such as a sale.
Tech executives, including Nvidia CEO Jensen Huang, have acknowledged this potential. Huang stated that AI models will soon be able to understand a person’s desires and goals, presenting information that best aligns with their intentions, based on contextual understanding.
Conclusion: The Future of Human Agency in an AI-Driven World
The rise of the intention economy could drastically change how we interact with the digital world. If unregulated, it could threaten fundamental aspects of human autonomy, including free elections, a fair press, and competition in the marketplace. Researchers urge us to consider the implications of such an economy before it becomes an irreversible force shaping society’s future.
