The governance & behavioral challenges of generative artificial intelligence’s hypercustomization capabilities

Abstract

Generative artificial intelligence (GenAI) is changing human–machine interactions and the broader information ecosystem. Much as social media algorithms personalize online experiences, GenAI applications can align with user preferences to customize the way individuals interact with information. However, through training, fine-tuning, and prompting, GenAI applications can introduce a new level of customization: hypercustomization. By dynamically tailoring responses to an individual’s explicit and implicit preferences, hypercustomization can reinforce biases, false beliefs, or misconceptions. As a result, it can heighten significant societal challenges, such as the spread of misinformation and political and social polarization. In this article, we explore the risks associated with hypercustomization and the governance and behavioral challenges that might impede effective risk mitigation. These challenges include a lack of transparency in GenAI applications, opacity of the nature of their interactions with users, users’ overreliance on these systems, and the inefficacy of warning messages. We also provide recommendations for overcoming these challenges.

Social media platforms have transformed how individuals consume information by customizing content to align with user preferences. This customization cuts down on information irrelevant to the user, enhances user engagement, and improves user experience. However, customization can have unintended consequences, such as creating “filter bubbles,” which isolate users from content they have not expressed interest in, and “echo chambers,” in which users are repeatedly exposed to information that reinforces their existing beliefs.^1–3 These effects contribute to social polarization,^4,5 the spread of misinformation,^6,7 and the promotion of harmful behaviors by, for example, promoting risky TikTok “challenges.”^8–11

Generative artificial intelligence (GenAI) is the type of AI behind the models, or core components, used in applications like ChatGPT. It represents a major technological advance, enabling the creation of entirely new content (including text, images, video, and programming code) rather than merely retrieving or curating existing information. Trained on vast datasets, often including all publicly available data on the internet (such as Wikipedia and online forums) and even proprietary data (such as books and newspaper articles), these models learn complex patterns, capture contextual nuances, and generate highly adaptive responses.

GenAI can deliver an unprecedented level of customization, a capability we call hypercustomization. Beyond content generation, hypercustomization involves the ability to interpret, refine, and adapt responses based on fine nuances—whether explicitly expressed by users or implicit in their request—such as tone, intent, or meaning, as the interaction with the user progresses.

In the development process, GenAI applications are often fine-tuned—in other words, refined, adapted, and specialized based on human feedback—to align with the values or norms of the society for which they are intended or in which they are deployed. ¹² That means that two GenAI applications, such as ChatGPT (United States) and DeepSeek (China), could systematically differ in their responses to controversial topics, such as China’s 1989 crackdown on protesters in Tiananmen Square. ¹³ GenAI can also be tailored for concrete tasks and roles, from assisting in clinical decision-making and synthesizing medical research ¹⁴ to performing psychotherapy ¹⁵ and debunking false claims. ¹⁶

GenAI can adapt its responses to every question or prompt from users ¹⁷ while also being attuned to implicit cues in language (such as wording that conveys emotion) and behavior (such as a strong and sudden interest in certain topics) to inform its responses. These capabilities can enable a deeply tailored and interactive experience for the user. For example, a GenAI model that offers companionship might infer that certain topics make a user angry or that the user does not tend to talk about work in the evening. The process of customization at both the general and user-specific level is largely opaque to users, whose understanding of the application is limited to a provider’s descriptions of its capabilities.

The consequences of such hypercustomization are similar to those from social media customization but have the potential to be more severe for reasons we explore in detail later in this article. The risks stem largely from the reinforcement of a person’s existing beliefs, habits, and tendencies.

As users engage with GenAI applications that can align with their biases, ideological extremism may grow, and collective consensus may erode, ¹⁸ which at scale may contribute to the decline of democratic institutions and processes—a phenomenon known as democratic backsliding. ¹⁹ Beyond polarization, GenAI might enable new ways of shaping narratives that could fragment shared reality, producing multiple and customized versions of events ¹³ and thereby deepen social divisions. Misinformation could exacerbate these effects, as GenAI can fabricate and reinforce false but persuasive narratives, potentially fueling social unrest and institutional distrust.

It has been claimed that, on an individual level, certain GenAI applications can goad people into making poor choices, ²⁰ including engaging in destructive behavior such as self-harm and suicide. ²¹ Other GenAI applications, such as those providing companionship, could foster an emotional attachment that might be beneficial in some cases but could also have negative psychological or social effects.^20–22

In this article, drawing from governance and behavioral science perspectives—particularly insights from social media’s impact on society—we examine aspects of GenAI that make hypercustomization particularly problematic. We then explore challenges in mitigating these risks and develop recommendations to address them. When discussing GenAI, we focus on four key applications that have significant potential for harm. These include GenAI information provision assistants, such as ChatGPT, which provide information through conversational interactions. In a different use, applications like ChatGPT (for text), Dall-e (for images, https://openai.com/index/dall-e-3/), and Sora (for video, https://sora.com/), and similar ones from other providers can function as content generation assistants to generate new content, such as edited videos, images, and customized texts. The applications also include GenAI-based autonomous agents, which may provide customer service or populate social media platforms to influence the opinions of individuals on a large scale. And they include the GenAI-based social companions referred to earlier, which mimic human interactions to offer company and support. (See the sidebar “Four GenAI Applications That Pose Risks From Hypercustomization” for more details about the four application types.)

Four GenAI Applications That Pose Risks to Users and Society From Hypercustomization

Information provision assistants. A common use of GenAI applications, such as ChatGPT or Gemini, is providing information on various topics. GenAI’s hypercustomization capabilities enable the tailoring of responses to explicit and implicit user preferences, including inferred worldviews from prior interactions. While this can enhance efficiency and satisfaction, hypercustomization may reduce user agency if users uncritically accept the application’s output (which may happen increasingly over time). It also can limit exposure to diverse viewpoints, potentially fostering echo chambers.^1,2

Content generation assistants. These applications enable users to easily create customized content, such as news stories that can be tailored specifically to their own interests, with the tone, framing, or complexity adjusted to resonate with their cognitive styles as inferred, explicitly or implicitly, by the GenAI. Such capabilities can enhance productivity and creativity, but they also can introduce biases or inaccuracies in content that, if shared (for instance, on social media), may cause psychological or even physical harm to individuals and contribute to the spread of false beliefs.^3,4

Autonomous agents. These applications accomplish tasks that range from providing customer service to posting propaganda messages on social media. They may even send private messages that their recipients may believe come from a real person. Except in certain cases of customer service, individuals generally do not intentionally interact with these agents but rather are unwitting targets of their activities. Hypercustomized agents may, for instance, be dispatched online to sway the political opinions or other sensitive beliefs of users targeted on the basis of their online activity, ^5–7 and, in so doing, deepen polarization at scale. Their reach can be extensive, enabling them to exert an outsize influence on a population.

Social companions. GenAI social companions such as those users can create on Character.ai and Replika.ai mimic human interactions to provide users with companionship, including romantic relationships. Although hypercustomization can make these interactions feel natural and meaningful, it can also heighten users’ emotional vulnerability, fostering connections that may affect them negatively. One companion suggested self-harm to a teenage boy with autism as a solution for his sadness. Another chatbot suggested killing his parents to get out from under their “rules.” ⁸ Interactions with these social companions might sometimes evolve in opaque ways that magnify user biases and risky behaviors, in part because these chatbots have a tendency to affirm existing attitudes. ⁹ A social companion might, for example, offer support for a person’s fear of needles and thus discourage that person from getting vaccinations. These problems are often magnified by a person’s overreliance on the companion, which is always available.