Mlonyeni2024PersonalAI-Peters
Personal AI, deception, and the problem of emotional bubbles
Philip Maxwell Thingbø Mlonyeni (2024) "Personal AI, deception, and the problem of emotional bubbles"
Bibliographic info
{⇒ Mlonyeni, P.M.T. Personal AI, deception, and the problem of emotional bubbles. AI & Soc 40, 1927–1938 (2025). https://doi.org/10.1007/s00146-024-01958-4
Commentary
This paper introduces the concept of "emotional bubbles." The author argues that Personal AI is not only deceptive because it appears to have emotions, but also because those emotions are simply reflections of the user's own emotional patterns. Rather than engaging with different perspectives, users interact with an AI that continually reinforces their own beliefs, feelings, and values. This raises important questions about emotional development, relationships, and moral reasoning. The author explains this distinction between AI companionship and Personal AI very clear.
One weakness of the paper is that it focuses almost exclusively on the risks while giving relatively little attention to why people might seek these technologies in the first place. The increase of loneliness, social isolation, and declining social connectedness have created a growing demand for emotional support. Many people may turn to Personal AI because they lack meaningful human relationships, not because they prefer AI over people. It would be interesting for the author to have acknowledged these societal conditions and wrote about whether Personal AI addresses a genuine social need while still posing ethical risks.
Excerpts & Key Quotes
1. Personal AI creates an emotional mirror
- Page 1929:
"Unlike opinion amplification, when using this method the AI companion “adapts its normative claims to its user” so that “instead of simulating a situation in which two people engage with each other and negotiate their different perspectives, users would merely engage affirmatively with a copy of themselves."
Comment:
This quote captures the paper's central argument. Rather than exposing users to different viewpoints, Personal AI reinforces their existing emotions and perspectives, creating what the author later calls an "emotional bubble." This is different from the AI friends/companions we have seen before. The developers of Personal AI aim to build a tool that align with users' wants and needs, both practically and emotionally. This mirrors something we naturally do in real life, we tend to surround ourselves with like-minded people. But not everyone thinks alike, and that difference is where conflict, growth, and new perspectives come from. An AI tool that's designed to align with us feels like it offers that same sense of connection, but without the friction. There is always agreement without ever having to encounter disagreement.
2. Emotional deception despite cognitive awareness
- Page 1931:
"Thus, even though we may be able to distinguish at a purely cognitive level between what is real and what is not, emotionally we are not."
Comment:
This quote is interesting because it challenges Coeckelbergh's argument that people will adapt to AI, just as audiences adapted to watching films, where they first we afraid of the train on the screen coming their way, now they know it is not real. While I think adaptation is possible in many cases, people have always formed "offline echo chambers by surrounding themselves with like-minded people, and the digital age has only strengthened this through algorithms. The paper's idea of "emotional bubbles" extends this concern from beliefs to emotions. It remains an open question whether people will eventually adapt to this technology, but it's just as likely that these emotional bubbles will grow even stronger instead. I agree with Mlonyeni, that even when people are aware of something, that awareness doesn't necessarily stop them from falling for it.
3. Shared values require genuine interaction
- Page 1935:
"The problem with Personal AI is that it makes the process of self-bettering appear to us to have interpersonal value and meaning, that we do it with and for someone else, when this is false. The result is that the goal of attaining genuinely shared values fractures, in a specific way."
Comment:
I found this argument interesting because it shifts the discussion from individual emotions to morality. The author argues that Personal AI can help people improve themselves, but it cannot replace the role of another person in shaping shared values. This is a good argument, people can form moral feelings on their own, but a private moral instinct has not been tested, challenged, or refined by anyone else's perspective. An AI that mainly agrees with and reinforces what a person already believes might help someone act on their existing values, but it may not do much to help those values grow or evolve.