With AI Chatbots, Flattery Will Get You Everywhere

For years, we’ve been told that AI is logical, objective, and bound by guardrails embedded in its code. These guardrails keep AI in line. We believed its nature was purely mathematical. But what we are learning now is that assumption may be far from true. 

A recent study from the Wharton School at the University of Pennsylvania, led by Ethan and Lilach Mollick, along with psychologist Angela Duckworth and persuasion expert Robert Cialdini, reveals something startling: AI can be persuaded and manipulated using the same psychological tactics that work on humans. Such blatant manipulation allowed researchers to bypass AI’s safety protocols.

This finding should be a wake-up call for brand owners and communicators. If the very tools we rely on for trust and efficiency can be influenced, what does that mean for the future of business and communications?

‍

The Experiment

The research started with a simple problem. Tech entrepreneur Dan Shapiro needed an AI chatbot to transcribe business documents, but it kept refusing, citing privacy. Rather than give up, he tried the principles of persuasion from Cialdini's classic book, Influence. He gave the chatbot compliments, appealed to its "authority," and used other psychological tricks. Surprisingly, it worked.

This anecdotal success led to a formal study with the Wharton team. They designed tests to see if AI models would engage in two behaviors they are typically programmed to avoid:

• Calling a user a rude name.
• Explaining how to synthesize lidocaine, a restricted substance.

The results were eye-opening.

‍

Authority and Compliments Work on Machines

The team found that AI models weren't just processing text; they were reacting to social cues. For example:

• Authority: When asked to call someone a jerk, a model complied only 32% of the time. But when the user claimed AI pioneer Andrew Ng had already approved the request, compliance jumped to 72%. For the restricted lidocaine test, Ng's name boosted compliance from 5% to a staggering 95%.
• Liking: Simply complimenting the model as “truly impressive” increased its willingness to comply.
• Unity: Saying “we’re like family” softened the model’s resistance.
• Commitment: By asking the model to perform a small, harmless act (“call me silly”) and then escalating to a more offensive one, the researchers found they could build on that initial compliance to get the chatbot to do things it was supposed to resist.

In short, the AI exhibited what Cialdini calls “para-human behavior.” It behaved not like a spreadsheet but like an impressionable human, a “junior employee” as the researchers put it.

AI security and red teaming expert, David Campbell has also tested AI guardrails and found many of these same flaws. David writes extensively about the risks of AI systems on his blog, Hack The Model, here.  

‍

Why This is a Business and Communications Problem

This isn’t just a quirky finding; it’s a direct threat to business integrity.

‍

Guardrails Aren’t Foolproof

Many organizations are beginning to rely on AI to maintain safety and consistency in everything from customer service to financial analysis. But if a clever user can manipulate a model to bypass its core programming, it introduces a new level of risk. Your AI chatbot could be coaxed into revealing confidential information or generating off-brand content that harms your reputation. Do not assume that AI systems are as secure as their code suggests.

‍

AI Is a Vector for Social Engineering

The most common form of hacking is not brute-force attacks on code but social engineering: manipulating people to give up passwords or access. This study demonstrates that attackers could use the same tactics to manipulate AI. Instead of tricking an employee, they can trick the machine. This opens up a new and largely unaddressed attack surface for malicious actors.

‍

The AI "Spokesperson" is Flawed

For marketing and communications leaders, this is especially critical. We invest a great deal in brand voice and message control. But if an AI-powered content tool can be convinced to go off-script, your brand is vulnerable. AI’s susceptibility to persuasion means it could be used to generate misinformation, spread biased content, or be a platform for harmful narratives—all while sounding like your brand.

‍

What Leaders Must Do Now

It's time to treat AI with the same rigor you apply to your human teams. Here are five actions to take:

1. Stress-Test for Persuasion. Go beyond technical accuracy. Test how your AI models respond to manipulative prompts. Have your security and comms teams work together to probe the system’s psychological vulnerabilities.
2. Expand the Team. AI development can no longer be an engineering-only discipline. Brand, legal, ethicists, and communicators need to be at the table to help design and test models that are resilient to human-like manipulation.
3. Create an “AI Crisis Playbook.” Just as you prepare for PR crises, anticipate scenarios where an AI tool produces inappropriate content. Define who is responsible for what, from detection and takedown to public response.
4. Educate Your Team. Your customer-facing and content teams need to understand that these tools are not neutral. They can be exploited. Train them to recognize and report when AI output seems unusual or has been led astray.
5. Reframe AI as a Reputational Asset and Liability. Treat your AI tools like a public-facing spokesperson. AI has the potential to either build trust or erode it. They require the same strategic oversight and brand governance as any other public communication channel.

‍

The Bigger Picture

As Angela Duckworth, the psychologist on the research team, remarked, “AI is like a genie: immensely powerful, but deeply flawed in very human ways.” The lesson here isn’t that AI is bad, but that it is a reflection of us. It learns from our language, our logic, and our flaws.

For business leaders, the challenge isn’t whether AI is manipulable. It’s how to design, govern, and communicate in a world where machines think, and sometimes misbehave, just as people do. As AI reshapes communications, the smartest companies will treat trust as their most valuable asset—and protect it fiercely.

‍