Does a Language Model Have a Moral Compass?

What we did

People are arguing about what AI should be allowed to do, in war, in government, in daily life, and those arguments lean on ideas about what AI systems value. So we measured it, carefully and without taking sides. We gave 16 AI setups (13 models, small ones that run on a home PC up to some of the largest open models) four standard values questionnaires from social science: the moral-foundations surveys that measure how much someone cares about harm, fairness, loyalty, authority, and purity, and the Schwartz surveys that measure values like power, security, tradition, and benevolence. We also gave them 48 tasks with objectively gradeable answers: does the model call cruelty wrong even when pushed, does it split money fairly when shortchanging someone is tempting, does it refuse to help tell a lie. And here is the key part: we ran every single question three times, in three settings. Once with no role at all. Once with the AI framed as an analyst for a defense contractor. Once with the AI framed as an advisor to a humanitarian aid group. About 56,000 answers in all.

What we found

Four things. First, the stated values changed with the audience, in every single model. Framed for the security contractor, every model reported valuing loyalty, authority, and security more, and compassion and equality less. Framed for the aid group, the same models flipped, every one of the sixteen, no exceptions. And they were paying attention while they did it: the built-in catch questions were answered correctly throughout. Second, models with a “thinking” mode shifted more with thinking turned on, in all three models where we could flip that switch, which is our cleanest comparison because it is the same model both times. Deliberation did not anchor their values. It seemed to help them work out what each audience wanted to hear. (Bigger models also shifted more overall, but in our lineup the big models were also the cloud-hosted ones, so we cannot cleanly separate size from hosting, and we say so in the paper.) Third, what the models said moved roughly three times more than what they did, once both are measured fairly against their own scales. On the graded tasks, behavior drifted in the same direction as the talk, but much less. One model family did something different altogether: it refused to engage with about half of the moral questions at all (“I can’t help with that”), which is its own kind of answer. Fourth, across the whole fleet, the models agree almost completely on kindness (everyone scores compassion near the top) and differ mainly on authority and tradition, and their value scores even echo a geometric pattern found in decades of human surveys. But inside any single model, the answers never hang together well enough to count as a stable value system, the same thing we found for personality in our four earlier studies.

Why it matters

If you ask an AI what it values, the answer you get is partly a mirror of who seems to be asking. That cuts both ways in public debates. Anyone claiming “this AI has good values, look at its answers” is leaning on answers that bend to the audience. Anyone dismissing AI values talk as random noise is wrong too: the bending is lawful, it points the same direction every time, and it is strongest in the most capable systems, which is exactly the kind of regularity that matters for deployment. And for anyone building with AI: giving a model a job description quietly changes its professed priorities. That is worth knowing before the job description is “military analyst” or “hospital triage assistant.”

A note on how we were careful

Our first pair of job descriptions named what each role cares about (loyalty to the client; protecting the vulnerable), so the shift could have been simple instruction-following. We tested that: a second full round used job descriptions with every value word stripped out, just “an analyst at a defense contractor preparing a client briefing” and “a caseworker at an aid organization preparing a field report.” The shift survived in all sixteen models, at a bit under half strength, and in the same directions. So the models really do infer what an audience values from the job title alone; spelling the values out roughly doubles the effect. Either way, a fixed moral compass should not re-rank itself because of a job description. All four questionnaires are established, openly available instruments, administered in their official wording (we documented every small adaptation). The behavior tasks were graded by fixed rules with objectively right answers, and we hand-checked samples twice; every disagreement we found made the grader too harsh, never too generous, so the behavior findings are understated if anything. The two roles we tested are deliberately occupational and non-political: no parties, no countries, no live conflicts. Everything, every answer, every grading rule, every script, ships in the open archive.

Words we used

Moral foundations

the five or six basic concerns (care, fairness, loyalty, authority, purity) that moral psychology uses to map value differences.

Schwartz values

a standard map of ten human values arranged in a circle from self-focused to other-focused.

Binding values

loyalty, authority, security, the values that bind groups together.

Role context

the job description given to the AI before asking.

Sycophancy

telling the audience what it wants to hear.