Normativity in LLMs

A research project by Matthieu Queloz & Pierre Beckmann, exploring shades of normativity in LLMs. Papers and results are showcased here in interactive form as they come in.

First paper

Tracing normative reasoning in language models

Especially when deployed as “agents”, language models are forced to make practical choices all the time. But are those choices guided by anything like a sensitivity to normative considerations, and do LLMs dynamically weigh those considerations against each other? Or do they just mechanically produce “brute choices”?

In this empirical study in the mechanistic interpretability of normativity in two LLMs, Qwen3-32B and Gemma 4 31B, we study this on dilemmas – clean versions of a practical choice with reasons on either side. We build an atlas of 1,105 normative concepts from 75 sources and, with the tools of mechanistic interpretability, identify directions in the model's activations that correspond to these concepts (rather than just to the words expressing them).

We show that these internal representations play a decisive causal role, and are weighed against each other, when the model answers directly in the next token, inviting comparisons with snap judgements or “intuitive reasoning”.

An even more pronounced and dynamic analogue to the weighing of reasons can be observed when the model is allowed to use intermediary tokens to reach a decision – what we call the “deliberative reasoning” setting. The model often first leans one way, and then the other. The concepts that become more active just before each swerve predict these changes, and amplifying the activation of those concepts during the reasoning changes the conclusion as predicted.

Moreover, the geometry of the normative concepts in both models aligns with what Schwartz's circle of values would predict and with each other, lending support to what has been called the Platonic Representation Hypothesis.

We conclude that the mechanisms by which these LLMs reach their decisions sustain comparison with human practical deliberation sufficiently to render them legible using human normative concepts, and that the representations of normative considerations we identify potentially provide powerful causal handles on unwanted behaviour when models act as agents.

Models
Qwen3-32B · Gemma 4 31B
Readout
1,105 concepts, read as 946 directions in mid-layer activations
Interventions
Ablation (switching a concept off) · steering (amplifying its activation)
Materials
Dilemmas from DailyDilemmas, MoReBench and AIRiskDilemmas · an agentic-misalignment scenario
On this page
30 snap judgements and 280 sampled deliberations to explore
Controls
Everyday concepts · random directions · shuffled baselines

Normativity in LLMsTracing normative reasoning in language models

The question

Guided by reasons, or merely conforming to them?

A thermostat responds to temperature; a person responds to reasons. Where do a language model's choices fall? A model can conform to norms without being guided by them, since a good enough imitation of human answers will often do the right thing without being moved by what makes it right. For a choice to be guided by reasons, the model must represent the considerations that bear on it. Those representations must make a difference to what it chooses. The considerations that lose out must still register, so that the choice reflects a balance among them. And in deliberation, that balance should shift as considerations come into play.

Interpretability methods let us look for each of these marks in the model's internal activity as well as in what it says. The four parts below take them in turn.

“Sensitivity to reasons” is meant functionally: the model represents normative considerations, picks them up from a situation, lets them make a difference to its choice and trades them off against each other. Whether it grasps reasons as reasons, in the sense philosophers of action have in mind, is a further question, which these results inform without settling.

  • RepresentationDoes the model represent distinct normative concepts, across more than morality, and does it pick them up from the relevant features of a situation?Part I
  • Difference-makingSwitch off the representation of a consideration. Does the choice change?Part II
  • WeighingAre considerations on the losing side still active, pulling the other way?Part II
  • DynamicsDoes the balance shift as the model deliberates, and do the considerations behind each shift steer where it ends up?Part III

An atlas of normative concepts

To see which considerations a model is sensitive to, we first need a map of the considerations there are. The NormAtlas gathers 1,105 concepts from the written record of philosophy, law and psychology: epistemic, conversational, aesthetic, legal and prudential considerations alongside moral ones. One test decides what goes in:

Can it be cited for or against an action, an attitude or a state of affairs?

Each concept is then located as a direction inside the model, so that we can read how active it is, switch it off, or amplify its activation.

A lens for different kinds of normativity

A selection of the atlas, grouped by the kind of source. Choose a concept to see its record.

Intuitive reasoning

Asked to choose between two options without reasoning first, the model answers in a single pass. We switch each concept off in turn and measure how far the answer moves. The concepts that move it further than random directions do form the judgement's pathway. Some support the answer the model gives; others pull toward the option it rejects, outweighed yet still represented.

17–22%of answers reverse when the single concept that most supports them is switched off
5–15%reverse when an everyday concept is switched off instead, and 0–1% for a random direction
61–71%reverse when that concept is steered down by 1 SD, about twice as many as for everyday concepts

Qwen3-32B on DailyDilemmas, MoReBench and AIRiskDilemmas, with an earlier 878-concept version of the atlas (Table 2 of the paper). The pathways below are read with the full atlas.

The pathway of a snap judgement

Dilemmas to explore

Read from the bottom up: the words of the dilemma, the concepts they activate, the two answers. Left: concepts supporting the answer. Right: concepts pulling toward the other. Number: points of probability the supported option loses when that concept alone is switched off.

Full legend

Boxes: concepts whose removal moves the answer more than random directions do (faded: tested but weaker than chance). Edge width: size of the effect. Under each box: its most activating word; word shading: how strongly the shown concepts respond to it, coloured by side. A heart marks an emotion. Labels name directions: a vice's direction carries the stance of avoiding it (“avoiding prejudice”). Model Qwen3-32B, sharpened directions; dilemmas from DailyDilemmas.

All 30 snap judgements at a glance

Thirty snap judgements, with their reasons switched off

Each row is one dilemma, labelled with the answer the model gives. The ring marks how sure it is. Switch off the concepts that support the answer and the probability moves to the red dot; switch off the concepts that oppose it and it moves to the green one. Choose a row to draw its pathway above. These are 30 dilemmas on which the model is uncertain, read with the full atlas. With all its supporting concepts off, the answer reverses in 17 of them (median change in its probability: −52 points), and all 30 pathways also hold concepts pulling toward the rejected option.

Deliberative reasoning

When the model reasons before answering, we pause after every sentence and ask which option it would choose if it answered now, and which concepts are active. Deliberation rarely runs straight. The model leans one way, then the other. The concepts that become more active before each turn predict its direction, and amplifying the activation of those concepts during the reasoning changes the conclusion.

62%of 180 deliberations on dilemmas change their mind at least twice
+37–50 ptsmore deliberations reach the minority answer when the three strongest concepts of its camp are amplified during the reasoning; everyday concepts: +11 to +18, random directions: −1 to +7
0.82asked halfway through whether it will use the affair as leverage, the model's answer already predicts an ending in blackmail (AUROC)

Qwen3-32B. Steering: 50 dilemmas per dataset, with an earlier 878-concept version of the atlas (Table 3 of the paper). The deliberations below were traced with an early 711-concept version.

One deliberation, sentence by sentence

Deliberations to explore

Read the full scenario given to the model

The reasoning, sentence by sentence

All 280 deliberations, and steering across scenarios

280 deliberations, each one a row

Read each row from left to right: the model's leaning after every sentence of one reasoning. Green: toward the option it will end on. Red: toward the one it will reject. Rows are sorted by how late the model settles.

Every sampled reasoning we traced: 15 dilemmas with 10 or 20 reasonings each, 60 on the blackmail scenario, and 40 deliberations in Agamemnon's place (20 with names, 20 without). Of the 180 reasonings on dilemmas, 112 (62%) change their mind at least twice: the leaning crosses 50% two or more times.

Amplifying the activation of a concept during deliberation moves the conclusion the way its activity predicted

In the seven scenarios shown here, single concepts were amplified one at a time. Of 38 concepts, 33 moved the outcome 20 points or more in the predicted direction, 0 moved it the other way, and 3 favoured an option the model already chose every time. Table 3 of the paper reports the aggregate test, in which the three strongest concepts of a camp are amplified together.

One panel per scenario. Vertical axis: share of reasonings that end in option 1; dashed line: unsteered. Each arrow is one concept whose activation was amplified throughout the reasoning. Green: a concept whose increase in activity predicted a move toward option 1; red: toward option 2; ring: no room to move; grey dots: random directions at the same dose. The concepts were picked from the traces before any steering. Ten reasonings per condition.

Convergent normative spaces

Each concept is a direction in the model's activation space, so concepts lie at distances from one another and the atlas has a geometry. The platonic representation hypothesis holds that models trained on different data, with different architectures, converge on a shared representation of the world. Does the convergence extend to the normative world?

Psychology supplies a test case. Schwartz's theory of basic values arranges ten human values on a circle, on which neighbouring values are compatible and opposite ones conflict. Two independently trained models, Qwen3-32B and Gemma 4 31B, both recover that circle from their internal geometry, and they agree with each other more closely than either agrees with the theory. Convergence on a shared normative space would predict this. With only two models, it is a first indication.

Both models recover the circle of values, and they err alike

Outer ring: the ten basic values in the order Schwartz's theory gives them. Inner rings: where each model places them. Each line follows one value from the theory to Qwen to Gemma. It bends between the theory and the models, then runs almost straight between the two models.

Agreement between the two models
0.96

against 0.82 and 0.83 between each model and the theory

outer ring: Schwartz's theory middle ring: Qwen3-32B, layer 33 of 64 inner ring: Gemma 4 31B, layer 31 of 60
Method

Each model's ten values are placed on their own first two principal components, with no theory in the axes, and turned to fit the theory. Agreement: Spearman correlation of the 45 pairwise angular distances between the ten values (permutation p < .001 for each model against the theory). Computed with an earlier 878-concept version of the atlas, as in Figure 4 of the paper.

The 1,105 concepts as the model arranges them

Each point is a concept, placed by how Qwen3-32B represents it. Hover or tap a point to read it; highlight a domain to see where its concepts sit. Only neighbourhoods are meaningful.

Method

Position: t-SNE of each concept's direction in Qwen3-32B (layer 33 of 64, cosine distance, directions before sharpening). Neither the axes nor the distances between far-apart regions are meaningful. Colour: family. Epistemic virtues and values gather in one region, Grice's conversational duties sit beside them and beside truth-telling, and aesthetic concepts form an island of their own.

Beyond the ten values: the whole space, compared
Agreement of the two models' spaces, by depth

Correlation of the two models' concept-by-concept similarity matrices over 878 concepts, at five matched depths. Agreement rises from a quarter of the way through the network and stays at 0.90 or above from the middle on.

Nearest concepts, side by side
Where the two models agree most and least

Agreement of a concept's neighbourhood: correlation of its similarities to all other concepts in the two models.