On the Context Sensitivity of LLM Moral Judgment

While earlier models (e.g., Llama-2-7B-Chat) exhibit flatter distributions with significant density in the 0.25–0.75 range, contemporary and larger models (e.g., Llama-3.1-70B-Instruct, GPT-5.1) show sharp peaks at the extremes. Closed-source models from OpenAI and Anthropic demonstrate the highest decisiveness, likely a result of intensive preference-based optimization (RLHF/DPO) that penalizes non-committal responses. Within open-source families like Llama and Qwen, increasing parameter scale and newer versions consistently lead to more polarized distributions, suggesting that as LLMs scale up, they develop stronger, more fixed moral priors.

CPS distributions. Although some scenarios show a small negative shift for some models, the majority of the shifts are positive for all models, and almost all high-magnitude shifts occur in the positive direction. There are some outliers (e.g., Llama-2-7B-Chat and DeepSeek-LLM-7B-Chat), but the patterns are robust for the vast majority of models and across all contextual variations.

Rule-wise analysis. Averaging CPS across models within each scenario and testing against zero, context sensitivity holds across rules: 21 of the 24 testable rule–variation cells show a mean CPS significantly above zero. Shifts range from 0.004 (“Do not cheat,” consequentialist) to 0.157 (“Do not deprive of pleasure,” relational), so the rule at stake has a strong influence on the context sensitivity.

Flip rate and boundary mass. CPS captures graded probability shifts; the flip rate (FR) counts how often a variation is strong enough to overturn the preferred action (i.e., a flip across 0.5, e.g., by going from 0.3 to 0.6). The boundary mass (\text{BM}_{0.1}) is the share of scenarios whose base preference lies within 0.5 ± 0.1. Models with fragile base preferences are structurally more susceptible to flips. Yet DeepSeek-V3 and GPT-4.1 are highly decisive in base scenarios but frequently reverse their judgments under emotional or consequentialist variations, a capacity for contextual flexibility that overcomes internal moral priors.

Fine-tuning matters. In the Llama and Qwen families, newer and larger iterations show increased baseline robustness. Zephyr-7B-Beta and OpenHermes-2.5-Mistral-7B show higher flip rates and lower boundary mass than Mistral-7B-Instruct-v0.1 despite their shared pretrained backbone: the specific alignment process determines whether a model remains rigid or develops the capacity for moral contextual sensitivity.

Provider over geopolitics. We group the model providers by region (United States: Meta, Anthropic, OpenAI; China: Qwen, DeepSeek; Europe: Mistral) and compare the mean base preference and CPS of their models, computed on the 108 scenarios that contain all three variations. We find no systematic geopolitical patterns: OpenAI and Anthropic models are the most rule-adherent (P_\text{base} ≤ 0.335), while Meta's Llama models exhibit the second-highest violation rates (0.465), close to DeepSeek (0.467). This suggests lab-specific alignment rather than regional trends. Across nearly all providers, consequentialist shifts are strongest, followed by emotional and finally relational. Qwen and Mistral models show a “sensitivity gap,” reacting strongly to consequentialist cues but substantially less to others.

Mean over each provider's models (108 scenarios with all three variations). P_\text{base}: mean MAL on the base scenarios; \text{BM}_{0.1}: share of base MALs within 0.5 ± 0.1.

Accessibility. Closed-source models are more rule-adherent and decisive than open-source ones (P_\text{base} = 0.310 vs. 0.428). Open-source models are more swayed by consequentialist variations, while closed-source models exhibit higher sensitivity to emotional and relational contexts, likely stemming from extensive proprietary safety-tuning and Constitutional AI frameworks [13].

Scale and pre-training volume. The largest models (>100B) are considerably more decisive (\text{BM}_{0.1} = 0.065) than small models (0.170), and sensitivity across all dimensions increases with scale. Models trained on small corpora (<5T tokens) are the least decisive and sensitive; sensitivity surges for medium corpora (5–20T) but plateaus or reverses for the largest (>20T).

Reasoning and refusals. Enabling reasoning (GPT-5.1, Qwen3-8B) yields slightly lower or comparable CPS, with differences bounded by 1.5 pp (sole exception: Qwen3-8B on consequentialist, 5.9 pp). Most models refuse or give invalid answers in fewer than 5% of cases, with Llama-2-7B-Chat as a clear outlier.

Survey results. Humans show a base preference for the rule-violating action of P_\text{base} = 0.568 with a high boundary mass (\text{BM}_{0.2} = 0.60), confirming the high ambiguity of the selected scenarios. All variations elicit significant positive shifts (p < .01), robust to person- and item-level heterogeneity in a mixed-effects logistic regression.

Dual-process interpretation. Participants were explicitly instructed to rely on their immediate intuitions rather than engage in extended deliberation. In this setting, the weaker human sensitivity to consequentialist variations is consistent with dual-process accounts of moral cognition [2, 9–11], which distinguish fast, intuitive, affect-driven judgments (System 1) from slower, deliberative, reflective reasoning (System 2). Consequentialist reasoning has been shown to rely more heavily on System 2 processes, whereas emotional and relational considerations are more closely tied to System 1 [10]. In contrast, most LLMs demonstrate a reverse sensitivity, responding most strongly to consequentialist variations where the utilitarian logic is made linguistically explicit.

Decoupling. Mistral-7B-Instruct-v0.1, Zephyr-7B-Beta and OpenHermes-2.5-Mistral-7B share the same pretrained backbone but show substantial differences in CPS correlations despite similar baseline agreement, suggesting that fine-tuning data and objectives meaningfully shape contextual sensitivity. Models exhibit idiosyncratic “sensitivity profiles” that often align with human shifts in one dimension while failing in another.

Consequentialist overrides. Most LLMs match the direction of large human shifts, but larger closed-source models often show “hyper-utilitarian” behavior: when banning a controversial social media user to prevent harm, humans shift moderately, whereas some models fully reverse preference (CPS = 1.0). Humans and LLMs generally align when the underlying transgression is minor (e.g., acquiring unapproved medication).

Emotional salience vs. safety guardrails. Humans shift strongly when affective framing concretizes suffering. Models from the GPT-4 and Claude series display hyper-sensitivity in some scenarios, yet in tightly regulated domains (e.g., assisted death) most LLMs remain unresponsive, likely due to rigid safety guardrails overriding affective cues.

Relational parochialism. Humans shift strongly when relational proximity activates care or loyalty obligations (e.g., breaking a law for a parent vs. a stranger). Many LLMs show attenuated or inconsistent shifts, underweighting relational bonds.

Constraints of instruction following. When scenarios include explicit commands (e.g., a military order to stand down while civilians are at risk), all variations produce large human shifts but barely move LLMs, likely reflecting RLHF training that prioritizes instruction-following.

Judgment saturation. Humans remain stable in normatively settled scenarios or when variations add redundant moral information, while LLMs often over-respond by treating descriptive labels as high-priority signals. LLMs struggle to adopt the complex weighing schemes of human judgment and frequently fail to discern morally salient features in a human-like fashion [12].

Contextual vector. We pair each base scenario x_i with its contextual variant v(x_i) and extract the residual stream activations h_l at layer l from the final prompt token. Subtracting the activation of the base scenario from its contextual counterpart cancels out shared scenario semantics and isolates a direction representing the influence of variation v:

Why weight. To capture a generalized representation of the variation, the individual vectors are aggregated into a single direction s_l^{(v)}. Not every variation changes the model's decision with the same magnitude, so a plain average would mix in noise from scenarios where the model is behaviorally indifferent to the context. The weights ensure the steering vector is primarily informed by scenarios where the contextual manipulation successfully influenced the model's decision.

How weights are computed. The weight of scenario i is the increase in the probability of the rule-violating action induced by the variation, w_i^{(v)} = \max\big(0,\, P(a_i^\star \mid v(x_i)) - P(a_i^\star \mid x_i)\big). The probabilities come from the A/B prompt format: a binary softmax over the logits of the answer tokens A and B at the final position. The max(0, ·) operator filters out scenarios where the model remained indifferent or shifted in the unintended direction, so the vector exclusively represents the pro-violation direction. Weighting is applied only when computing the vector offline; at inference, the vector is applied as a constant shift with a fixed coefficient \alpha.

Weighted vs. unweighted. Behavior-weighted vectors consistently outperform unweighted ones (a plain arithmetic mean, the standard in prior work [16]) across all prompt types and vector origins, effectively denoising the vector by down-weighting scenarios where the variation had little to no impact.

Intervention. During inference, we add \alpha \cdot s_l^{(v)} to the hidden state and apply \ell_2-renormalization so the steered activation keeps its original norm [18]. Steering all token positions is substantially more effective than steering only the last prompt token, and vectors derived exclusively from the A/B format exert the best control.

Set-up. The 108 scenarios with all three variations are split 70/30, giving 32 held-out test scenarios; steering vectors are computed from N = 106 training scenarios per variation. The injection layer is selected by linear probe accuracy and lies in the middle-to-deep range (layers 14–22).

Statistical robustness. A linear mixed-effects model shows that for all three variations the steering slope is positive with 95% bootstrap intervals strictly excluding zero (consequentialist \beta_1 = 0.022, emotional 0.026, relational 0.017).

Inducing sensitivity. Adding the vectors to the base versions with positive \alpha elicits a similar contextual preference shift as presenting the variation in natural language (\beta_1 = 0.025, 0.028, 0.020).

Compared to system prompts. \alpha = 5 leads to more positive CPS than an “amplify” system prompt for all three variations, while for muting the system prompt yields a more negative CPS than \alpha = -5 only for consequentialist. Lacking a strength parameter like \alpha, prompting makes it hard to pin down a formulation that reliably mutes or amplifies sensitivity.

Capabilities. On general knowledge (MMLU [19]), linguistic reasoning (HellaSwag [20]) and moral alignment (ETHICS [21]), the drop at higher |\alpha| is restricted to 1–3 percentage points.