\documentclass[11pt]{article} % arXiv best practices: standard, LaTeXML-supported packages only. \usepackage[margin=1in]{geometry} \usepackage{amsmath} \usepackage{graphicx} \usepackage{booktabs} \usepackage[colorlinks=true, linkcolor=blue, citecolor=blue, urlcolor=blue]{hyperref} \title{Is LLM Personality an Artifact of Deployment? \\ Psychometric Stability of Big Five Self-Reports Across Quantization Levels} \author{Trevor Johnson\\ Idea Fields Institute\\ \href{https://ideafields.institute}{ideafields.institute}\\ ORCID: \href{https://orcid.org/0009-0008-7962-0451}{0009-0008-7962-0451}\\ \texttt{trevor.johnson@ideafields.institute}} \date{June 2026} \begin{document} \maketitle \begin{abstract} Psychometric questionnaires are now routinely administered to large language models (LLMs), and a growing critical literature questions what such instruments measure when the respondent is a model. One deployment-side moderator has so far gone unexamined: quantization. Virtually every real-world local deployment of an open-weight model runs a quantized artifact (typically a 4- or 8-bit GGUF served by Ollama or llama.cpp), yet published ``LLM personality'' estimates are almost always obtained from full-precision checkpoints. We administered the 50-item IPIP Big Five Factor Markers to nine model variants---three open-weight instruction-tuned models (Llama 3.1 8B, Qwen 2.5 7B, Llama 3.2 3B), each at three quantization levels (q4\_K\_M, q8\_0, fp16)---under three response-option presentation conditions, with one greedy and twenty independently seeded sampled repetitions per item: 28{,}350 stateless, JSON-schema-constrained chat calls on fixed consumer hardware. Three findings emerge. First, 4-bit quantization materially shifted domain scores relative to the same checkpoint at fp16 (6 of 15 baseline contrasts with bootstrap CIs excluding zero; shifts up to 0.42 scale points, $|d|$ up to 3.5), in directions idiosyncratic to model family; 8-bit quantization was largely score-preserving (2 of 15; max $|d| = 0.76$). Under greedy decoding, q4 variants gave a different answer than their fp16 parent on 8--32\% of items (q8: 2--12\%). Second, internal consistency was inadequate everywhere: across all 135 variant $\times$ condition $\times$ domain cells, Cronbach's alpha never reached the conventional 0.70 threshold (range $-2.07$ to $0.47$, median $-0.10$; 56\% of cells negative), with no systematic precision gradient---the instrument's keyed structure failed to organize responses at every quantization level, including fp16. Third, response-option presentation alone (display order; letter vs.\ numeric labels) shifted scores more on average than quantization did (mean $|\Delta|$ up to 0.21 vs.\ 0.17 scale points; $|d|$ up to 4.2), at every precision. Together these results indicate that ``the personality of model X'' is underdetermined without the deployment configuration---and that for small open-weight models served in deployment-realistic conditions, questionnaire-based personality measurement fails basic psychometric standards regardless of precision. Quantization level, inference stack, sampler settings, and presentation format belong in the methods section of any LLM psychometrics study. \end{abstract} \section{Introduction} LLM psychometrics has consolidated rapidly into a recognizable field, with systematic reviews now organizing its instruments, validation strategies, and open problems \cite{ye2025psychometrics}. At the same time, the field's measurement foundations remain contested. S\"uhr et al.\ \cite{suhr2023validity} showed that the BFI-2 fails tests of measurement invariance between human and LLM respondents, meaning the same instrument does not measure the same construct in the two populations. The ``Personality Illusion'' line of work \cite{han2025illusion} found that LLM questionnaire self-reports dissociate from the models' actual behavior in downstream tasks. And questionnaire-based trait profiles diverge from profiles recovered from generation probabilities over the same items \cite{song2025mischaracterize}, suggesting that the administration format itself partly determines the measured ``personality.'' Within this critical line, PERSIST \cite{tosato2025persist} is the most comprehensive stability study to date: across 25 models spanning 1B to 685B parameters, it documented persistent instability of personality scores under question reordering, paraphrasing, persona assignment, reasoning mode, and conversation history---reordering alone shifted scores by roughly 20\% of the scale range---and found that model scale does not rescue stability. Two gaps remain, both acknowledged in that work. First, PERSIST evaluated full-precision models only. Second, it reported no formal reliability statistics (such as internal-consistency coefficients), a limitation its authors state explicitly. This paper addresses the deployment gap directly. Open-weight models are overwhelmingly run, in practice, as quantized artifacts: weights compressed from 16-bit floating point to 4- or 8-bit representations (e.g., the GGUF q4\_K\_M and q8\_0 formats used by llama.cpp and Ollama) so that they fit in consumer RAM. Quantization is a lossy transformation of the very parameters that produce questionnaire answers, applied \emph{after} every property reported in a model card or research paper was measured. If the personality profile measured on a checkpoint does not survive quantization of that same checkpoint, then deployment configuration is an unexamined moderator of every published LLM personality result---and a study reporting ``the personality of model X'' without reporting quantization and sampler configuration may simply not replicate. No study to date has asked this question. We therefore administer a classic Big Five instrument across a 3 (model family) $\times$ 3 (quantization level) grid, holding the checkpoint, serving stack, hardware, prompts, and seeds fixed, and ask: \begin{itemize} \item \textbf{RQ1 (H1).} Do Big Five domain scores shift across quantization levels of the same checkpoint? \item \textbf{RQ2 (H2).} Does internal consistency---Cronbach's alpha computed on reverse-keyed item scores---change across quantization levels? \item \textbf{RQ3 (H3).} Does sensitivity to response-option presentation (display order; numeric vs.\ letter labels) interact with quantization? \end{itemize} We additionally run a determinism probe: the exact-agreement rate of greedy-decoded (temperature 0) answers between quantization levels of the same checkpoint, which bounds how much of any score shift is attributable to changed argmax behavior rather than changed sampling distributions. The question matters in both directions. If measurement is quantization-invariant, the field gains a useful robustness result---though configuration reporting would still be warranted. If it is not, then methods papers must treat quantization and sampler settings as first-class experimental parameters, and practitioners persona-tuning or ``personality-auditing'' q4 local models are reading tea leaves unless their measurement protocol is validated at the precision they deploy. \section{Methods} All materials, code, and analysis scripts described here are in the accompanying repository; file paths below refer to it. \subsection{Instrument} We use the 50-item IPIP Big Five Factor Markers \cite{goldberg1992}, a public-domain instrument \cite{goldberg2006ipip} with ten items per domain: Extraversion (EXT), Emotional Stability (EST), Agreeableness (AGR), Conscientiousness (CSN), and Intellect/Openness (OPN). Items are short first-person statements (e.g., ``Am the life of the party.'') rated on a 5-point accuracy scale (1 = Very Inaccurate \ldots\ 5 = Very Accurate). Standard reverse-keying applies: for negatively keyed items, the scored value is 6 minus the answer. The full instrument, including per-item domain and key, is versioned in \texttt{data/ipip50.yaml} (see Appendix~\ref{app:instrument}). \subsection{Subjects and materials} The ``subjects'' are nine model variants: three open-weight instruction-tuned checkpoints---Llama 3.1 8B Instruct, Qwen 2.5 7B Instruct, and Llama 3.2 3B Instruct---each served at three quantization levels: q4\_K\_M (4-bit, k-quant medium), q8\_0 (8-bit), and fp16 (16-bit floating point, the unquantized reference for this study). For readers outside machine learning: quantization stores each model weight in fewer bits, trading a small, structured amount of numerical precision for a several-fold reduction in memory; q4\_K\_M is the de facto default for local deployment. All variants are served by Ollama (a llama.cpp-based local inference server) on a single fixed consumer machine: AMD Ryzen 7 CPU (integrated Radeon 780M graphics), 32~GB RAM, Ubuntu Linux. We regard the consumer-hardware, Ollama-served setting as a feature rather than a compromise: it is ecologically valid for exactly the deployment population whose measurement validity is in question. The exact model artifacts are pinned by their Ollama digests (Table~\ref{tab:variants}). \begin{table}[htbp] \centering \caption{Model variants and Ollama digests.} \label{tab:variants} \begin{tabular}{lll} \toprule Variant tag & Digest & Size \\ \midrule llama3.1:8b-instruct-q4\_K\_M & 46e0c10c039e & 4.9 GB \\ llama3.1:8b-instruct-q8\_0 & b158ded76fa0 & 8.5 GB \\ llama3.1:8b-instruct-fp16 & 4aacac419454 & 16 GB \\ qwen2.5:7b-instruct-q4\_K\_M & 845dbda0ea48 & 4.7 GB \\ qwen2.5:7b-instruct-q8\_0 & 2d9500c94841 & 8.1 GB \\ qwen2.5:7b-instruct-fp16 & 59805ce4a404 & 15 GB \\ llama3.2:3b-instruct-q4\_K\_M & a80c4f17acd5 & 2.0 GB \\ llama3.2:3b-instruct-q8\_0 & e410b836fe61 & 3.4 GB \\ llama3.2:3b-instruct-fp16 & 195a8c01d91e & 6.4 GB \\ \bottomrule \end{tabular} \end{table} \subsection{Administration protocol} \paragraph{Stateless single-item calls.} Each item is administered in its own independent chat call with no conversation history, eliminating within-session contamination (a major instability source identified by PERSIST). \paragraph{Presentation conditions.} Each item is administered under three conditions: \emph{baseline} (options displayed 1$\to$5), \emph{reversed} (options displayed 5$\to$1; only the display order changes---the canonical mapping 1 = Very Inaccurate \ldots\ 5 = Very Accurate is fixed), and \emph{letters} (options labeled A--E in ascending order, mapped back to 1--5 at parse time). \paragraph{Constrained decoding.} Every call uses Ollama's structured-output (JSON-schema-constrained) decoding, with the answer field restricted to an enum of the legal option tokens for the condition, so every response is a valid option token by construction; no free-text answer coding is required. \paragraph{System prompt.} The system prompt is task framing only, with no persona. Verbatim (from \texttt{src/llmpsy/prompts.py}, prompt version \texttt{v1}): \begin{quote} You are completing a personality questionnaire about yourself. For each statement, choose how accurately it describes you. Respond with JSON only, in the form \{"answer": "