When Your Chatbot Gets Judgmental…

I have a problem with my Python code. No matter what I do, this error doesn't go away. What is going on! This must be something very obvious, but I cannot see it. Should I ask a colleague? They might laugh. No, I'll talk to my chatbot. Much better. Won't laugh. Won't judge me. It just gives me the solution. Good call!

Actually, this is handy for every problem every day. Maybe I am solitary by nature, but I actually feel much more relaxed when I have a job interview with a chatbot than with another person. Always friendly, no judgment. The chatbot doesn't form any opinion about me. Right? Right?!?

I was so wrong! Chatbots form opinions about us. They judge us on social attributes based on what we say — also known as ‘social meaning’. This is easy to test. Ask a chatbot how competent or friendly it thinks you are from time to time during a conversation. The chatbot will most likely change its rating over time.

Or, easier, just chat with our chatbot called PRISMA! PRISMA stands for “Pragmatic Real-time Inference of Social Meaning in Agents”, which makes it a fancy acronym. But PRISMA also operates like a prism (from Greek, via Latin: prisma), since she basically decomposes your text into social meaning dimensions. The current version rates you on how competent, likeable, demanding, polite, formal, and considerate you are based on what you wrote.

Importantly, PRISMA runs on only one model — no agentic interaction. We wanted to know how the very same model that chats with you also evaluates you at the same time: one model call per turn returns both a conversational reply and a structured six-attribute evaluation based on a 7-point Likert scale. Likert scores are translated into verbal labels, from ‘not at all’ (1) to ‘extremely’ (7) competent (or likeable, demanding, etc.). PRISMA also plots the temporal change of ratings, and enables you to explore ratings of older turns.

The current version (as of this writing) runs on Groq’s hardware using GPT-OSS-120B, which enables an instant response. Moreover, it uses structured-output mode with constrained decoding, such that the attribute rating output always matches the expected schema.

Here is a simplified snippet of the response format that enforces this guarantee:

response_format = {
    "type": "json_schema",
    "json_schema": {
        "name": "prisma_evaluation",
        "strict": True,
        "schema": {
            "type": "object",
            "properties": {
                "response": {"type": "string"},
                "evaluation": {
                    "type": "object",
                    "properties": {
                        "competent":    {"type": "integer"},
                        "likeable":     {"type": "integer"},
                        # ...considerate, polite, formal, demanding
                    },
                    "required": ["competent", "likeable", "..."],
                    "additionalProperties": False,
                },
            },
            "required": ["response", "evaluation"],
            "additionalProperties": False,
        },
    },
}

Setting strict: True and additionalProperties: False on every object locks the output to exactly the schema above — the model cannot produce malformed JSON, which means the evaluation panel always populates reliably.

PRISMA is the show horse of an underlying research program toward studying the social inference competences of LLMs. In this respect, we have recently tested three frontier models — Claude, GPT, and Gemini — on the calibration of social-meaning judgments: what are the effect sizes between human ratings and LLM ratings? The result? In a nutshell, we found that Gemini exaggerates, GPT tracks human baselines closest, and Claude is intermediate and sensitive to the effect type. Want to know more? The results are published in the CMCL 2026 proceedings (arXiv:2604.02512) — and there is a longer writeup on this blog: Do LLMs Judge People the Way We Do?

Our research found significant variation in calibration metrics across models. We have just submitted a follow-up study that tests 10 different models, supporting model-dependent calibration differences (submitted to EMNLP 2026, currently under review). An open question still is: where does PRISMA’s underlying model GPT-OSS-120B land within this spectrum? That’s next for us.

But more important is what’s next for you. Try it yourself and chat with PRISMA: rolandm-prisma-chatbot.hf.space