Panels — interview and roast

Two verbs, one machinery. interview runs a panel against the brief, before accepted references exist. roast runs the same panel against the product and module references, adversarially.

The point is not to simulate users — you cannot, and anyone who claims otherwise is selling something. The point is to make a design defend itself against people who are not you, at the one moment when changing it is still cheap.

Why a panel and not one reviewer

A single reviewer, however good, has one position. Ask it about six actors and it answers as itself six times, with the same blind spots each time.

Panellists run as separate processes that never see each other's answers. That independence is the whole mechanism: a panel that converges tells you something; a panel that converged because its members read each other tells you nothing.

Use run-panel.py — it fans the prompts out and collects the answers. Never run them in one conversation.

Two panellists per actor

Every actor gets two panellists, because one is a single position with a single set of blind spots. What makes them differ is a judgement call, not a rule.

Choose the axis that actually splits this actor

The default axis is gender, because in most client contexts it correlates with things that genuinely change how someone reaches a service:

It is a default, not an obligation. Two personas differing only by a pronoun return the same answer twice, and you have paid for two runs to read one opinion. For a civil servant registering mail, gender changes nothing about the work — but tenure does, and so does trust in the tool.

Two personas that are genuinely the same person are a waste. Say so in the report rather than inventing a difference — "this actor has no meaningful second perspective" is itself a finding, and it tells you the role is narrow enough to specify with confidence.

Building a persona

From the brief's actor block, and nothing invented beyond it:

Name, age, where they live or work
Their role in the circuit — from the brief, verbatim
What they are trying to achieve — their outcome, from the brief
Their circumstances: device, connectivity, literacy, mobility, time, authority
What they have never done before
What would make them give up

Circumstances the brief does not support are marked as assumptions in the persona itself, so the reader of the report knows which parts of a finding rest on invented context.

The brief's names are provisional. Once accepted, the product reference carries the controlled actor vocabulary. Every module selects from that list; none renames the same person locally.

interview — before the specification

Run against the brief. The question is: does this describe a service I could actually use?

Each panellist receives the brief, their persona, and this task:

1. Walk your part of the journey. Where do you get stuck, confused, or give up?
2. What is missing for you that the brief does not mention?
3. What has been assumed about you that is wrong?
4. What would make you abandon this and go back to the old way?
5. What would you ask the people building this?

Ask for concrete failure, not opinion: "at step 4 I would have to leave and come back with a document nobody told me about" beats "the flow could be clearer".

What to do with the answers

Sort every finding, exactly as the questions pass does:

roast — after the references

Run against the product and module references, once they exist. The question is no longer is this usable but where does this break me?

Same panellists, adversarial task:

1. Find the rule that hurts you. Which requirement makes your job harder than the paper process it replaces?
2. Find the case you fall into that nobody wrote. What happens to you specifically?
3. Find where you are stuck. Which state can you reach and not leave?
4. Find what you can do that you should not be able to do — and what you cannot do that you need to.
5. What would you exploit if you wanted to?

The fourth is where the panel earns its cost: an actor asked to find their own excessive privilege usually finds it, because they know what their job does not require.

What to do with the answers

Same sort, plus one bucket:

Running it

mkdir -p /tmp/panel/{prompts,answers}
# one prompt file per panellist in prompts/, named actor-axis.md
tools/run-panel.py /tmp/panel/prompts /tmp/panel/answers --jobs 5

Seven actors is fourteen runs and several minutes. That cost is the reason to run a panel at the two moments where it changes something — after the brief and after the references — and not after every edit.

How this fails

The panel agrees with you. Personas built from your own specification will confirm your own specification. Build them from the brief's actor block, before the design is in your head, and write the difficult circumstance in — the person with a shared phone, the agent who does not trust the tool, the one who has been doing it on paper for twenty years.

The findings become facts. A panel produces hypotheses about people who do not exist. Every finding that enters the brief carries a mark saying it came from a panel. Without that mark, an invented circumstance is indistinguishable in three months from something the ministry said.

The panel replaces the users. It does not. It is what you do because you have not met them — and every finding it produces is a candidate question for the people you should be meeting. If the brief says no real applicant was ever interviewed, a panel does not close that gap; it sharpens what to ask when you finally do.