🗣️ Roundtable

Don't ask one model. Convene a council.

A production-grade web app where your question is answered by several LLMs that debate over multiple rounds - critiquing and revising each other's answers anonymously - before a chairman synthesizes a final answer and an honest dissent report.

Unlike one-shot "LLM jury" tools, models see each other's critiques and revise across iterations, and the UI exposes the entire deliberation: every answer per round, every critique, what changed between revisions, and where the models still disagree.

Live demo · Quick start · Architecture · Design decisions

Why this exists

Asking a single model gives you a single model's blind spots. Ensembling by majority vote throws away the reasoning. Roundtable keeps the reasoning: models answer independently, critique each other blind to authorship, revise (or defend), and a neutral chairman synthesizes - surfacing genuine disagreement instead of averaging it away.

Everything a reviewer needs to trust the answer is on screen: the critique matrix, the revision diffs, the convergence score per round, and the per-model cost.

✨ Features


Multi-round debate engine	Round 0 independent answers → per-round critique → revision → convergence check → chairman synthesis, as an explicit, unit-tested state machine.
Live streaming	Per-model, per-stage progress streamed to the browser over SSE. Answers stream token-by-token.
Anonymized critique	Each model reviews the others' answers, relabeled `A/B/C...` in a per-reviewer randomized order - it cannot tell whose answer is whose.
Revision diffs	Toggleable inline word-diff between a model's round N and N+1 answers, with a structured changelog ("what changed & why" / "defending original").
Critique matrix	N×N grid of who-scored-whom; hover for the full critique; toggle to de-anonymize.
Dissent report	The chairman explicitly records unresolved disagreements and which model held which position.
Self-preference mitigation	Answers are anonymized to the chairman too; a warning badge flags when the chairman shares a provider family with a council member.
Cost transparency	Live cost meter during the debate + a per-model breakdown chart after, with per-user monthly spend.
Budget cap	An optional per-debate spend ceiling. Crossing it skips the remaining rounds and goes straight to synthesis - a cheaper answer instead of a runaway bill.
The gavel	Conclude a running debate gracefully: it finishes the phase in flight, records the partial round, and synthesizes what exists - unlike cancel, which aborts and wastes the spend.
Pressure response ("spine")	Per model, per round, cross the critique it received with whether it revised: revised, defended, caved (changed a well-scored answer), stonewalled (kept a poorly-scored one). Surfaces sycophancy and stubbornness with zero extra LLM calls.
Peer prediction	Each reviewer also predicts the council's average score for an answer. Comparing prediction to reality splits dissent into informed contrarians (low score, accurate read) and miscalibrated reviewers (low score, wrong read), with a per-reviewer "council read" calibration column.
Claim provenance	An advisory post-synthesis audit traces every substantive claim in the final answer back to the council's answers and flags unsourced "chairman additions" - synthesis hallucination detection - run on the cheap convergence model.
Masquerade audit	Reviewers also guess who wrote each anonymized answer; the guesses are scored against the real mapping. A hit rate near chance proves the masks are holding; well above flags style leakage - the app continuously audits its own bias mitigation.
Re-run & key recovery	Re-run any past debate (`/debate?from=<id>` prefills its question and council). A start that fails for a missing key drops you into the key dialog with the composed debate intact, not a dead end.
BYOK, two modes	Bring your own OpenRouter key - saved (AES-256-GCM encrypted at rest) or session-only (encrypted HttpOnly cookie). No server-paid key.
Portfolio demo mode	Public, no-login page that replays real recorded debates through the full UI with simulated streaming.
Export & share	Download a Markdown deliberation report, or mint an unlisted share link.
Robust by design	Per-model timeouts, exponential backoff, one-shot JSON repair, and graceful model drop-out (the debate proceeds with ≥2 healthy models). Debates continue server-side if the client disconnects.

🎬 Demo mode

/demo ships seeded, pre-recorded debates you can replay end-to-end - timeline, streaming playback, revision diffs, critique matrix - with zero configuration and no API key. This is the recruiter-first experience; it runs entirely on MOCK_LLM fixtures.

Screenshots / GIF live in docs/. Run the app (pnpm dev) and open /demo to see it live.

🚀 Quick start (no API key)

Everything runs offline in mock mode - a deterministic fake model powers real debates so you can develop and demo without spending a cent.

# 1. Install
corepack enable && pnpm install

# 2. Configure - the ONLY required secret for mock mode is the encryption key
cp .env.example .env
#   set ENCRYPTION_KEY (openssl rand -base64 32), point DATABASE_URL at a Postgres,
#   and set MOCK_LLM=1

# 3. Database
pnpm prisma db push      # create the schema
pnpm db:seed             # seed the public demo debates

# 4. Run
pnpm dev                 # http://localhost:3000

No Postgres handy? docker run -d -e POSTGRES_USER=roundtable -e POSTGRES_PASSWORD=roundtable -e POSTGRES_DB=roundtable -p 5432:5432 postgres:16-alpine.

Running real debates

Flip MOCK_LLM=0, then add your OpenRouter key in the app (the key badge in the header). The key is validated against OpenRouter and stored encrypted (saved) or in an HttpOnly cookie (session-only). Optional: set AUTH_GITHUB_ID/SECRET + AUTH_SECRET to enable GitHub sign-in and persistent per-user history.

🧱 Architecture

The debate orchestrator is a framework-agnostic TypeScript module - pure functions plus a state machine - with the Next.js route handler as a thin adapter. It never imports Next.js, Prisma, React, or a network client, so it is unit-testable without HTTP and reusable as a library/CLI.

flowchart TB
    subgraph client["Browser"]
        UI["Debate console<br/>(timeline · panels · diffs · matrix)"]
        RED["Event reducer<br/>(applyEvent → DebateView)"]
    end
    subgraph server["Next.js server (Node runtime)"]
        API["/api/debates/run<br/>(SSE adapter + rate limit)"]
        RUN["debate-runner<br/>(client select · persist · stream)"]
        subgraph core["core/ - framework-agnostic engine"]
            ORCH["orchestrator<br/>(state machine)"]
            PR["prompts/"]
            AN["anonymize · json-repair · scoring · convergence"]
        end
        LLM["LlmClient interface"]
        OR["OpenRouter client<br/>(Vercel AI SDK)"]
        MOCK["Mock client<br/>(deterministic)"]
        DB[("Postgres<br/>via Prisma")]
    end
    OPENROUTER(["OpenRouter gateway"])

    UI --> API
    API --> RUN --> ORCH
    ORCH --> PR & AN
    ORCH --> LLM
    LLM --> OR --> OPENROUTER
    LLM --> MOCK
    RUN -- "DebateEvent stream" --> API -- "SSE" --> RED --> UI
    RUN -- "persist each stage" --> DB
    API -- "replay snapshot" --> DB

A single typed DebateEvent discriminated union (stage_started, token_delta, answer_completed, critique_completed, convergence_result, model_failed, synthesis_completed, provenance_completed, budget_reached, gavel_struck, debate_completed, ...) is the one source of truth: the SSE endpoint serializes it, the persistence layer reacts to it, and the client reducer folds it into the DebateView that every component renders - live streaming and recorded replay share the exact same components.

The debate state machine

stateDiagram-v2
    [*] --> Answers: round 0 (parallel, streamed)
    Answers --> Critique: ≥2 healthy
    Answers --> Failed: <2 healthy
    Critique --> Revision: anonymized peer review (JSON)
    Revision --> Convergence: revise or defend (JSON + changelog)
    Convergence --> Critique: score < threshold & rounds left
    Convergence --> Synthesis: score ≥ threshold OR max rounds
    Critique --> Synthesis: gavel struck (partial round recorded)
    Convergence --> Synthesis: budget cap reached OR gavel struck
    Synthesis --> Provenance: trace claims (advisory)
    Provenance --> [*]: final answer + dissent + claim check

Project layout

src/
├── core/          # ★ framework-agnostic engine (no Next/Prisma/React imports)
│   ├── orchestrator.ts     # the debate state machine
│   ├── prompts/            # versioned, documented prompt templates
│   ├── anonymize.ts        # seeded PRNG + Fisher-Yates shuffle
│   ├── json-repair.ts      # defensive parse + Zod validate
│   ├── convergence.ts      # pure early-stop decision
│   ├── scoring.ts          # critique score matrix + peer-prediction analytics
│   ├── spine.ts            # per-model pressure response (revised/defended/caved/stonewalled)
│   ├── llm-client.ts       # the LlmClient interface the engine depends on
│   ├── mock-client.ts      # deterministic offline model
│   └── *.test.ts           # Vitest unit tests
├── lib/           # env (Zod), OpenRouter client, AES-256-GCM crypto, SSE, BYOK
├── db/            # Prisma client + repositories (write path + replay reconstruction)
├── app/           # Next.js App Router: routes (thin adapters) + pages
└── components/    # shadcn/ui primitives + the debate UI

🧠 Design decisions

Why multiple rounds instead of one-shot voting? A single critique pass catches surface errors; iteration lets a model change its mind when a critique is right and defend when it's wrong. The structured changelog forces an explicit changed boolean so "defending the original with reasons" is a first-class outcome, not social-pressure capitulation.

How self-preference bias is mitigated. Models reliably over-reward their own style. Three defenses: (1) during critique, each reviewer sees peers' answers anonymized and in a per-reviewer randomized order (a seeded shuffle, so it's reproducible and testable); (2) the chairman also sees anonymized answers - labels are mapped back to real models only after it commits its judgment; (3) the UI shows a provider-overlap warning when the chairman shares a vendor family with a council member, and the chairman defaults to a model outside the council.

Cost tradeoffs. A debate is N models × (1 answer + R×(1 critique + 1 revision)) + R convergence checks + 1 synthesis calls - cost grows roughly linearly in rounds and models. Mitigations: early stopping via a cheap convergence model (gemini-flash class), a configurable max-rounds cap, per-model/per-stage cost recorded so you can see exactly where spend goes, and a live meter so nothing runs away silently.

Robustness over perfection. Model APIs are flaky and models emit malformed JSON. Every structured call gets one JSON-repair retry; every call is timeout-bounded and cancellable; a model that fails a stage is dropped from that stage, not the debate, as long as ≥2 remain. Stage results are persisted as they complete, so a client disconnect never loses a debate.

Why the engine is isolated. Keeping core/ free of framework imports means the debate logic is exercised by fast unit tests with a deterministic mock client - the state machine, convergence, anonymization shuffle, and JSON repair are all covered without a browser, network, or database.

🧪 Testing

pnpm test         # Vitest - orchestrator, convergence, anonymization, JSON repair, crypto
pnpm typecheck    # tsc --noEmit (strict)
pnpm lint         # ESLint
pnpm build        # production build

The orchestrator suite drives full debates through the mock client and asserts on the emitted event stream: happy path, early convergence, max-rounds, model drop-out (and the ≥2-healthy floor), JSON-repair recovery, chairman provider-conflict detection, budget-cap and gavel early-exits, the advisory provenance audit, and cancellation. Around 130 unit tests cover the engine plus the lib layer (crypto, rate limiting, cost estimation, the event reducer, Markdown export, spine and peer-prediction analytics, and the demo fixtures). CI also runs a schema + seed smoke test against a real Postgres.

⌨️ Run the engine from the terminal

Because src/core is framework-agnostic, the same orchestrator the web app drives over SSE also runs as a CLI - no Next.js, database, or network required:

pnpm debate "Is a modular monolith better than microservices for a small team?"

It streams each stage to stdout and prints the chairman's final answer, dissent, any unsourced "chairman additions" from the provenance audit, and cost, all on the deterministic mock client.

🚢 Deployment (Linux VPS + Caddy)

Not serverless - the app is a long-running Node server designed for a VPS, with SSE debate streams that last minutes.

# On the VPS
cp .env.example .env         # set ENCRYPTION_KEY, SITE_ADDRESS (your domain), auth vars
docker compose up -d         # Postgres + app (standalone) + Caddy (auto-HTTPS)
docker compose exec app node node_modules/prisma/build/index.js db seed   # optional demos

Multi-stage Dockerfile builds Next.js standalone output on node:20-slim; the entrypoint applies the schema (prisma db push) on boot.
docker-compose.yml wires app + Postgres + Caddy with health checks and named volumes.
Caddyfile reverse-proxies with flush_interval -1 (no buffering) and long read/write windows so SSE streams flush immediately - automatic HTTPS via SITE_ADDRESS.
CI/CD - ci.yml runs lint · typecheck · test · build on every push; deploy.yml builds & pushes the image to GHCR and SSH-deploys to the VPS (docker compose pull && up -d).

Environment

All variables are validated at boot with Zod (src/lib/env.ts); a misconfigured deploy fails loudly at startup. ENCRYPTION_KEY (32-byte base64) is the only secret required for public/demo mode. See .env.example.

🛠️ Tech stack

Next.js 15 (App Router) · TypeScript (strict) · Tailwind CSS + shadcn/ui · Vercel AI SDK · OpenRouter · PostgreSQL + Prisma · Auth.js (GitHub) · Zod · Vitest · Docker + Caddy · GitHub Actions.

Built as a portfolio project. The debate engine is deliberately reusable - lift src/core/ into a CLI or a different frontend and it just works.

rasmusjy / roundtable