README.md
15,299 bytes
| 1 | <div align="center"> |
|---|---|
| 2 | |
| 3 | # 🗣️ Roundtable |
| 4 | |
| 5 | **Don't ask one model. Convene a council.** |
| 6 | |
| 7 | A production-grade web app where your question is answered by several LLMs that **debate over multiple rounds** - critiquing and revising each other's answers anonymously - before a chairman synthesizes a final answer and an honest **dissent report**. |
| 8 | |
| 9 | Unlike one-shot "LLM jury" tools, models _see each other's critiques and revise across iterations_, and the UI exposes the **entire deliberation**: every answer per round, every critique, what changed between revisions, and where the models still disagree. |
| 10 | |
| 11 | [Live demo](#-demo-mode) · [Quick start](#-quick-start-no-api-key) · [Architecture](#-architecture) · [Design decisions](#-design-decisions) |
| 12 | |
| 13 | </div> |
| 14 | |
| 15 | --- |
| 16 | |
| 17 | ## Why this exists |
| 18 | |
| 19 | Asking a single model gives you a single model's blind spots. Ensembling by majority vote throws away the reasoning. **Roundtable** keeps the reasoning: models answer independently, critique each other blind to authorship, revise (or defend), and a neutral chairman synthesizes - surfacing genuine disagreement instead of averaging it away. |
| 20 | |
| 21 | Everything a reviewer needs to _trust_ the answer is on screen: the critique matrix, the revision diffs, the convergence score per round, and the per-model cost. |
| 22 | |
| 23 | ## ✨ Features |
| 24 | |
| 25 | | | | |
| 26 | |---|---| |
| 27 | | **Multi-round debate engine** | Round 0 independent answers → per-round critique → revision → convergence check → chairman synthesis, as an explicit, unit-tested state machine. | |
| 28 | | **Live streaming** | Per-model, per-stage progress streamed to the browser over SSE. Answers stream token-by-token. | |
| 29 | | **Anonymized critique** | Each model reviews the _others'_ answers, relabeled `A/B/C...` in a per-reviewer randomized order - it cannot tell whose answer is whose. | |
| 30 | | **Revision diffs** | Toggleable inline word-diff between a model's round _N_ and _N+1_ answers, with a structured changelog ("what changed & why" / "defending original"). | |
| 31 | | **Critique matrix** | N×N grid of who-scored-whom; hover for the full critique; toggle to de-anonymize. | |
| 32 | | **Dissent report** | The chairman explicitly records unresolved disagreements and which model held which position. | |
| 33 | | **Self-preference mitigation** | Answers are anonymized to the chairman too; a warning badge flags when the chairman shares a provider family with a council member. | |
| 34 | | **Cost transparency** | Live cost meter during the debate + a per-model breakdown chart after, with per-user monthly spend. | |
| 35 | | **Budget cap** | An optional per-debate spend ceiling. Crossing it skips the remaining rounds and goes straight to synthesis - a cheaper answer instead of a runaway bill. | |
| 36 | | **The gavel** | Conclude a running debate gracefully: it finishes the phase in flight, records the partial round, and synthesizes what exists - unlike cancel, which aborts and wastes the spend. | |
| 37 | | **Pressure response ("spine")** | Per model, per round, cross the critique it received with whether it revised: _revised_, _defended_, _caved_ (changed a well-scored answer), _stonewalled_ (kept a poorly-scored one). Surfaces sycophancy and stubbornness with zero extra LLM calls. | |
| 38 | | **Peer prediction** | Each reviewer also predicts the council's average score for an answer. Comparing prediction to reality splits dissent into informed contrarians (low score, accurate read) and miscalibrated reviewers (low score, wrong read), with a per-reviewer "council read" calibration column. | |
| 39 | | **Claim provenance** | An advisory post-synthesis audit traces every substantive claim in the final answer back to the council's answers and flags unsourced "chairman additions" - synthesis hallucination detection - run on the cheap convergence model. | |
| 40 | | **Masquerade audit** | Reviewers also guess who wrote each anonymized answer; the guesses are scored against the real mapping. A hit rate near chance proves the masks are holding; well above flags style leakage - the app continuously audits its own bias mitigation. | |
| 41 | | **Re-run & key recovery** | Re-run any past debate (`/debate?from=<id>` prefills its question and council). A start that fails for a missing key drops you into the key dialog with the composed debate intact, not a dead end. | |
| 42 | | **BYOK, two modes** | Bring your own OpenRouter key - saved (AES-256-GCM encrypted at rest) or session-only (encrypted HttpOnly cookie). No server-paid key. | |
| 43 | | **Portfolio demo mode** | Public, no-login page that replays real recorded debates through the full UI with simulated streaming. | |
| 44 | | **Export & share** | Download a Markdown deliberation report, or mint an unlisted share link. | |
| 45 | | **Robust by design** | Per-model timeouts, exponential backoff, one-shot JSON repair, and graceful model drop-out (the debate proceeds with ≥2 healthy models). Debates continue server-side if the client disconnects. | |
| 46 | |
| 47 | ## 🎬 Demo mode |
| 48 | |
| 49 | `/demo` ships seeded, pre-recorded debates you can replay end-to-end - timeline, streaming playback, revision diffs, critique matrix - **with zero configuration and no API key**. This is the recruiter-first experience; it runs entirely on `MOCK_LLM` fixtures. |
| 50 | |
| 51 | > Screenshots / GIF live in [`docs/`](docs/). Run the app (`pnpm dev`) and open `/demo` to see it live. |
| 52 | |
| 53 | ## 🚀 Quick start (no API key) |
| 54 | |
| 55 | Everything runs offline in **mock mode** - a deterministic fake model powers real debates so you can develop and demo without spending a cent. |
| 56 | |
| 57 | ```bash |
| 58 | # 1. Install |
| 59 | corepack enable && pnpm install |
| 60 | |
| 61 | # 2. Configure - the ONLY required secret for mock mode is the encryption key |
| 62 | cp .env.example .env |
| 63 | # set ENCRYPTION_KEY (openssl rand -base64 32), point DATABASE_URL at a Postgres, |
| 64 | # and set MOCK_LLM=1 |
| 65 | |
| 66 | # 3. Database |
| 67 | pnpm prisma db push # create the schema |
| 68 | pnpm db:seed # seed the public demo debates |
| 69 | |
| 70 | # 4. Run |
| 71 | pnpm dev # http://localhost:3000 |
| 72 | ``` |
| 73 | |
| 74 | No Postgres handy? `docker run -d -e POSTGRES_USER=roundtable -e POSTGRES_PASSWORD=roundtable -e POSTGRES_DB=roundtable -p 5432:5432 postgres:16-alpine`. |
| 75 | |
| 76 | ### Running real debates |
| 77 | |
| 78 | Flip `MOCK_LLM=0`, then add your [OpenRouter](https://openrouter.ai/) key in the app (the key badge in the header). The key is validated against OpenRouter and stored encrypted (saved) or in an HttpOnly cookie (session-only). Optional: set `AUTH_GITHUB_ID/SECRET` + `AUTH_SECRET` to enable GitHub sign-in and persistent per-user history. |
| 79 | |
| 80 | ## 🧱 Architecture |
| 81 | |
| 82 | The debate **orchestrator is a framework-agnostic TypeScript module** - pure functions plus a state machine - with the Next.js route handler as a thin adapter. It never imports Next.js, Prisma, React, or a network client, so it is unit-testable without HTTP and reusable as a library/CLI. |
| 83 | |
| 84 | ```mermaid |
| 85 | flowchart TB |
| 86 | subgraph client["Browser"] |
| 87 | UI["Debate console<br/>(timeline · panels · diffs · matrix)"] |
| 88 | RED["Event reducer<br/>(applyEvent → DebateView)"] |
| 89 | end |
| 90 | subgraph server["Next.js server (Node runtime)"] |
| 91 | API["/api/debates/run<br/>(SSE adapter + rate limit)"] |
| 92 | RUN["debate-runner<br/>(client select · persist · stream)"] |
| 93 | subgraph core["core/ - framework-agnostic engine"] |
| 94 | ORCH["orchestrator<br/>(state machine)"] |
| 95 | PR["prompts/"] |
| 96 | AN["anonymize · json-repair · scoring · convergence"] |
| 97 | end |
| 98 | LLM["LlmClient interface"] |
| 99 | OR["OpenRouter client<br/>(Vercel AI SDK)"] |
| 100 | MOCK["Mock client<br/>(deterministic)"] |
| 101 | DB[("Postgres<br/>via Prisma")] |
| 102 | end |
| 103 | OPENROUTER(["OpenRouter gateway"]) |
| 104 | |
| 105 | UI --> API |
| 106 | API --> RUN --> ORCH |
| 107 | ORCH --> PR & AN |
| 108 | ORCH --> LLM |
| 109 | LLM --> OR --> OPENROUTER |
| 110 | LLM --> MOCK |
| 111 | RUN -- "DebateEvent stream" --> API -- "SSE" --> RED --> UI |
| 112 | RUN -- "persist each stage" --> DB |
| 113 | API -- "replay snapshot" --> DB |
| 114 | ``` |
| 115 | |
| 116 | A single typed **`DebateEvent`** discriminated union (`stage_started`, `token_delta`, `answer_completed`, `critique_completed`, `convergence_result`, `model_failed`, `synthesis_completed`, `provenance_completed`, `budget_reached`, `gavel_struck`, `debate_completed`, ...) is the one source of truth: the SSE endpoint serializes it, the persistence layer reacts to it, and the client reducer folds it into the `DebateView` that every component renders - live streaming and recorded replay share the exact same components. |
| 117 | |
| 118 | ### The debate state machine |
| 119 | |
| 120 | ```mermaid |
| 121 | stateDiagram-v2 |
| 122 | [*] --> Answers: round 0 (parallel, streamed) |
| 123 | Answers --> Critique: ≥2 healthy |
| 124 | Answers --> Failed: <2 healthy |
| 125 | Critique --> Revision: anonymized peer review (JSON) |
| 126 | Revision --> Convergence: revise or defend (JSON + changelog) |
| 127 | Convergence --> Critique: score < threshold & rounds left |
| 128 | Convergence --> Synthesis: score ≥ threshold OR max rounds |
| 129 | Critique --> Synthesis: gavel struck (partial round recorded) |
| 130 | Convergence --> Synthesis: budget cap reached OR gavel struck |
| 131 | Synthesis --> Provenance: trace claims (advisory) |
| 132 | Provenance --> [*]: final answer + dissent + claim check |
| 133 | ``` |
| 134 | |
| 135 | ### Project layout |
| 136 | |
| 137 | ``` |
| 138 | src/ |
| 139 | ├── core/ # ★ framework-agnostic engine (no Next/Prisma/React imports) |
| 140 | │ ├── orchestrator.ts # the debate state machine |
| 141 | │ ├── prompts/ # versioned, documented prompt templates |
| 142 | │ ├── anonymize.ts # seeded PRNG + Fisher-Yates shuffle |
| 143 | │ ├── json-repair.ts # defensive parse + Zod validate |
| 144 | │ ├── convergence.ts # pure early-stop decision |
| 145 | │ ├── scoring.ts # critique score matrix + peer-prediction analytics |
| 146 | │ ├── spine.ts # per-model pressure response (revised/defended/caved/stonewalled) |
| 147 | │ ├── llm-client.ts # the LlmClient interface the engine depends on |
| 148 | │ ├── mock-client.ts # deterministic offline model |
| 149 | │ └── *.test.ts # Vitest unit tests |
| 150 | ├── lib/ # env (Zod), OpenRouter client, AES-256-GCM crypto, SSE, BYOK |
| 151 | ├── db/ # Prisma client + repositories (write path + replay reconstruction) |
| 152 | ├── app/ # Next.js App Router: routes (thin adapters) + pages |
| 153 | └── components/ # shadcn/ui primitives + the debate UI |
| 154 | ``` |
| 155 | |
| 156 | ## 🧠 Design decisions |
| 157 | |
| 158 | **Why multiple rounds instead of one-shot voting?** A single critique pass catches surface errors; iteration lets a model _change its mind_ when a critique is right and _defend_ when it's wrong. The structured changelog forces an explicit `changed` boolean so "defending the original with reasons" is a first-class outcome, not social-pressure capitulation. |
| 159 | |
| 160 | **How self-preference bias is mitigated.** Models reliably over-reward their own style. Three defenses: (1) during critique, each reviewer sees peers' answers **anonymized and in a per-reviewer randomized order** (a seeded shuffle, so it's reproducible and testable); (2) the **chairman also sees anonymized answers** - labels are mapped back to real models only _after_ it commits its judgment; (3) the UI shows a **provider-overlap warning** when the chairman shares a vendor family with a council member, and the chairman defaults to a model outside the council. |
| 161 | |
| 162 | **Cost tradeoffs.** A debate is N models × (1 answer + R×(1 critique + 1 revision)) + R convergence checks + 1 synthesis calls - cost grows roughly linearly in rounds and models. Mitigations: early stopping via a **cheap** convergence model (gemini-flash class), a configurable max-rounds cap, per-model/per-stage cost recorded so you can see exactly where spend goes, and a live meter so nothing runs away silently. |
| 163 | |
| 164 | **Robustness over perfection.** Model APIs are flaky and models emit malformed JSON. Every structured call gets one JSON-repair retry; every call is timeout-bounded and cancellable; a model that fails a stage is dropped from _that stage_, not the debate, as long as ≥2 remain. Stage results are persisted as they complete, so a client disconnect never loses a debate. |
| 165 | |
| 166 | **Why the engine is isolated.** Keeping `core/` free of framework imports means the debate logic is exercised by fast unit tests with a deterministic mock client - the state machine, convergence, anonymization shuffle, and JSON repair are all covered without a browser, network, or database. |
| 167 | |
| 168 | ## 🧪 Testing |
| 169 | |
| 170 | ```bash |
| 171 | pnpm test # Vitest - orchestrator, convergence, anonymization, JSON repair, crypto |
| 172 | pnpm typecheck # tsc --noEmit (strict) |
| 173 | pnpm lint # ESLint |
| 174 | pnpm build # production build |
| 175 | ``` |
| 176 | |
| 177 | The orchestrator suite drives full debates through the mock client and asserts on the emitted event stream: happy path, early convergence, max-rounds, model drop-out (and the ≥2-healthy floor), JSON-repair recovery, chairman provider-conflict detection, budget-cap and gavel early-exits, the advisory provenance audit, and cancellation. Around 130 unit tests cover the engine plus the lib layer (crypto, rate limiting, cost estimation, the event reducer, Markdown export, spine and peer-prediction analytics, and the demo fixtures). CI also runs a schema + seed smoke test against a real Postgres. |
| 178 | |
| 179 | ## ⌨️ Run the engine from the terminal |
| 180 | |
| 181 | Because `src/core` is framework-agnostic, the same orchestrator the web app drives over SSE also runs as a CLI - no Next.js, database, or network required: |
| 182 | |
| 183 | ```bash |
| 184 | pnpm debate "Is a modular monolith better than microservices for a small team?" |
| 185 | ``` |
| 186 | |
| 187 | It streams each stage to stdout and prints the chairman's final answer, dissent, any unsourced "chairman additions" from the provenance audit, and cost, all on the deterministic mock client. |
| 188 | |
| 189 | ## 🚢 Deployment (Linux VPS + Caddy) |
| 190 | |
| 191 | Not serverless - the app is a long-running Node server designed for a VPS, with SSE debate streams that last minutes. |
| 192 | |
| 193 | ```bash |
| 194 | # On the VPS |
| 195 | cp .env.example .env # set ENCRYPTION_KEY, SITE_ADDRESS (your domain), auth vars |
| 196 | docker compose up -d # Postgres + app (standalone) + Caddy (auto-HTTPS) |
| 197 | docker compose exec app node node_modules/prisma/build/index.js db seed # optional demos |
| 198 | ``` |
| 199 | |
| 200 | - **Multi-stage `Dockerfile`** builds Next.js `standalone` output on `node:20-slim`; the entrypoint applies the schema (`prisma db push`) on boot. |
| 201 | - **`docker-compose.yml`** wires app + Postgres + Caddy with health checks and named volumes. |
| 202 | - **`Caddyfile`** reverse-proxies with `flush_interval -1` (no buffering) and long read/write windows so SSE streams flush immediately - automatic HTTPS via `SITE_ADDRESS`. |
| 203 | - **CI/CD** - [`ci.yml`](.github/workflows/ci.yml) runs lint · typecheck · test · build on every push; [`deploy.yml`](.github/workflows/deploy.yml) builds & pushes the image to GHCR and SSH-deploys to the VPS (`docker compose pull && up -d`). |
| 204 | |
| 205 | ### Environment |
| 206 | |
| 207 | All variables are validated at boot with Zod ([`src/lib/env.ts`](src/lib/env.ts)); a misconfigured deploy fails loudly at startup. `ENCRYPTION_KEY` (32-byte base64) is the only secret required for public/demo mode. See [`.env.example`](.env.example). |
| 208 | |
| 209 | ## 🛠️ Tech stack |
| 210 | |
| 211 | Next.js 15 (App Router) · TypeScript (strict) · Tailwind CSS + shadcn/ui · Vercel AI SDK · OpenRouter · PostgreSQL + Prisma · Auth.js (GitHub) · Zod · Vitest · Docker + Caddy · GitHub Actions. |
| 212 | |
| 213 | --- |
| 214 | |
| 215 | <div align="center"> |
| 216 | <sub>Built as a portfolio project. The debate engine is deliberately reusable - lift <code>src/core/</code> into a CLI or a different frontend and it just works.</sub> |
| 217 | </div> |
| 218 | |