|
|
@@ -0,0 +1,196 @@ |
|
1 |
+<div align="center"> |
|
2 |
+ |
|
3 |
+# 🗣️ Roundtable |
|
4 |
+ |
|
5 |
+**Don't ask one model. Convene a council.** |
|
6 |
+ |
|
7 |
+A production-grade web app where your question is answered by several LLMs that **debate over multiple rounds** - critiquing and revising each other's answers anonymously - before a chairman synthesizes a final answer and an honest **dissent report**. |
|
8 |
+ |
|
9 |
+Unlike one-shot "LLM jury" tools, models _see each other's critiques and revise across iterations_, and the UI exposes the **entire deliberation**: every answer per round, every critique, what changed between revisions, and where the models still disagree. |
|
10 |
+ |
|
11 |
+[Live demo](#-demo-mode) · [Quick start](#-quick-start-no-api-key) · [Architecture](#-architecture) · [Design decisions](#-design-decisions) |
|
12 |
+ |
|
13 |
+</div> |
|
14 |
+ |
|
15 |
+--- |
|
16 |
+ |
|
17 |
+## Why this exists |
|
18 |
+ |
|
19 |
+Asking a single model gives you a single model's blind spots. Ensembling by majority vote throws away the reasoning. **Roundtable** keeps the reasoning: models answer independently, critique each other blind to authorship, revise (or defend), and a neutral chairman synthesizes - surfacing genuine disagreement instead of averaging it away. |
|
20 |
+ |
|
21 |
+Everything a reviewer needs to _trust_ the answer is on screen: the critique matrix, the revision diffs, the convergence score per round, and the per-model cost. |
|
22 |
+ |
|
23 |
+## ✨ Features |
|
24 |
+ |
|
25 |
+| | | |
|
26 |
+|---|---| |
|
27 |
+| **Multi-round debate engine** | Round 0 independent answers → per-round critique → revision → convergence check → chairman synthesis, as an explicit, unit-tested state machine. | |
|
28 |
+| **Live streaming** | Per-model, per-stage progress streamed to the browser over SSE. Answers stream token-by-token. | |
|
29 |
+| **Anonymized critique** | Each model reviews the _others'_ answers, relabeled `A/B/C...` in a per-reviewer randomized order - it cannot tell whose answer is whose. | |
|
30 |
+| **Revision diffs** | Toggleable inline word-diff between a model's round _N_ and _N+1_ answers, with a structured changelog ("what changed & why" / "defending original"). | |
|
31 |
+| **Critique matrix** | N×N grid of who-scored-whom; hover for the full critique; toggle to de-anonymize. | |
|
32 |
+| **Dissent report** | The chairman explicitly records unresolved disagreements and which model held which position. | |
|
33 |
+| **Self-preference mitigation** | Answers are anonymized to the chairman too; a warning badge flags when the chairman shares a provider family with a council member. | |
|
34 |
+| **Cost transparency** | Live cost meter during the debate + a per-model breakdown chart after, with per-user monthly spend. | |
|
35 |
+| **BYOK, two modes** | Bring your own OpenRouter key - saved (AES-256-GCM encrypted at rest) or session-only (encrypted HttpOnly cookie). No server-paid key. | |
|
36 |
+| **Portfolio demo mode** | Public, no-login page that replays real recorded debates through the full UI with simulated streaming. | |
|
37 |
+| **Export & share** | Download a Markdown deliberation report, or mint an unlisted share link. | |
|
38 |
+| **Robust by design** | Per-model timeouts, exponential backoff, one-shot JSON repair, and graceful model drop-out (the debate proceeds with ≥2 healthy models). Debates continue server-side if the client disconnects. | |
|
39 |
+ |
|
40 |
+## 🎬 Demo mode |
|
41 |
+ |
|
42 |
+`/demo` ships seeded, pre-recorded debates you can replay end-to-end - timeline, streaming playback, revision diffs, critique matrix - **with zero configuration and no API key**. This is the recruiter-first experience; it runs entirely on `MOCK_LLM` fixtures. |
|
43 |
+ |
|
44 |
+> Screenshots / GIF live in [`docs/`](docs/). Run the app (`pnpm dev`) and open `/demo` to see it live. |
|
45 |
+ |
|
46 |
+## 🚀 Quick start (no API key) |
|
47 |
+ |
|
48 |
+Everything runs offline in **mock mode** - a deterministic fake model powers real debates so you can develop and demo without spending a cent. |
|
49 |
+ |
|
50 |
+```bash |
|
51 |
+# 1. Install |
|
52 |
+corepack enable && pnpm install |
|
53 |
+ |
|
54 |
+# 2. Configure - the ONLY required secret for mock mode is the encryption key |
|
55 |
+cp .env.example .env |
|
56 |
+# set ENCRYPTION_KEY (openssl rand -base64 32), point DATABASE_URL at a Postgres, |
|
57 |
+# and set MOCK_LLM=1 |
|
58 |
+ |
|
59 |
+# 3. Database |
|
60 |
+pnpm prisma db push # create the schema |
|
61 |
+pnpm db:seed # seed the public demo debates |
|
62 |
+ |
|
63 |
+# 4. Run |
|
64 |
+pnpm dev # http://localhost:3000 |
|
65 |
+``` |
|
66 |
+ |
|
67 |
+No Postgres handy? `docker run -d -e POSTGRES_USER=roundtable -e POSTGRES_PASSWORD=roundtable -e POSTGRES_DB=roundtable -p 5432:5432 postgres:16-alpine`. |
|
68 |
+ |
|
69 |
+### Running real debates |
|
70 |
+ |
|
71 |
+Flip `MOCK_LLM=0`, then add your [OpenRouter](https://openrouter.ai/) key in the app (the key badge in the header). The key is validated against OpenRouter and stored encrypted (saved) or in an HttpOnly cookie (session-only). Optional: set `AUTH_GITHUB_ID/SECRET` + `AUTH_SECRET` to enable GitHub sign-in and persistent per-user history. |
|
72 |
+ |
|
73 |
+## 🧱 Architecture |
|
74 |
+ |
|
75 |
+The debate **orchestrator is a framework-agnostic TypeScript module** - pure functions plus a state machine - with the Next.js route handler as a thin adapter. It never imports Next.js, Prisma, React, or a network client, so it is unit-testable without HTTP and reusable as a library/CLI. |
|
76 |
+ |
|
77 |
+```mermaid |
|
78 |
+flowchart TB |
|
79 |
+ subgraph client["Browser"] |
|
80 |
+ UI["Debate console<br/>(timeline · panels · diffs · matrix)"] |
|
81 |
+ RED["Event reducer<br/>(applyEvent → DebateView)"] |
|
82 |
+ end |
|
83 |
+ subgraph server["Next.js server (Node runtime)"] |
|
84 |
+ API["/api/debates/run<br/>(SSE adapter + rate limit)"] |
|
85 |
+ RUN["debate-runner<br/>(client select · persist · stream)"] |
|
86 |
+ subgraph core["core/ - framework-agnostic engine"] |
|
87 |
+ ORCH["orchestrator<br/>(state machine)"] |
|
88 |
+ PR["prompts/"] |
|
89 |
+ AN["anonymize · json-repair · scoring · convergence"] |
|
90 |
+ end |
|
91 |
+ LLM["LlmClient interface"] |
|
92 |
+ OR["OpenRouter client<br/>(Vercel AI SDK)"] |
|
93 |
+ MOCK["Mock client<br/>(deterministic)"] |
|
94 |
+ DB[("Postgres<br/>via Prisma")] |
|
95 |
+ end |
|
96 |
+ OPENROUTER(["OpenRouter gateway"]) |
|
97 |
+ |
|
98 |
+ UI --> API |
|
99 |
+ API --> RUN --> ORCH |
|
100 |
+ ORCH --> PR & AN |
|
101 |
+ ORCH --> LLM |
|
102 |
+ LLM --> OR --> OPENROUTER |
|
103 |
+ LLM --> MOCK |
|
104 |
+ RUN -- "DebateEvent stream" --> API -- "SSE" --> RED --> UI |
|
105 |
+ RUN -- "persist each stage" --> DB |
|
106 |
+ API -- "replay snapshot" --> DB |
|
107 |
+``` |
|
108 |
+ |
|
109 |
+A single typed **`DebateEvent`** discriminated union (`stage_started`, `token_delta`, `answer_completed`, `critique_completed`, `convergence_result`, `model_failed`, `synthesis_completed`, `debate_completed`, ...) is the one source of truth: the SSE endpoint serializes it, the persistence layer reacts to it, and the client reducer folds it into the `DebateView` that every component renders - live streaming and recorded replay share the exact same components. |
|
110 |
+ |
|
111 |
+### The debate state machine |
|
112 |
+ |
|
113 |
+```mermaid |
|
114 |
+stateDiagram-v2 |
|
115 |
+ [*] --> Answers: round 0 (parallel, streamed) |
|
116 |
+ Answers --> Critique: ≥2 healthy |
|
117 |
+ Answers --> Failed: <2 healthy |
|
118 |
+ Critique --> Revision: anonymized peer review (JSON) |
|
119 |
+ Revision --> Convergence: revise or defend (JSON + changelog) |
|
120 |
+ Convergence --> Critique: score < threshold & rounds left |
|
121 |
+ Convergence --> Synthesis: score ≥ threshold OR max rounds |
|
122 |
+ Synthesis --> [*]: final answer + dissent report |
|
123 |
+``` |
|
124 |
+ |
|
125 |
+### Project layout |
|
126 |
+ |
|
127 |
+``` |
|
128 |
+src/ |
|
129 |
+├── core/ # ★ framework-agnostic engine (no Next/Prisma/React imports) |
|
130 |
+│ ├── orchestrator.ts # the debate state machine |
|
131 |
+│ ├── prompts/ # versioned, documented prompt templates |
|
132 |
+│ ├── anonymize.ts # seeded PRNG + Fisher-Yates shuffle |
|
133 |
+│ ├── json-repair.ts # defensive parse + Zod validate |
|
134 |
+│ ├── convergence.ts # pure early-stop decision |
|
135 |
+│ ├── scoring.ts # critique score matrix |
|
136 |
+│ ├── llm-client.ts # the LlmClient interface the engine depends on |
|
137 |
+│ ├── mock-client.ts # deterministic offline model |
|
138 |
+│ └── *.test.ts # Vitest unit tests |
|
139 |
+├── lib/ # env (Zod), OpenRouter client, AES-256-GCM crypto, SSE, BYOK |
|
140 |
+├── db/ # Prisma client + repositories (write path + replay reconstruction) |
|
141 |
+├── app/ # Next.js App Router: routes (thin adapters) + pages |
|
142 |
+└── components/ # shadcn/ui primitives + the debate UI |
|
143 |
+``` |
|
144 |
+ |
|
145 |
+## 🧠 Design decisions |
|
146 |
+ |
|
147 |
+**Why multiple rounds instead of one-shot voting?** A single critique pass catches surface errors; iteration lets a model _change its mind_ when a critique is right and _defend_ when it's wrong. The structured changelog forces an explicit `changed` boolean so "defending the original with reasons" is a first-class outcome, not social-pressure capitulation. |
|
148 |
+ |
|
149 |
+**How self-preference bias is mitigated.** Models reliably over-reward their own style. Three defenses: (1) during critique, each reviewer sees peers' answers **anonymized and in a per-reviewer randomized order** (a seeded shuffle, so it's reproducible and testable); (2) the **chairman also sees anonymized answers** - labels are mapped back to real models only _after_ it commits its judgment; (3) the UI shows a **provider-overlap warning** when the chairman shares a vendor family with a council member, and the chairman defaults to a model outside the council. |
|
150 |
+ |
|
151 |
+**Cost tradeoffs.** A debate is N models × (1 answer + R×(1 critique + 1 revision)) + R convergence checks + 1 synthesis calls - cost grows roughly linearly in rounds and models. Mitigations: early stopping via a **cheap** convergence model (gemini-flash class), a configurable max-rounds cap, per-model/per-stage cost recorded so you can see exactly where spend goes, and a live meter so nothing runs away silently. |
|
152 |
+ |
|
153 |
+**Robustness over perfection.** Model APIs are flaky and models emit malformed JSON. Every structured call gets one JSON-repair retry; every call is timeout-bounded and cancellable; a model that fails a stage is dropped from _that stage_, not the debate, as long as ≥2 remain. Stage results are persisted as they complete, so a client disconnect never loses a debate. |
|
154 |
+ |
|
155 |
+**Why the engine is isolated.** Keeping `core/` free of framework imports means the debate logic is exercised by fast unit tests with a deterministic mock client - the state machine, convergence, anonymization shuffle, and JSON repair are all covered without a browser, network, or database. |
|
156 |
+ |
|
157 |
+## 🧪 Testing |
|
158 |
+ |
|
159 |
+```bash |
|
160 |
+pnpm test # Vitest - orchestrator, convergence, anonymization, JSON repair, crypto |
|
161 |
+pnpm typecheck # tsc --noEmit (strict) |
|
162 |
+pnpm lint # ESLint |
|
163 |
+pnpm build # production build |
|
164 |
+``` |
|
165 |
+ |
|
166 |
+The orchestrator suite drives full debates through the mock client and asserts on the emitted event stream: happy path, early convergence, max-rounds, model drop-out (and the ≥2-healthy floor), JSON-repair recovery, chairman provider-conflict detection, and cancellation. |
|
167 |
+ |
|
168 |
+## 🚢 Deployment (Linux VPS + Caddy) |
|
169 |
+ |
|
170 |
+Not serverless - the app is a long-running Node server designed for a VPS, with SSE debate streams that last minutes. |
|
171 |
+ |
|
172 |
+```bash |
|
173 |
+# On the VPS |
|
174 |
+cp .env.example .env # set ENCRYPTION_KEY, SITE_ADDRESS (your domain), auth vars |
|
175 |
+docker compose up -d # Postgres + app (standalone) + Caddy (auto-HTTPS) |
|
176 |
+docker compose exec app node node_modules/prisma/build/index.js db seed # optional demos |
|
177 |
+``` |
|
178 |
+ |
|
179 |
+- **Multi-stage `Dockerfile`** builds Next.js `standalone` output on `node:20-slim`; the entrypoint applies the schema (`prisma db push`) on boot. |
|
180 |
+- **`docker-compose.yml`** wires app + Postgres + Caddy with health checks and named volumes. |
|
181 |
+- **`Caddyfile`** reverse-proxies with `flush_interval -1` (no buffering) and long read/write windows so SSE streams flush immediately - automatic HTTPS via `SITE_ADDRESS`. |
|
182 |
+- **CI/CD** - [`ci.yml`](.github/workflows/ci.yml) runs lint · typecheck · test · build on every push; [`deploy.yml`](.github/workflows/deploy.yml) builds & pushes the image to GHCR and SSH-deploys to the VPS (`docker compose pull && up -d`). |
|
183 |
+ |
|
184 |
+### Environment |
|
185 |
+ |
|
186 |
+All variables are validated at boot with Zod ([`src/lib/env.ts`](src/lib/env.ts)); a misconfigured deploy fails loudly at startup. `ENCRYPTION_KEY` (32-byte base64) is the only secret required for public/demo mode. See [`.env.example`](.env.example). |
|
187 |
+ |
|
188 |
+## 🛠️ Tech stack |
|
189 |
+ |
|
190 |
+Next.js 15 (App Router) · TypeScript (strict) · Tailwind CSS + shadcn/ui · Vercel AI SDK · OpenRouter · PostgreSQL + Prisma · Auth.js (GitHub) · Zod · Vitest · Docker + Caddy · GitHub Actions. |
|
191 |
+ |
|
192 |
+--- |
|
193 |
+ |
|
194 |
+<div align="center"> |
|
195 |
+<sub>Built as a portfolio project. The debate engine is deliberately reusable - lift <code>src/core/</code> into a CLI or a different frontend and it just works.</sub> |
|
196 |
+</div> |