profileShare

rasmusjy / voicetask

Read-only snapshot

No repository description.

main default branch 105 files Expires Sep 13, 2026, 9:06 AM
PLAN.md 9,933 bytes
History ← Back to files

PLAN: VoiceTask

Stack

  • Node.js >= 20, TypeScript strict, npm.
  • Server: Fastify (fastify, @fastify/multipart, @fastify/static, @fastify/cors).
  • Client: Vite + React + TypeScript. No UI framework, plain CSS in one stylesheet.
  • LLM: @anthropic-ai/sdk. STT: OpenAI REST API via fetch inside the provider only (no OpenAI SDK dependency).
  • Validation/schemas: zod (shared between API validation and LLM structured output).
  • Tests: vitest. Typecheck: tsc --noEmit.
  • Local demo: npm run demo passes an explicit demo flag to the server. The flag is applied after .env is read, so it always forces deterministic mock providers, and it locks that process on mocks even if keys are saved from the setup screen while it runs.
  • Provider setup: keys can be entered in the app. Providers are resolved per request instead of once at boot, so a saved key takes effect without a restart, and a server with no key still starts and serves the setup screen.

No other runtime dependencies without a BLOCKED.md entry.

Layout

shared/          types + zod schemas (Session, Segment, Coverage, API payloads)
                 transcript.ts: formatTranscript(segments), renders plain text shared by
                 the transcript download route and the client's copy-to-clipboard button
server/
  index.ts       Fastify bootstrap, serves client build in prod
  routes/        sessions.ts, audio.ts, generate.ts, blockers.ts, fs.ts, export.ts, config.ts
  config/
    runtimeConfig.ts  provider mode status, .env merge/write, live switch of process env
  store/         sessionStore.ts (JSON files under data/sessions/<id>/session.json)
  providers/
    types.ts     SttProvider, InterviewLlm interfaces
    factory.ts   env-based selection, MOCK_PROVIDERS=1 forces mocks, lazy per-request wrappers
    sttOpenai.ts sttMock.ts
    llmAnthropic.ts llmMock.ts
  engine/
    coverage.ts  categories, state transitions
    interview.ts one turn: transcript + coverage in, InterviewTurn out
    summary.ts   running summary of segments older than the last 40
  generator/
    prompts.ts   per-file generation prompts
    generate.ts  orchestrates the 5 files + sources.json, backup logic
    provenance.ts marker validation (FR-013)
  blockers/
    parse.ts     BLOCKED.md parser
  exporter/
    zip.ts       dependency-free ZIP (store method) writer used by the export route
client/
  src/App.tsx    session list/create, interview view
  src/api.ts     typed fetch wrappers over shared types
  src/audio.ts   MediaRecorder push-to-talk
  src/share.ts   deterministic handoff and recommendation copy
  src/tts.ts     speechSynthesis wrapper with browser-locale voice selection
  src/labels.ts  plain-language copy for coverage category ids
  src/components/ Transcript, CoveragePanel, QuestionCard, GeneratePanel, FolderBrowser,
                   TranscriptExport, OutcomePreview, ProviderSetup

Product experience direction

  • Audience: a non-technical founder, product person, or stakeholder who has an idea but not a written build brief.
  • Home screen job: explain the outcome, remove setup anxiety, and start one interview.
  • Visual system: cool cloud canvas, white surfaces, dark navy type, signal indigo actions, coral voice accents, and green completion states. Display type is Sora, body type is DM Sans, and transcript source markers use a system monospace face.
  • Layout: a clear two-column first screen pairs the product promise with the create form. A voice-to-requirement preview is the signature element and shows a spoken thought becoming a source-backed requirement.
  • Interaction: familiar cards, labeled actions, visible focus, restrained motion, and reduced-motion support. Progress and recording state never rely on color alone.
  • Accessibility: question updates use a polite live region. Custom controls expose focus, and modal folder browsing contains and restores keyboard focus.

Coverage categories (fixed, order matters for tie-breaking)

  1. goal (what and why), 2. users, 3. core-flow, 4. data, 5. integrations,
  2. edge-cases, 7. constraints (stack, platform, performance), 8. non-goals, 9. verification (what "done" looks like).

Key interfaces (shared/)

interface Segment { id: string; ts: string; speaker: "user" | "interviewer"; text: string }
type CoverageLevel = "missing" | "partial" | "clear"
type Coverage = Record<CategoryId, CoverageLevel>
interface Session {
  id: string; name: string; targetDir: string; createdAt: string;
  segments: Segment[]; coverage: Coverage; summary: string;
  status: "interviewing" | "done"; openBlockers: string[]
}
interface InterviewTurn {
  coverage: Coverage; nextQuestion: string;
  contradiction: { segmentIds: string[]; description: string } | null;
  done: boolean; summaryUpdate: string | null
}

InterviewTurn is also the zod schema used as the LLM structured output format.

Anthropic API usage (llmAnthropic.ts)

  • Model from ANTHROPIC_MODEL, default claude-opus-4-8. Key from ANTHROPIC_API_KEY.
  • Interview turn: client.messages.parse() with output_config: { format: zodOutputFormat(InterviewTurnSchema) }, thinking: { type: "adaptive" }, max_tokens: 4096. Input: system prompt (interviewer persona + category definitions + rules from SPEC FR-005..FR-008), then summary + last 40 segments + current coverage as the user message. Do not set temperature (removed on this model family).
  • Spec pack generation: one call per output file (5 calls), plain text output, client.messages.stream() with finalMessage(), max_tokens: 32000. Each prompt receives the full transcript with segment ids and instructs: English output, provenance markers [S<n>] on every FR (SPEC.md call only), verification commands per task (TASKS.md call only).
  • Schema-mismatch handling per SPEC edge cases: one retry, then error. Use the SDK's typed error classes.

STT provider (sttOpenai.ts)

  • POST https://api.openai.com/v1/audio/transcriptions, model from STT_MODEL (default gpt-4o-mini-transcribe), key from OPENAI_API_KEY, multipart upload of the webm blob. Returns plain text.
  • Reject transcripts that are empty after trimming (maps to the empty-audio edge case).

Mocks (sttMock.ts, llmMock.ts)

  • sttMock: returns "mock transcript <n>" with an incrementing counter, or, if the uploaded "audio" buffer is valid UTF-8 text, echoes it back (lets tests inject specific answers through the audio path).
  • llmMock interview: deterministic script keyed on turn count; marks one category clear per turn in the fixed category order, asks a canned question about the next category, returns done: true when all are clear or when the last user segment is exactly "done".
  • llmMock generation: emits minimal valid files; SPEC.md contains two FRs, the first with a valid [S1] marker and the second with a bogus [S999] marker so the provenance validator (FR-013) is exercised end to end.

API routes

POST /api/sessions                {name, targetDir} -> Session
GET  /api/sessions                -> Session[] (id, name, status only)
GET  /api/sessions/:id            -> Session
POST /api/sessions/:id/audio      multipart audio -> {segment, turn: InterviewTurn}
POST /api/sessions/:id/answer     {text} -> {segment, turn: InterviewTurn}
POST /api/sessions/:id/generate   {overwrite?: boolean} -> {files: string[], warnings: string[]}
POST /api/sessions/:id/blockers   {} -> {questions: string[]}  (reads <targetDir>/BLOCKED.md)
GET  /api/fs/browse               ?path=<abs path, default home dir> -> {path, parent, directories: [{name, path}]}
POST /api/fs/mkdir                {path, name} -> {path}  (creates one subdirectory)
GET  /api/sessions/:id/export/spec-pack.zip  -> application/zip download of the generated spec pack
HEAD /api/sessions/:id/export/spec-pack.zip  -> 200 when a generated pack exists, otherwise the same 404 as GET
GET  /api/sessions/:id/export/transcript.md  -> text/markdown download of formatTranscript(session.segments)

All request/response bodies validated with the shared zod schemas. Errors: {error: string} with appropriate status codes.

Decisions already made (do not revisit)

  • Fastify over Express: built-in schema validation hooks and multipart support.
  • JSON file storage over SQLite: single user, small data, trivially inspectable.
  • One LLM call per generated file instead of one giant call: keeps each output small enough to be reliable and lets provenance validation run per file.
  • Browser TTS over API TTS: zero cost and zero latency-sensitive infra for v1.
  • The generated spec pack intentionally mirrors the structure of this very spec pack (SPEC/PLAN/TASKS/VERIFICATION/HANDOFF); templates in generator/prompts.ts should be derived from these files.
  • Zip export is hand-rolled (exporter/zip.ts, STORE method only, no compression) instead of adding a zip dependency: the format is small and well-specified, and it keeps the "no dependency without a BLOCKED.md entry" rule intact for a feature this contained.
  • /api/fs/browse and /api/fs/mkdir expose the local filesystem over HTTP with no auth check beyond what the rest of the app already assumes (single local user, no auth by design). This is acceptable only because the app is local-only per the Out of scope section; it must never ship if that decision changes.
  • Coverage category labels shown in the UI are looked up from client/src/labels.ts, kept separate from the CategoryId values in shared/types.ts so the wire format/category ids never change, only the display text.
  • Product sharing remains user-initiated. The handoff message contains only project-specific guidance. The separate recommendation action contains the public VoiceTask link and never modifies generated files.
  • Pack readiness is derived from the existing zip export route with a HEAD request, rather than adding duplicate persisted state to the session.