Skip to content

Developing WhyGraph

Working on WhyGraph itself (not using it) takes these make targets from a checkout:

Command Mode What it runs
make dev-local Development, native The portal from your checkout on :8777, restarted on every backend change, plus the Vite dev server with hot reload on :5174, against the dev Postgres from make dev-db (started for you). Fastest; your IDE's debugger works. Keys and endpoints exported in your shell are ignored, as in a real install: enter keys under Settings.
make dev-docker Development, in Docker The same, inside the WhyGraph image through the real whygraph up shim, with your checkout mounted read-only over the installed package.
make prod Production The image built exactly like a release (pinned CodeGraph, the pyproject.toml version), run through the shim with no source mounted: what users get.
make dev-production Development, native, production mode The portal's production mode from your checkout on :8778, with Vite on :5173 in front of it, on its own whygraph_prod database in the make dev-db Postgres. For work on sign-in, organizations and the host checks, and the platform dev-local links to. See below.
make check Before a pull request Both ruff checks, pytest, the frontend typecheck / tests / build then the Playwright suite, and the release smoke test against a freshly built image. The four chains run at once, each into its own log; it prints one line per stage and the log of each one that failed.
make check-fast While working make check without the image build and the smoke test.

dev-local and dev-docker run in the foreground: open http://localhost:5174, and press Ctrl-C to stop everything. prod serves the built UI itself on http://127.0.0.1:8777.

The whygraph command from a checkout

To get the whygraph command with your edits live (for whygraph version, whygraph portal and the stubs), install the checkout in editable mode. This is for development only; users install with Docker (Installation).

git clone https://github.com/mtrdesign/whygraph.git
uv tool install --editable ./whygraph

--editable skips the reinstall on every change. To run the portal from a checkout, use make dev-local or make dev-docker as above.

The portal database in development

The portal keeps its own data in Postgres, so every mode needs one:

  • make dev-local depends on make dev-db, which starts (or reuses) a container, whygraph-dev-postgres, laid out like the shim's database container (the same pinned image, read from POSTGRES_IMAGE in cli/commands/install.py, a per-major PGDATA, your user, the same health check) but published on 127.0.0.1:55432 (PG_DEV_PORT=... to change it) with the dev-only password whygraph-dev. Its files are in ~/.local/state/whygraph-dev/local/postgres, beside the dev data directory. make dev-db waits until it is healthy; make dev-local then points the portal at it with WHYGRAPH_DATABASE_URL=postgresql+psycopg://whygraph:whygraph-dev@127.0.0.1:55432/whygraph. make dev-db-down stops and removes the container and keeps its files; delete the directory for a fresh database.
  • make dev-docker and make prod go through the real shim, so they get their own database container, whygraph-portal-dev-postgres / whygraph-portal-prod-postgres, which their Ctrl-C whygraph down stops with the portal.
  • make e2e starts a throwaway Postgres (in memory) for its portal and removes it afterwards.

make dev-db and make dev-db-down are listed by make like every other helper.

Production mode in development

make dev-production runs the same native stack as dev-local with WHYGRAPH_MODE=production. It differs from make prod, which is the released image in local mode (what a user installs), not production mode:

  • It uses a separate database, whygraph_prod, created on demand in the make dev-db container, and its own data directory under ~/.local/state/whygraph-dev/production, so it never touches dev-local's state. No shared folders are given.
  • The base URL is http://whygraph.localhost:5173 - the Vite address, so the browser, the session cookie and the portal's host check all see one origin per host. Vite passes the Host header through unchanged, which is what lets <org>.whygraph.localhost:5173 reach the portal. Open http://whygraph.localhost:5173, not localhost: that one gets 421.
  • GitHub is a fake. Sign-in and projects need a GitHub, so make dev-production also starts a small fake one (tests/github_fake.py) on DEV_GITHUB_PORT that serves both apps: the OAuth App and the GitHub App. It points every WHYGRAPH_GITHUB_* variable at it, with a generated app key, client secrets and webhook secret kept under ~/.local/state/whygraph-dev/production/github/ (generating them needs openssl). Nothing leaves your machine. Its sign-in page offers four users, ben, cy, dee and nofa (no two-factor authentication), to try members, roles and two-factor refusals.
  • The fake's repositories. It serves two fixture repositories, ben/demo and ben/notes, as bare repositories in ~/.local/state/whygraph-dev/production/github/repos, covered by installation 100 on Ben's account: sign in as ben to import them. Its control routes change its state and send the matching signed webhook to the portal, so you can try the webhook paths without a real GitHub:

    curl -X POST http://127.0.0.1:18767/_fake/push -d '{"repo": "ben/demo"}'          # a commit + a push
    curl -X POST http://127.0.0.1:18767/_fake/uninstall -d '{"installation": 100}'    # access lost
    curl -X POST http://127.0.0.1:18767/_fake/remove-repo -d '{"repo": "ben/demo"}'   # access lost
    
  • A real GitHub is used only when every OAuth App and GitHub App variable is set, because the two apps share WHYGRAPH_GITHUB_URL and a real one cannot be mixed with the fake; with none set the fake serves both, and a partial set stops make dev-production before anything starts, naming the missing variables. Create a dev OAuth App and a dev GitHub App with the callback URLs http://whygraph.localhost:5173/auth/github and http://whygraph.localhost:5173/auth/github-app (see GitHub sign-in and The GitHub App), then copy .env.dev.example to .env.dev (gitignored) and fill in all seven variables. make dev-production loads .env.dev (or the file DEV_ENV_FILE=... names) when it exists; values in it win over your shell's. The same file can also hold DEV_FOLDERS (Sharing more folders).

  • Webhooks from a real GitHub cannot reach whygraph.localhost, so relay them through a smee.io channel: set the dev app's webhook URL to the channel, then run npx smee-client -u https://smee.io/<channel> -t http://whygraph.localhost:5173/github/webhook beside make dev-production. The relay goes through Vite, which proxies /github to the portal and keeps the base host it requires. Without the relay, pushes still arrive through the hourly check and Scan now.
  • The first run prints a Bootstrap secret: line in the terminal (the portal's log); the page at http://whygraph.localhost:5173 asks for it to create the first account.
  • Under the hood make dev-production runs scripts/dev_portal.py --preserve-host --vite-port 5173: Vite then forwards Host unchanged and only answers *.whygraph.localhost, and the script prints the whygraph.localhost URL (the bare 127.0.0.1:8778 address is refused with 421 in production).
  • It runs beside dev-local, which is what linking a checkout needs: the two differ in every port (:8778 with Vite on :5173 here, :8777 with Vite on :5174 there) and use different databases. See Linking the two dev portals.
  • Use Chromium or Firefox: they resolve every *.localhost name to the loopback address, while Safari's handling of *.localhost varies by version.

Linking the two dev portals

A platform project is linked from a local portal, so trying one out means running both: make dev-production (the platform) in one terminal and make dev-local (the local portal) in another. They no longer collide - the local portal's Vite is on :5174, the platform's on :5173.

  1. In dev-production, sign in to the fake GitHub as ben, import ben/demo and scan it.
  2. Open the project's Connect local WhyGraph card, press Change next to the port and enter 5174. In the dev loop that is Vite's port, not the portal's 8777, because the browser talks to Vite (it serves the /link and /connect/callback pages and proxies /api to the portal).
  3. The button lands on the local portal's /link page. Press Connect; because the platform started the link, it skips its Allow page and the browser comes back to http://127.0.0.1:5174/connect/callback.
  4. Point the link at a checkout under make dev-local's shared folder, ~/.local/state/whygraph-dev/repos, whose origin names the platform's repository (or which holds the commit the platform scanned). The fake GitHub serves git only to an installation token, so the quickest one is a clone of the platform's own server clone - ~/.local/state/whygraph-dev/production/data/repos/<org>/<slug> - with its origin set to http://127.0.0.1:18767/ben/demo.git.

make dev-local exports the two switches this needs, because a platform on http://whygraph.localhost:5173 is neither https nor a name every resolver maps to loopback:

  • WHYGRAPH_DEV_PLATFORM_HTTP=1 - accept an http platform, and only on a loopback host.
  • NO_PROXY=.localhost - keep an HTTP_PROXY out of the portal's calls to the platform.

Neither is part of the supported interface; a real platform is always https.

make dev-docker cannot link: inside its container *.localhost names resolve to the container itself, not to your dev-production. Link from make dev-local. make e2e runs the whole flow (import on a platform, link, an MCP call, revoke, remove) against a fake GitHub, with no setup.

Which one to use

  • dev-local for most work: the UI, routes, MCP tools, chat. A saved .py file under src/whygraph/ restarts the portal within a few seconds (RELOAD=0 turns that off); a saved .tsx file hot-reloads in the browser. dev-local serves /mcp/<slug>/code only when a codegraph is on your host PATH, and that may not be the pinned version, in which case the schema pin withholds the tools it does not recognize (the project's Overview says so). Point the MCP Inspector at it with make inspect SLUG=<slug> MCP_PATH=code. A restart of the dev portal may leave a codegraph serve --mcp process behind; the next start sweeps it, or run pkill -f 'codegraph serve --mcp'.
  • dev-docker for anything that behaves differently in the container: the shim (cli/commands/install.py), the Dockerfile, file ownership, paths, the environment a scan gets, and the in-image git / gh / codegraph. It rebuilds the image only when its inputs change (pyproject.toml, uv.lock, the Dockerfile, package-lock.json); everyday edits reach the container through the mount.
  • dev-production for anything that only exists in production mode: accounts, sessions, organization hosts, the admin page, projects imported from GitHub and the webhook.
  • make check-fast while working, and make check before pushing: its smoke test catches what dev-local cannot see.

What stays isolated

Every mode keeps to itself, so none of them touches your own portal:

  • its own container name (whygraph-portal-dev, whygraph-portal-prod, and whygraph-portal-smoke for the smoke test), data directory and HOME, all under ~/.local/state/whygraph-dev ($XDG_STATE_HOME/whygraph-dev when that is set; DEV=... moves it). It is not under $TMPDIR on purpose: macOS deletes files there that were not opened for three days, which breaks a dev Postgres and loses the data directory's secret.key;
  • one scratch folder of repos, ~/.local/state/whygraph-dev/repos, shared by every mode and created offline on first use: a copy of your checkout (git clone --no-hardlinks, real history to dogfood) and a tiny three-commit demo repo. Delete one to get a fresh copy next time; one that is no longer a working repository (its HEAD names no commit) is rebuilt the same way.

Your checkout itself is never added to a dev portal, so it never gets WhyGraph's hooks or markers - unless you share a folder that holds it (below).

dev-local, dev-docker and prod share port 8777, so run one of those three at a time. Override it with DEV_PORT=.... dev-production uses :8778 for the portal and :5173 for Vite, so it can run beside any of them.

Sharing more folders

To try a dev portal on repos outside the scratch folder, list their folders in DEV_FOLDERS in .env.dev (template: .env.dev.example), :-separated:

DEV_FOLDERS=$HOME/Projects/demos:$HOME/work/clones

make dev-local, make dev-docker and make prod share them beside ~/.local/state/whygraph-dev/repos; make dev-production ignores them (production mode has no shared folders). For one run, pass make dev-local DEV_FOLDERS=... instead; a non-empty value there or in your shell wins over the file. A folder that is missing, relative, /, holds a , or ", or overlaps ~/.local/state/whygraph-dev stops the target before anything starts. The Docker modes match their shared folders to the list on every run, so a folder you remove from it is no longer mounted. Under Docker Desktop the folder must also be in its file-sharing settings (/Users is by default).

These are your real repos

Adding a project from such a folder writes into that repo what any portal does: the git hooks, .whygraph/portal.env (naming the dev port, 8777), and the agent entries and files. If your own portal (whygraph up) also serves that repo, whichever initialized it last owns those. To keep your checkouts untouched, share a folder of throwaway clones instead.

Helpers

Command What it does
make test pytest only, on every core (ARGS="-n0 tests/test_x.py" for one file, serially - a debugger needs -n0)
make test-fast The tests that failed last time first, then the rest; stops at the first failure
make e2e The Playwright suite against a throwaway portal and a fake scanner, both themes (ARGS="--project=light" for one)
make audit The screenshot audit: walks every screen of both portals (local, production, linked) at desktop and phone width in light and dark, and writes PNGs, manifest.json and gaps.md under src/playground/e2e/.artifacts/audit/ (ARGS="--project=local" for one). About 30 minutes, run by hand before a merge, never in CI; see src/playground/e2e/audit/README.md
make inspect SLUG=<slug> The MCP Inspector against a project's /mcp/<slug> on the running dev portal; add MCP_PATH=code for its CodeGraph endpoint, /mcp/<slug>/code
make db SLUG=<slug> A DBGate viewer on :8081 for the project's WhyGraph database in the dev data directory (projects/local/<slug>/whygraph.db) and its repo's CodeGraph index, plus a Portal connection to the make dev-db Postgres (through host.docker.internal:55432). Copy docker-compose.example.yml to docker-compose.yml first; an older copy needs re-copying to get the history volume and the Portal connection
make dev-db / make dev-db-down Start (and wait for) / stop the dev portal's Postgres; see above
make image Build the image like the release does (IMAGE=... to retag)
make docs / make docs-build Serve this site with live reload / build it strictly, as CI does

The phone project

make e2e also has a phone Playwright project (390x844, light), run after light and production (ARGS="--project=phone" for just that, which pulls its dependencies in). Its one spec, e2e/tests/phone.spec.ts, opens the main routes of both portals and asserts that nothing scrolls horizontally (except an element marked data-scroll-x, an opted-in scroller) and that every page has an h1, visible or sr-only. A page not fixed yet is a test.fail() entry in the spec's KNOWN_OVERFLOW, so fixing it turns the run red until the entry is removed.

Requirements and gotchas

  • Node 22.12 or newer for the playground toolchain (nvm use 22).
  • Docker for dev-local (its Postgres), dev-docker, prod, make e2e and make check.
  • openssl for make e2e and for make dev-production with the fake GitHub: both generate the fake GitHub App's key and webhook secret with it.
  • The tests need a Postgres. The portal tests run against a real one: by default pytest starts a throwaway container of the pinned image for the session (on a random loopback port, in memory) and gives every test its own database, so plain uv run pytest needs Docker. Without Docker, set WHYGRAPH_TEST_DATABASE_URL to an admin URL of a Postgres you run (the role needs CREATEDB), for example postgresql+psycopg://postgres:test@127.0.0.1:5432/postgres. CI starts the throwaway container like a laptop does. WHYGRAPH_TEST_POSTGRES_IMAGE replaces the throwaway server's image, for a registry mirror. With neither Docker nor the URL, the portal tests fail with a message saying so - they are never skipped.
  • The tests run in parallel. pytest runs with -n auto (pytest-xdist): one worker per core, or per performance core on a Mac, each with its own throwaway Postgres. Pass -n0 to run serially, for example under a debugger. CI splits the suite into three shards by the durations stored in .test_durations; refresh that file with uv run pytest --store-durations after adding many slow tests (an unlisted test only counts as average).
  • uv run syncs first, which fails behind a TLS-intercepting proxy. Run the targets with UV_NO_SYNC=1 make ... (or UV_RUN="uv run --no-sync") there.
  • The Docker modes use two development-only shim settings, WHYGRAPH_DEV_SRC and WHYGRAPH_PORTAL_NAME. They are not part of the supported interface.

Writing harness prompts

The agent, skill and rule files under src/whygraph/assets/ are prompts that a model reads and follows. New and changed prompt text should follow the small subset of ASD-STE100 (Simplified Technical English) below, so that a model has one way to read each instruction. Existing prompts were not rewritten to it: plan-tool.sh check applies the sentence rules to plan text, and a test enforces the glossary across the prompts.

  • One action per sentence. Do not join two actions with "and then", "and also" or "then".
  • At most 20 words per sentence. A backticked span counts as one word.
  • One name for one thing. Use the same word each time, and do not swap in a synonym.
  • Start each instruction with its verb. Write "Run the check", not "The check should be run".
  • Keep the modality of the source. A rewrite keeps every "may", "might" and "can". A generator keeps the strength that its evidence states.

Use these words for the harness terms:

Use Do not use Note
the user the owner The person who runs the slash command.
plan plan markdown The plan file and its contents.
slash command entry point caller and slash prompt stay allowed.
report - A subagent's result that has a heading.
reply - The text any agent returns.
halt - The implementor stops on a failure.
stop - A slash command ends its run.

A change to a prompt lands in all four trees (claude-code, cursor, vscode, codex), so that test_body_parity stays true. test_harness_prompts_use_glossary_terms in tests/test_assets.py fails when an asset uses a banned term. The plan checker applies the same length and one-action rules to a plan.

Agent evals

scripts/agent_evals is a live check of the agent harness: the planner, implementor, evaluator and reviewer, run through their real slash commands. Each case runs claude -p against a fixture and judges only the end state: files, the reply and the session transcripts. It is not pytest. Its cases are listed in scripts/agent_evals/README.md.

make agent-eval ARGS="--cases O1 --stage 1"

It costs money, and it never runs in CI

Every case calls a paid model on your own claude login. A full suite costs about $14 to $16. Run it by hand before you trust a change to a harness prompt. CI never runs it, and scripts/ is linted but not collected by pytest.

  • Cases with an O, C or U prefix run in a throwaway clone of scripts/agent_evals/fixtures/mini and need only the claude login.
  • Cases with a P prefix run in a real repo, by default the clean ui-demo repo at ~/.local/state/whygraph-dev/repos/ui-demo (--repo DIR to change it). They need the dev portal (make dev-local, port 8777) with that repo registered and scanned. A P case refuses a dirty work tree, works on the temporary branch eval-tmp, and restores the repo afterwards. Without the portal it prints a skip reason and spends nothing.
  • The ledger is one JSON line per run, with the case, the stage and the cost. make agent-eval writes it to plans/47-eval/spend.jsonl (AGENT_EVAL_LEDGER to change it) and caps the total at $50 (AGENT_EVAL_MAX_USD). Before each run, the sum of the ledger plus the run's budget is checked against the cap, and the run is refused when it would exceed it. A run with no cost in its result is charged its budget.

--dry-run loads every case and prints what would run or be skipped, and calls no model.

Estimate calibration

The Describe core / Describe all estimate (src/whygraph/portal/estimate.py) replays how a scan splits each commit into model calls. Its constants are measured on claude-sonnet-5-5, and each other model it knows has a profile in MODEL_PROFILES: how many input tokens and how much output it produces per Sonnet token on the same diffs. A model without a profile is estimated as Sonnet.

scripts/estimate_calibration.py measures a model. It samples waiting commits from a copy of a project's WhyGraph database, describes each one the way a scan does, and prints the estimate next to the real usage. It never writes a description, and its JSON output holds only counts.

SSL_CERT_FILE=/etc/ssl/cert.pem uv run python scripts/estimate_calibration.py \
    --db ~/.local/state/whygraph-dev/local/data/projects/local/biome/whygraph.db \
    --repo ~/.local/state/whygraph-dev/repos/biome \
    --model anthropic:claude-sonnet-5-5 --model openai:gpt-4o \
    --sample 60 --out /tmp/calibration.json

It costs money, and it never runs in CI

Without --yes the script only prints the sample and what it would cost. With --yes it calls each model, about $1 to $2 per model for 60 commits at Sonnet prices. Keys come from the environment.

To add a profile, run the new model and claude-sonnet-5-5 on the same sample, and divide the model's input and output tokens by Sonnet's. Then add the factors to MODEL_PROFILES and the sample to tests/fixtures/estimate_calibration_biome.json, so test_estimate_matches_each_measured_model keeps them within 5 %.