Developing WhyGraph¶
Working on WhyGraph itself (not using it) takes these make targets from a checkout:
| Command | Mode | What it runs |
|---|---|---|
make dev-local |
Development, native | The portal from your checkout on :8777, restarted on every backend change, plus the Vite dev server with hot reload on :5174, against the dev Postgres from make dev-db (started for you). Fastest; your IDE's debugger works. Keys and endpoints exported in your shell are ignored, as in a real install: enter keys under Settings. |
make dev-docker |
Development, in Docker | The same, inside the WhyGraph image through the real whygraph up shim, with your checkout mounted read-only over the installed package. |
make prod |
Production | The image built exactly like a release (pinned CodeGraph, the pyproject.toml version), run through the shim with no source mounted: what users get. |
make dev-production |
Development, native, production mode | The portal's production mode from your checkout on :8778, with Vite on :5173 in front of it, on its own whygraph_prod database in the make dev-db Postgres. For work on sign-in, organizations and the host checks, and the platform dev-local links to. See below. |
make check |
Before a pull request | Both ruff checks, pytest, the frontend typecheck / tests / build then the Playwright suite, and the release smoke test against a freshly built image. The four chains run at once, each into its own log; it prints one line per stage and the log of each one that failed. |
make check-fast |
While working | make check without the image build and the smoke test. |
dev-local and dev-docker run in the foreground: open http://localhost:5174, and press
Ctrl-C to stop everything. prod serves the built UI itself on http://127.0.0.1:8777.
The whygraph command from a checkout¶
To get the whygraph command with your edits live (for whygraph version, whygraph portal and the
stubs), install the checkout in editable mode. This is for development only; users install with
Docker (Installation).
--editable skips the reinstall on every change. To run the portal from a checkout, use
make dev-local or make dev-docker as above.
The portal database in development¶
The portal keeps its own data in Postgres, so every mode needs one:
make dev-localdepends onmake dev-db, which starts (or reuses) a container,whygraph-dev-postgres, laid out like the shim's database container (the same pinned image, read fromPOSTGRES_IMAGEincli/commands/install.py, a per-majorPGDATA, your user, the same health check) but published on127.0.0.1:55432(PG_DEV_PORT=...to change it) with the dev-only passwordwhygraph-dev. Its files are in~/.local/state/whygraph-dev/local/postgres, beside the dev data directory.make dev-dbwaits until it is healthy;make dev-localthen points the portal at it withWHYGRAPH_DATABASE_URL=postgresql+psycopg://whygraph:whygraph-dev@127.0.0.1:55432/whygraph.make dev-db-downstops and removes the container and keeps its files; delete the directory for a fresh database.make dev-dockerandmake prodgo through the real shim, so they get their own database container,whygraph-portal-dev-postgres/whygraph-portal-prod-postgres, which their Ctrl-Cwhygraph downstops with the portal.make e2estarts a throwaway Postgres (in memory) for its portal and removes it afterwards.
make dev-db and make dev-db-down are listed by make like every other helper.
Production mode in development¶
make dev-production runs the same native stack as dev-local with WHYGRAPH_MODE=production. It
differs from make prod, which is the released image in local mode (what a user installs), not
production mode:
- It uses a separate database,
whygraph_prod, created on demand in themake dev-dbcontainer, and its own data directory under~/.local/state/whygraph-dev/production, so it never touchesdev-local's state. No shared folders are given. - The base URL is
http://whygraph.localhost:5173- the Vite address, so the browser, the session cookie and the portal's host check all see one origin per host. Vite passes theHostheader through unchanged, which is what lets<org>.whygraph.localhost:5173reach the portal. Openhttp://whygraph.localhost:5173, notlocalhost: that one gets421. - GitHub is a fake. Sign-in and projects need a GitHub, so
make dev-productionalso starts a small fake one (tests/github_fake.py) onDEV_GITHUB_PORTthat serves both apps: the OAuth App and the GitHub App. It points everyWHYGRAPH_GITHUB_*variable at it, with a generated app key, client secrets and webhook secret kept under~/.local/state/whygraph-dev/production/github/(generating them needsopenssl). Nothing leaves your machine. Its sign-in page offers four users,ben,cy,deeandnofa(no two-factor authentication), to try members, roles and two-factor refusals. -
The fake's repositories. It serves two fixture repositories,
ben/demoandben/notes, as bare repositories in~/.local/state/whygraph-dev/production/github/repos, covered by installation100on Ben's account: sign in asbento import them. Its control routes change its state and send the matching signed webhook to the portal, so you can try the webhook paths without a real GitHub: -
A real GitHub is used only when every OAuth App and GitHub App variable is set, because the two apps share
WHYGRAPH_GITHUB_URLand a real one cannot be mixed with the fake; with none set the fake serves both, and a partial set stopsmake dev-productionbefore anything starts, naming the missing variables. Create a dev OAuth App and a dev GitHub App with the callback URLshttp://whygraph.localhost:5173/auth/githubandhttp://whygraph.localhost:5173/auth/github-app(see GitHub sign-in and The GitHub App), then copy.env.dev.exampleto.env.dev(gitignored) and fill in all seven variables.make dev-productionloads.env.dev(or the fileDEV_ENV_FILE=...names) when it exists; values in it win over your shell's. The same file can also holdDEV_FOLDERS(Sharing more folders). - Webhooks from a real GitHub cannot reach
whygraph.localhost, so relay them through a smee.io channel: set the dev app's webhook URL to the channel, then runnpx smee-client -u https://smee.io/<channel> -t http://whygraph.localhost:5173/github/webhookbesidemake dev-production. The relay goes through Vite, which proxies/githubto the portal and keeps the base host it requires. Without the relay, pushes still arrive through the hourly check and Scan now. - The first run prints a
Bootstrap secret:line in the terminal (the portal's log); the page athttp://whygraph.localhost:5173asks for it to create the first account. - Under the hood
make dev-productionrunsscripts/dev_portal.py --preserve-host --vite-port 5173: Vite then forwardsHostunchanged and only answers*.whygraph.localhost, and the script prints thewhygraph.localhostURL (the bare127.0.0.1:8778address is refused with421in production). - It runs beside
dev-local, which is what linking a checkout needs: the two differ in every port (:8778with Vite on:5173here,:8777with Vite on:5174there) and use different databases. See Linking the two dev portals. - Use Chromium or Firefox: they resolve every
*.localhostname to the loopback address, while Safari's handling of*.localhostvaries by version.
Linking the two dev portals¶
A platform project is linked from a local portal, so trying one out means running both:
make dev-production (the platform) in one terminal and make dev-local (the local portal) in
another. They no longer collide - the local portal's Vite is on :5174, the platform's on :5173.
- In
dev-production, sign in to the fake GitHub asben, importben/demoand scan it. - Open the project's Connect local WhyGraph card, press Change next to the port and enter
5174. In the dev loop that is Vite's port, not the portal's
8777, because the browser talks to Vite (it serves the/linkand/connect/callbackpages and proxies/apito the portal). - The button lands on the local portal's
/linkpage. Press Connect; because the platform started the link, it skips its Allow page and the browser comes back tohttp://127.0.0.1:5174/connect/callback. - Point the link at a checkout under
make dev-local's shared folder,~/.local/state/whygraph-dev/repos, whoseoriginnames the platform's repository (or which holds the commit the platform scanned). The fake GitHub serves git only to an installation token, so the quickest one is a clone of the platform's own server clone -~/.local/state/whygraph-dev/production/data/repos/<org>/<slug>- with itsoriginset tohttp://127.0.0.1:18767/ben/demo.git.
make dev-local exports the two switches this needs, because a platform on
http://whygraph.localhost:5173 is neither https nor a name every resolver maps to loopback:
WHYGRAPH_DEV_PLATFORM_HTTP=1- accept anhttpplatform, and only on a loopback host.NO_PROXY=.localhost- keep anHTTP_PROXYout of the portal's calls to the platform.
Neither is part of the supported interface; a real platform is always https.
make dev-docker cannot link: inside its container *.localhost names resolve to the container
itself, not to your dev-production. Link from make dev-local. make e2e runs the whole flow
(import on a platform, link, an MCP call, revoke, remove) against a fake GitHub, with no setup.
Which one to use¶
dev-localfor most work: the UI, routes, MCP tools, chat. A saved.pyfile undersrc/whygraph/restarts the portal within a few seconds (RELOAD=0turns that off); a saved.tsxfile hot-reloads in the browser.dev-localserves/mcp/<slug>/codeonly when acodegraphis on your hostPATH, and that may not be the pinned version, in which case the schema pin withholds the tools it does not recognize (the project's Overview says so). Point the MCP Inspector at it withmake inspect SLUG=<slug> MCP_PATH=code. A restart of the dev portal may leave acodegraph serve --mcpprocess behind; the next start sweeps it, or runpkill -f 'codegraph serve --mcp'.dev-dockerfor anything that behaves differently in the container: the shim (cli/commands/install.py), the Dockerfile, file ownership, paths, the environment a scan gets, and the in-imagegit/gh/codegraph. It rebuilds the image only when its inputs change (pyproject.toml,uv.lock, the Dockerfile,package-lock.json); everyday edits reach the container through the mount.dev-productionfor anything that only exists in production mode: accounts, sessions, organization hosts, the admin page, projects imported from GitHub and the webhook.make check-fastwhile working, andmake checkbefore pushing: its smoke test catches whatdev-localcannot see.
What stays isolated¶
Every mode keeps to itself, so none of them touches your own portal:
- its own container name (
whygraph-portal-dev,whygraph-portal-prod, andwhygraph-portal-smokefor the smoke test), data directory andHOME, all under~/.local/state/whygraph-dev($XDG_STATE_HOME/whygraph-devwhen that is set;DEV=...moves it). It is not under$TMPDIRon purpose: macOS deletes files there that were not opened for three days, which breaks a dev Postgres and loses the data directory'ssecret.key; - one scratch folder of repos,
~/.local/state/whygraph-dev/repos, shared by every mode and created offline on first use: a copy of your checkout (git clone --no-hardlinks, real history to dogfood) and a tiny three-commit demo repo. Delete one to get a fresh copy next time; one that is no longer a working repository (itsHEADnames no commit) is rebuilt the same way.
Your checkout itself is never added to a dev portal, so it never gets WhyGraph's hooks or markers - unless you share a folder that holds it (below).
dev-local, dev-docker and prod share port 8777, so run one of those three at a time.
Override it with DEV_PORT=.... dev-production uses :8778 for the portal and :5173 for Vite,
so it can run beside any of them.
Sharing more folders¶
To try a dev portal on repos outside the scratch folder, list their folders in DEV_FOLDERS in
.env.dev (template: .env.dev.example), :-separated:
make dev-local, make dev-docker and make prod share them beside ~/.local/state/whygraph-dev/repos;
make dev-production ignores them (production mode has no shared folders). For one run, pass
make dev-local DEV_FOLDERS=... instead; a non-empty value there or in your shell wins over the
file. A folder that is missing, relative, /, holds a , or ", or overlaps
~/.local/state/whygraph-dev stops the target before anything starts. The Docker modes match their shared
folders to the list on every run, so a folder you remove from it is no longer mounted. Under Docker
Desktop the folder must also be in its file-sharing settings (/Users is by default).
These are your real repos
Adding a project from such a folder writes into that repo what any portal does: the git hooks,
.whygraph/portal.env (naming the dev port, 8777), and the agent entries and files. If your own
portal (whygraph up) also serves that repo, whichever initialized it last owns those. To keep
your checkouts untouched, share a folder of throwaway clones instead.
Helpers¶
| Command | What it does |
|---|---|
make test |
pytest only, on every core (ARGS="-n0 tests/test_x.py" for one file, serially - a debugger needs -n0) |
make test-fast |
The tests that failed last time first, then the rest; stops at the first failure |
make e2e |
The Playwright suite against a throwaway portal and a fake scanner, both themes (ARGS="--project=light" for one) |
make audit |
The screenshot audit: walks every screen of both portals (local, production, linked) at desktop and phone width in light and dark, and writes PNGs, manifest.json and gaps.md under src/playground/e2e/.artifacts/audit/ (ARGS="--project=local" for one). About 30 minutes, run by hand before a merge, never in CI; see src/playground/e2e/audit/README.md |
make inspect SLUG=<slug> |
The MCP Inspector against a project's /mcp/<slug> on the running dev portal; add MCP_PATH=code for its CodeGraph endpoint, /mcp/<slug>/code |
make db SLUG=<slug> |
A DBGate viewer on :8081 for the project's WhyGraph database in the dev data directory (projects/local/<slug>/whygraph.db) and its repo's CodeGraph index, plus a Portal connection to the make dev-db Postgres (through host.docker.internal:55432). Copy docker-compose.example.yml to docker-compose.yml first; an older copy needs re-copying to get the history volume and the Portal connection |
make dev-db / make dev-db-down |
Start (and wait for) / stop the dev portal's Postgres; see above |
make image |
Build the image like the release does (IMAGE=... to retag) |
make docs / make docs-build |
Serve this site with live reload / build it strictly, as CI does |
The phone project¶
make e2e also has a phone Playwright project (390x844, light), run after light and production
(ARGS="--project=phone" for just that, which pulls its dependencies in). Its one spec,
e2e/tests/phone.spec.ts, opens the main routes of both portals and asserts that nothing scrolls
horizontally (except an element marked data-scroll-x, an opted-in scroller) and that every page has an
h1, visible or sr-only. A page not fixed yet is a test.fail() entry in the spec's
KNOWN_OVERFLOW, so fixing it turns the run red until the entry is removed.
Requirements and gotchas¶
- Node 22.12 or newer for the playground toolchain (
nvm use 22). - Docker for
dev-local(its Postgres),dev-docker,prod,make e2eandmake check. opensslformake e2eand formake dev-productionwith the fake GitHub: both generate the fake GitHub App's key and webhook secret with it.- The tests need a Postgres. The portal tests run against a real one: by default
pyteststarts a throwaway container of the pinned image for the session (on a random loopback port, in memory) and gives every test its own database, so plainuv run pytestneeds Docker. Without Docker, setWHYGRAPH_TEST_DATABASE_URLto an admin URL of a Postgres you run (the role needsCREATEDB), for examplepostgresql+psycopg://postgres:test@127.0.0.1:5432/postgres. CI starts the throwaway container like a laptop does.WHYGRAPH_TEST_POSTGRES_IMAGEreplaces the throwaway server's image, for a registry mirror. With neither Docker nor the URL, the portal tests fail with a message saying so - they are never skipped. - The tests run in parallel.
pytestruns with-n auto(pytest-xdist): one worker per core, or per performance core on a Mac, each with its own throwaway Postgres. Pass-n0to run serially, for example under a debugger. CI splits the suite into three shards by the durations stored in.test_durations; refresh that file withuv run pytest --store-durationsafter adding many slow tests (an unlisted test only counts as average). uv runsyncs first, which fails behind a TLS-intercepting proxy. Run the targets withUV_NO_SYNC=1 make ...(orUV_RUN="uv run --no-sync") there.- The Docker modes use two development-only shim settings,
WHYGRAPH_DEV_SRCandWHYGRAPH_PORTAL_NAME. They are not part of the supported interface.
Writing harness prompts¶
The agent, skill and rule files under src/whygraph/assets/ are prompts that a model reads and follows. New and changed prompt text should follow the small subset of ASD-STE100 (Simplified Technical English) below, so that a model has one way to read each instruction. Existing prompts were not rewritten to it: plan-tool.sh check applies the sentence rules to plan text, and a test enforces the glossary across the prompts.
- One action per sentence. Do not join two actions with "and then", "and also" or "then".
- At most 20 words per sentence. A backticked span counts as one word.
- One name for one thing. Use the same word each time, and do not swap in a synonym.
- Start each instruction with its verb. Write "Run the check", not "The check should be run".
- Keep the modality of the source. A rewrite keeps every "may", "might" and "can". A generator keeps the strength that its evidence states.
Use these words for the harness terms:
| Use | Do not use | Note |
|---|---|---|
| the user | the owner | The person who runs the slash command. |
| plan | plan markdown | The plan file and its contents. |
| slash command | entry point | caller and slash prompt stay allowed. |
| report | - | A subagent's result that has a heading. |
| reply | - | The text any agent returns. |
| halt | - | The implementor stops on a failure. |
| stop | - | A slash command ends its run. |
A change to a prompt lands in all four trees (claude-code, cursor, vscode, codex), so that test_body_parity stays true. test_harness_prompts_use_glossary_terms in tests/test_assets.py fails when an asset uses a banned term. The plan checker applies the same length and one-action rules to a plan.
Agent evals¶
scripts/agent_evals is a live check of the agent harness: the planner, implementor, evaluator and reviewer, run through their real slash commands. Each case runs claude -p against a fixture and judges only the end state: files, the reply and the session transcripts. It is not pytest. Its cases are listed in scripts/agent_evals/README.md.
It costs money, and it never runs in CI
Every case calls a paid model on your own claude login. A full suite costs about $14 to $16. Run it by hand before you trust a change to a harness prompt. CI never runs it, and scripts/ is linted but not collected by pytest.
- Cases with an
O,CorUprefix run in a throwaway clone ofscripts/agent_evals/fixtures/miniand need only theclaudelogin. - Cases with a
Pprefix run in a real repo, by default the cleanui-demorepo at~/.local/state/whygraph-dev/repos/ui-demo(--repo DIRto change it). They need the dev portal (make dev-local, port 8777) with that repo registered and scanned. APcase refuses a dirty work tree, works on the temporary brancheval-tmp, and restores the repo afterwards. Without the portal it prints a skip reason and spends nothing. - The ledger is one JSON line per run, with the case, the stage and the cost.
make agent-evalwrites it toplans/47-eval/spend.jsonl(AGENT_EVAL_LEDGERto change it) and caps the total at $50 (AGENT_EVAL_MAX_USD). Before each run, the sum of the ledger plus the run's budget is checked against the cap, and the run is refused when it would exceed it. A run with no cost in its result is charged its budget.
--dry-run loads every case and prints what would run or be skipped, and calls no model.
Estimate calibration¶
The Describe core / Describe all estimate (src/whygraph/portal/estimate.py) replays how a scan splits each commit into model calls. Its constants are measured on claude-sonnet-5-5, and each other model it knows has a profile in MODEL_PROFILES: how many input tokens and how much output it produces per Sonnet token on the same diffs. A model without a profile is estimated as Sonnet.
scripts/estimate_calibration.py measures a model. It samples waiting commits from a copy of a project's WhyGraph database, describes each one the way a scan does, and prints the estimate next to the real usage. It never writes a description, and its JSON output holds only counts.
SSL_CERT_FILE=/etc/ssl/cert.pem uv run python scripts/estimate_calibration.py \
--db ~/.local/state/whygraph-dev/local/data/projects/local/biome/whygraph.db \
--repo ~/.local/state/whygraph-dev/repos/biome \
--model anthropic:claude-sonnet-5-5 --model openai:gpt-4o \
--sample 60 --out /tmp/calibration.json
It costs money, and it never runs in CI
Without --yes the script only prints the sample and what it would cost. With --yes it calls each model, about $1 to $2 per model for 60 commits at Sonnet prices. Keys come from the environment.
To add a profile, run the new model and claude-sonnet-5-5 on the same sample, and divide the model's input and output tokens by Sonnet's. Then add the factors to MODEL_PROFILES and the sample to tests/fixtures/estimate_calibration_biome.json, so test_estimate_matches_each_measured_model keeps them within 5 %.