Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
126 lines (94 loc) · 5.39 KB

CONTRIBUTING.md

File metadata and controls

126 lines (94 loc) · 5.39 KB
Raw

Contributing

Thanks for your interest. Before opening a PR, please read the codegen note below — it's the single most important thing to know about this repo.

The grammar is generated, not hand-written

The PostgreSQL SQL grammar (postgres/grammar.js), the PL/pgSQL grammar (plpgsql/grammar.js), and several query files are generated from PostgreSQL's own Bison grammar (gram.y, pl_gram.y) and keyword lists. Direct edits to generated files will be lost the next time we regenerate — for example when PostgreSQL 19 ships and we re-run codegen against a new PG_SOURCE_DIR.

If a generated file is wrong or missing something, fix the generator, not the file.

What's generated vs. what's hand-written

File / directory Source of truth
postgres/grammar.js script/codegen.js (+ parse-gram-y.js, parse-kwlist.js)
postgres/src/grammar.json, postgres/src/parser.c tree-sitter generate postgres/grammar.js
postgres/queries/injections.scm script/generate-injections.js
plpgsql/grammar.js script/generate-plpgsql-grammar.js
plpgsql/src/grammar.json, plpgsql/src/parser.c tree-sitter generate (in plpgsql/)
postgres/queries/highlights.scm hand-written
postgres/test/corpus/*.txt hand-written
postgres/known-conflicts.json hand-curated
plpgsql/src/scanner.c hand-written
plpgsql/queries/*.scm hand-written
plpgsql/test/corpus/*.txt hand-written
bindings/** hand-written

Generated files start with a Generated by … header — when in doubt, check the file's first line.

Dev environment setup

You need:

  • Node.js (for the tree-sitter CLI)
  • just — brew install just
  • A PostgreSQL source checkout (for codegen only — you can run tests without it)
  • A Rust toolchain only if you're working on the Rust bindings

1. Get a PostgreSQL source checkout

We currently target PostgreSQL 19 beta 2 (REL_19_BETA2). Clone postgres anywhere on disk and check out the matching tag:

git clone https://github.com/postgres/postgres.git ~/Source/postgres
git -C ~/Source/postgres checkout REL_19_BETA2

Then point the codegen at it:

export PG_SOURCE_DIR=~/Source/postgres

Add the export to your shell profile so just recipes pick it up.

2. Install repo dependencies

npm install

Running codegen

Warning

Regenerating the postgres grammar needs a very large amount of RAM. The tree-sitter generate step for postgres/grammar.js builds GLR parse tables for ~17,000 states and peaks at roughly 67 GB of memory (measured with tree-sitter CLI 0.26.7; takes ~8 minutes on a 128 GB machine). On a typical 16–32 GB machine it will be OOM-killed. This is a known tree-sitter limitation, not something we can fix in this repo.

You almost never need to run it. The generated artifacts (postgres/src/parser.c etc.) are committed (via Git LFS). If your change touches script/codegen.js, run just codegen-postgres — it only runs the Node codegen (well under 100 MB) and updates postgres/grammar.js, so you can review the grammar diff. Open the PR with the grammar.js change and a maintainer will run the full just generate-postgres and push the regenerated artifacts to your branch.

The plpgsql grammar is small — just generate-plpgsql runs anywhere in under a second.

End-to-end (postgres grammar + injections + plpgsql parser):

just generate

Sub-recipes if you only need part of it:

just codegen-postgres         # postgres/grammar.js only (no parse-table build, low RAM)
just generate-postgres        # postgres/grammar.js + tree-sitter generate (~67 GB RAM)
just generate-injections      # postgres/queries/injections.scm
just generate-plpgsql         # tree-sitter generate in plpgsql/

# To regenerate plpgsql/grammar.js itself from PG source (rarely needed):
node script/generate-plpgsql-grammar.js "$PG_SOURCE_DIR"

Run the corpus tests (covers both grammars):

just test

Where to make common changes

  • SQL grammar → edit script/codegen.js (or its helpers parse-gram-y.js / parse-kwlist.js), then just codegen-postgres (or just generate-postgres if you have the RAM for it — see the warning above).
  • GLR conflicts → edit postgres/known-conflicts.json directly. Use postgres/harvest-conflicts.sh to discover new conflicts iteratively.
  • Language-injection queries → edit script/generate-injections.js, then just generate-injections. Don't touch postgres/queries/injections.scm.
  • Highlight queries → edit postgres/queries/highlights.scm or plpgsql/queries/highlights.scm directly.
  • PL/pgSQL grammar → edit script/generate-plpgsql-grammar.js, then run that script and just generate-plpgsql.
  • PL/pgSQL external scanner → edit plpgsql/src/scanner.c directly.
  • Tests → add corpus cases to postgres/test/corpus/*.txt or plpgsql/test/corpus/*.txt.

Validating before opening a PR

just generate     # or just codegen-postgres on a low-RAM machine
just test
git diff --stat   # confirm only files you intended to change moved

If just generate produces unexpected diffs in files you didn't touch, your PostgreSQL checkout is probably on a different revision than the project targets — confirm you're on REL_19_BETA2.