Percolate An AI workflow engine in Postgres GitHub llms.txt

Install

There are three ways in, they install the same thing, and you can move between them later.

Docker Compose

What we are trying to do here is get a working database and services with nothing to compile and no ordering to get right.

curl -fsSL https://raw.githubusercontent.com/Percolation-Labs/get-percolate/main/compose/docker-compose.yml \
  -o docker-compose.yml
docker compose up -d

If you have a model key, put it in a .env beside the file before that up, because each container reads its environment once, when it starts:

echo 'OPENAI_API_KEY=sk-...' > .env      # agent turns, and the sample's embeddings
docker compose up -d                     # again, if the stack was already running

One key covers both halves on this path: the Agent Runtime reads OPENAI_API_KEY, and the workers' embedding credential LLM_API_KEY falls back to it. A key exported in some other shell after the stack is up reaches neither. Nothing else needs one — the database, workflows, graph and text search all run without it. The same .env moves ports that are already taken, most often 5432 by a local Postgres: P8_PG_PORT=5433, and P8_REST_PORT, P8_AGENT_PORT and P8_CONTENT_PORT for the others.

Six services come up, and that is a list of roles rather than a recommended process count: PostgREST for the REST surface, a worker for steps that leave the machine, an ingest-worker for reading uploaded files, the Content Server for uploads, the Agent Runtime for model turns, and MinIO for bytes. Three of those are one image under three commands, so a real deployment usually runs fewer — often just a worker, since sql and p8ql steps need no process at all — or many more, a pool per queue. The one pairing not to collapse is the two workers: --queue takes a single queue, so merging them means choosing which of outbound calls and ingestion silently stops happening.

The db image is pinned to a version, so docker compose up -d gives every reader the same database rather than whatever their machine last pulled. Updating therefore means changing the tag, not pulling a moving one:

# edit compose/docker-compose.yml: percolate-postgres:19-<old> -> :19-<new>
docker compose up -d

That is the deliberate half of a trade. A floating :19 makes "works on my machine" literally true and unfalsifiable: a stale local copy serves an older extension while every file in the repository says otherwise, and the bug report it produces is about a defect that is already fixed. ci/versions.py enforces the pinned tag against versions.toml, so the number moves in one place and every file that repeats it moves with it.

The new image brings the new extension files; one more statement installs them into the database that already exists:

set role app_owner;
alter extension percolate update;

set role app_owner is not decoration. An extension update creates its new objects as whoever runs it, so running this as a superuser would hand every new table a superuser owner and leave the policies on it inert — the one failure this collection is built to refuse. Updating as app_owner is what keeps ownership and row-level security exactly as a fresh install has them.

Every release ships percolate--<old>--0.1.6.sql for each version published since 0.1.1 — a 0.1.0 database has no upgrade path and is replaced, not updated — and it is the whole schema replayed: every statement in it is written to survive being applied twice, and build-sql.py refuses one that is not. An upgraded database was checked against a fresh install of the same version object by object — 629 of them, identical.

Give it a minute, then ask it whether it is really there. If you have no psql on the host — and nothing above installed one — the database container has its own:

docker compose exec db psql -U p8 -d percolate -c "select * from workflow.compiler_capabilities()"
# or, with a local client:
psql postgres://p8:p8@localhost:5432/percolate -c "select * from workflow.compiler_capabilities()"
docker compose exec db psql …, on a fresh volume
{"accepts": {"sql": true, "p8ql": true, "rest": true, "work": true,
             "agent": true, "embed": true, "timer": true, "matrix": true,
             "signal": true, "batching": true, "rate_key": true,
             "continue_on": true, "sub_workflow": true,
             "output_schema": true, "p8ql_vector_desugar": true},
 "missing": [], "available": true}

And ask it what it was built from, which is a different question:

select * from percolate_build();
 component | version | installed_version | commit_sha | artifact                              | built_at             | stamp_current
-----------+---------+-------------------+------------+---------------------------------------+----------------------+---------------
 parser    | 0.1.6   | 0.1.6   |            | v0.1.6, amd64 .so sha256:9c2f…, arm64 .so sha256:4be1… | 2026-09-03 16:04:00Z | t
 schema    | 0.1.6   | 0.1.6   | 9f832b1    |                                       | 2026-09-03 19:52:00Z | t

The two rows answer differently, because the two halves are made differently. The schema is generated from a commit, so commit_sha identifies it and artifact is empty. The compiled parser is downloaded from a release — no commit of the schema's repository produced that binary — so artifact names the release and both architectures' .so digests, and commit_sha is empty rather than filled with the nearest plausible commit. A -dirty suffix on a commit means that build came from a working tree that matched no commit at all.

stamp_current compares what the image stamped against what the database is running now. It goes f after ALTER EXTENSION percolate UPDATE, which moves the database without touching the stamp — so the row still describes the image you started from and no longer describes this database. That is the case worth catching, because everything else in the row is then about the wrong build.

It answers from the image and the Helm chart; an install from the release files returns no rows, because the build is recorded when the image is built and the files carry no such record.

Why it works — the database installs itself, and `missing` is the field to read

Two images and nothing to compile. The bootstrap is baked into the image, so the database installs itself the first time it starts — there is no init directory to fetch alongside the compose file and no ordering for you to get right.

missing is what to look at rather than a version string. The compiled parser and the SQL schema ship independently and will eventually disagree, so this probes the installed build with one canary per feature. If it lists anything you have version skew, which otherwise shows up looking like a syntax error in a workflow that is not wrong.

percolate_build() answers the other axis, and the two are not interchangeable. Capabilities detect the parser and the schema drifting apart from each other; both halves can agree perfectly and both be several commits behind the source that produced them. That second case cost a day once — a test suite failing on two parser bugs that were already fixed in the tree, against a .so compiled before the fix — so the build now says what it came from rather than leaving it to be inferred.

Helm

What we are trying to do here is install the whole stack into a cluster, with passwords we generate ourselves.

cat > percolate-values.yaml <<EOF
secrets:
  postgresPassword: $(openssl rand -hex 24)
  authenticatorPassword: $(openssl rand -hex 24)
  workerPassword: $(openssl rand -hex 24)
  jwtSecret: $(openssl rand -hex 32)
  s3Key: $(openssl rand -hex 16)
  s3Secret: $(openssl rand -hex 24)
  openaiApiKey: "${OPENAI_API_KEY:-}"   # the Agent Runtime's model key
  llmApiKey: "${OPENAI_API_KEY:-}"      # what embedding steps resolve
EOF
helm install percolate oci://ghcr.io/percolation-labs/charts/percolate \
  -n percolate --create-namespace -f percolate-values.yaml

You get the database, PostgREST, the three services, two worker pools — http and ingest — and MinIO.

Keep percolate-values.yaml. Upgrades take it with -f; generating the passwords again writes new ones into the Secret while the database keeps the ones it started with, and every service then fails to log in. jwtSecret is also the secret you sign tokens with. The passwords are hex because the services connect with a postgres:// URL, and the / that base64 produces about four times in ten ends the password inside it. There are no CRDs and no operator — the only custom resources are KEDA's, and only if you turn autoscaling on.

Why it works — an OCI artefact, and four passwords the chart refuses to generate

The chart is an OCI artefact, so there is no chart repository, no index.yaml and no DNS involved, and each of those is a thing that can break on its own. helm repo add percolate https://percolation-labs.github.io/get-percolate/charts works too if you prefer a classic repo, and Flux and Argo can both point straight at charts/percolate in git with nothing published at all.

The four passwords are required and the chart will not generate them for you, which looks unhelpful until you see the failure it avoids. A generated value comes out different between helm template and helm install, and between a GitOps dry run and the apply after it — so a controller that diffs continuously will either report drift forever or "fix" it by rewriting the database password underneath a running StatefulSet.

Into a Postgres 19 you already run

What we are trying to do here is add the extensions to a database that already exists, without a service anywhere.

pgvector first, from your platform's packages (apt install postgresql-19-pgvector from the PGDG repository). Then the extension files, and then bootstrap.sql, which install.sh downloads into the current directory — run it as a superuser, in the database that will hold Percolate:

curl -fsSL https://raw.githubusercontent.com/Percolation-Labs/get-percolate/main/install.sh | sh

export P8_AUTH_PW=$(openssl rand -hex 24) P8_WORKER_PW=$(openssl rand -hex 24)
psql -d yourdb -v ON_ERROR_STOP=1 \
     -v auth_pw="$P8_AUTH_PW" -v worker_pw="$P8_WORKER_PW" -f bootstrap.sql

Keep the two passwords: PostgREST connects as authenticator and the workers as worker. A bare CREATE EXTENSION percolate does not work on this path — as a superuser the extension refuses to load, and as anyone else the roles it needs do not exist yet — which is what bootstrap.sql sequences. It is safe to run again.

Why it works — two extensions that ship differently, because they are different kinds of thing
Extension What it is How it ships
percolate the whole system — schemas, tables, functions, RLS pure SQL, one file, the same on every platform
percolate_parser the P8QL and YAML compilers a Rust .so, prebuilt per platform

A shared library is compiled against one ABI, so there is no portable form of the parser and we build it per platform instead. install.sh reads your pg_config — the only thing that knows where this particular Postgres keeps its extensions — checks the major version, and puts both files where they belong.

We publish parser builds for linux/amd64, linux/arm64 and macos/arm64. On anything else the script installs the SQL extension, tells you that define_yaml and p8ql: steps will not resolve until the parser is built, and exits non-zero, so a half install does not look like a successful one.

bootstrap.sql has two halves. As the superuser it creates the cluster roles and the two extensions that are not trusted, vector and percolate_parser; then it switches to app_owner and runs CREATE EXTENSION percolate CASCADE, which creates the trusted pgcrypto and pg_trgm and everything percolate itself holds. So all of it is owned by a non-superuser, which is not a detail: a superuser bypasses RLS unconditionally, so a superuser-owned install would leave every policy in the collection inert while looking correct. The extension asserts this about itself rather than trusting the installer to get it right.

pg_cron, if you want schedules

This one is optional and its absence is quiet, so it is worth setting up consciously.

What we are trying to do here is give the engine a clock, which it needs for three separate things.

Install the package (apt install postgresql-19-cron), then in postgresql.conf:

shared_preload_libraries = 'pg_cron'   # added to any list already there
cron.database_name = 'yourdb'          # the database bootstrap.sql ran in
cron.use_background_workers = on       # jobs run in-process, with no login

Restart the server and run bootstrap.sql again. It creates pg_cron and schedules the three jobs — workflow.tick(), workflow.reap_stale_tasks() and workflow.promote_due_timers(), every minute — as the scheduler role rather than as a superuser. The compose image and the Helm chart do all of this themselves.

Why it works — and what silently does not happen without it

pg_cron runs as a background worker, so there is no CREATE EXTENSION that can add it after startup. Without it, scheduled workflows never fire, the stale-task reaper never runs so a crashed worker's tasks are never recovered, and every timer step waits forever. None of those produce an error; they produce a system that looks idle.

One set of jobs covers the whole deployment however many schedules you have, because a schedule is a row in workflow.schedules rather than a cron entry.

cron.database_name is the usual reason a correctly-created job never runs — it can only be one database, and the default is postgres. cron.use_background_workers is the other: without it each job logs in over libpq as scheduler, which has no password, so wherever pg_hba.conf asks for one every run fails with connection failed while cron.job still shows the job active.

The first user, and a token

A fresh install has no users, no roles and no permissions — those tables are empty on purpose, because the alternative is a default administrator with a known password. Nothing over HTTP works until you create one: PostgREST answers permission denied for function upsert_agent with a 401, and the agent runtime answers a verified bearer token is required.

What we are trying to do here is get from an empty rbac to a bearer token that the REST interface and the agent runtime both accept.

select rbac.bootstrap_admin('me@example.com', 'a long passphrase') as uid \gset

That is the whole step. It creates the user, creates an admin role, grants it the permissions the system actually checks, and makes the first user_roles row — and it refuses to run once any role has been granted to anyone, which is what keeps it a bootstrap rather than a back door.

Use it rather than writing the insert statements yourself, because a wrong verb in one is invisible. role_permissions.action is free text compared with =, so a bad value is not an error anywhere: it inserts, the role looks populated, and every policy that consults it returns false. Granting workflow_runs/read where the RLS policies ask for select gives you an administrator who sees zero rows in workflow.runs_api, with no way to tell that from "my workflow never ran". The vocabulary is derived in one place inside the extension for this reason — prose that repeats it can drift away from it.

Then sign a JWT with the same secret the stack was given — P8_JWT_SECRET, which the compose file defaults to change-me-a-long-random-string-at-least-32-chars. The CLI installed in the next section does it, with the tenant claim the sample's data needs:

export P8_JWT_SECRET=change-me-a-long-random-string-at-least-32-chars
TOKEN=$(percolate auth token --email me@example.com \
          --orgs d0000000-0000-0000-0000-00000000000a)

With nothing installed, the same token is a few lines of standard library. Two claims are required: sub is the user id every RLS policy reads through rbac.current_user_id(), and role is the Postgres role PostgREST switches to; orgs is the tenant list, without which tenanted rows are invisible.

import base64, hmac, hashlib, json, time
b64 = lambda b: base64.urlsafe_b64encode(b).rstrip(b"=")
secret = b"change-me-a-long-random-string-at-least-32-chars"
head = b64(json.dumps({"alg": "HS256", "typ": "JWT"}).encode())
body = b64(json.dumps({"sub": "<the uid bootstrap_admin returned>",
                       "role": "authenticated",
                       "orgs": ["d0000000-0000-0000-0000-00000000000a"],
                       "exp": int(time.time()) + 3600}).encode())
sig  = b64(hmac.new(secret, head + b"." + body, hashlib.sha256).digest())
print((head + b"." + body + b"." + sig).decode())

That is the $TOKEN every curl in these docs wants.

the same call, before and after
$ curl -s localhost:3000/rpc/upsert_agent -H 'Content-Profile: agentic' \
       -H 'Content-Type: application/json' -d '{"p_spec":{"name":"harbourmaster"}}'
{"code":"42501","message":"permission denied for function upsert_agent"}  # HTTP 401

$ curl -s localhost:3000/rpc/upsert_agent -H 'Content-Profile: agentic' \
       -H "Authorization: Bearer $TOKEN" \
       -H 'Content-Type: application/json' -d '{"p_spec":{"name":"harbourmaster"}}'
{"id":"4dc4e929-…","name":"harbourmaster", …}                          # HTTP 200
Why the first grant cannot go through `grant_role`, and why `psql` could author an agent all along

rbac.grant_role is gated on rbac.has_permission(current_user_id(), 'users', 'admin'), and on an empty database nobody holds that — including the person trying to bootstrap. So the first user_roles row cannot come from grant_role, and every grant after it can. That is a real bootstrap step rather than an oversight: the alternative is a function that grants users/admin to whoever asks first.

bootstrap_admin is that step with the escalation closed rather than left to the caller. It refuses the moment rbac.user_roles holds anything, so it is reachable only while the system has no administrator — the one window in which handing out users/admin is not a privilege escalation. It is also not granted to web_anon or authenticated, so it does not exist over PostgREST; like the CREATE EXTENSION before it, it is run from a privileged local session.

It is also why the SQL on the agents page works from psql before any of this. agentic.may_author permits the call when rbac.current_user_id() is null and the session is a privileged local one — the migration path, deliberately — so psql is authoring as the database owner, not as a user. The moment the same call arrives over HTTP there is a JWT and therefore a user, and the permission is checked.

Passwords are for rbac.login, which exchanges them for a refresh token; it does not mint the access token, because whatever holds P8_JWT_SECRET does. There is no HTTP endpoint that hands you one, so a token comes from something holding the secret — percolate auth token, the snippet above, or your own auth provider signing with the same secret and the same claims.

The sample, which nothing loads for you

A fresh install is empty, and almost every worked example in these pages reads data. There is a sample for that, and loading it is a step you take rather than something a container did while you were not looking.

What we are trying to do here is get the domain the rest of this documentation queries, and be able to tell it apart from our own data afterwards.

percolate is the CLI from percolate-core, and nothing you have run so far installed it — the compose stack runs that image, it does not put the command on your PATH. It needs Python 3.11 or newer; on a Mac the system python3 is older than that and pip will report the package as simply not existing rather than as unsupported.

The extras are not optional decoration: sample is the YAML reader and agent is what turns plugin.yaml's agents — which are JSON Schema documents, not prompt strings — into rows. Without them the load refuses before it writes anything, which is the right behaviour and still a stop.

It goes in a virtualenv, because a bare pip install stops at externally-managed-environment on Homebrew's Python and on current Debian and Ubuntu:

python3 -m venv ~/.percolate && . ~/.percolate/bin/activate   # 3.11 or newer
pip install 'percolate-core[sample,agent]>=0.1.8'

samples/harbour is a path inside this repository, so it needs to be on disk — a compose install has only the one file you curled. The DSN has to be the owner's, because the load writes rbac.*; the CLI names the compose one in its own error if you forget:

git clone https://github.com/Percolation-Labs/get-percolate
cd get-percolate
export P8_ADMIN_DSN=postgres://p8:p8@localhost:5432/percolate
export P8_JWT_SECRET=change-me-a-long-random-string-at-least-32-chars
export LLM_API_KEY=sk-...          # the key in your .env
percolate sample load samples/harbour --as-email me@example.com

P8_JWT_SECRET is the compose file's default signing secret — use your own if you set one in .env — and the loader signs the token it uploads the corpus with. It checks LLM_API_KEY before uploading, but the embedding call is made by the stack's ingest worker, which is why the key had to be in .env when the stack started.

A port-operations company: two tenants, four operators, five vessels, three ports, inspections, a corpus of reports and a graph tying them together.

harbour 1.0.0
  sql schema.sql
  embedding model text-embedding-3-small (1536d)
  entity operator <- harbour.operators
  entity vessel <- harbour.vessels
  entity port <- harbour.ports
  ...
  edge operator:MERB -subsidiary_of-> operator:MERI
  plugin harbour (0 servers, 1 skills, 1 agents)
  workflow harbour_deficiencies

Most of it is tenanted, so mint a token that names one of the two organisations or you will see only the shared tier and think nothing loaded:

percolate auth token --email me@example.com \
  --orgs d0000000-0000-0000-0000-00000000000a     # Meridian; ...000b is Kestrel

It needs an embedding key, in LLM_API_KEY, because the corpus goes in through POST /files and is embedded by the running pipeline. --dry-run says what a load would need without writing anything; --skip-documents loads everything else, and LOOKUP, FUZZY, GRAPH and TEXT all work without it — only SEMANTIC and SEARCH need vectors. If you loaded before the stack had the key, the documents are registered and their embeddings failed; a second load uploads them again and those copies fail to parse, so remove the first ones before loading again — the statement is in the sample's README.

Why it works — a directory of the documents you would have written anyway, and why the vectors are not in it

The sample is samples/harbour/ in the repository, and it is worth opening before you run it. schema.sql is the company's own tables — nothing in it mentions Percolate, which is the point, because Percolate indexes tables you already have. Everything else is a document in the format you would author by hand: sources.yaml is what is addressable, graph.yaml is the relation vocabulary with edges named vessel:Meridian Dawn rather than by uuid, plugin.yaml is skills and agents as one removable bundle, workflows/ are workflow documents and documents/ is markdown. Reading it teaches the formats; a .sql dump would have hidden all of them behind four hundred INSERTs.

The vectors are deliberately not in the files. An earlier version of this fixture shipped literal four-dimension vectors so it would load with nothing running. It reproduced beautifully and taught the wrong thing: a reader who copied the pattern had a corpus no model had ever seen and rankings that meant nothing. Here the documents are embedded by the same pipeline yours will be, so what you search is what a model produced — and the sample costs one embedding call per document, which is the honest price of retrieval rather than an inconvenience.

Nothing loads on first boot for the same reason. Rows that appear because a container started are rows nobody chose, and a demo you cannot tell apart from a deployment is a bad demo. Applying the plugin with an empty manifest takes the skills and agents back out and says what it removed, and drop schema harbour cascade takes the rest:

select agentic.apply_plugin('{"name":"harbour","version":"1.0.0"}'::jsonb);
drop schema harbour cascade;

Uninstalling is the same call as installing rather than a second function, because the pruning path is the one that knows what else references a row — which is why the plugin columns are on delete restrict.

Next: agents, which is the shortest useful thing to do against a fresh install: write one spec, save it, and call it.