The workflow grammar

A sql or p8ql step becomes ready the moment its dependencies resolve, and autorun_sql_steps executes it inside that same transaction. There is no queue to poll and no process to deploy, which is why the run above has already finished by the time start_workflow returns to you.

The rows land on the task, so the graph walk the middle step did is readable straight out of workflow.tasks.output rather than having to be recomputed.

One thing to know lives here, and it is the ninth P8QL mode. A p8ql: step holding plain SQL executes, and inside a step the invoker is the engine owner — which owns every table and is not subject to their row-level security. So such a step reads across tenants. It is a beta trade and percolate.sql_policy = 'registered' refuses it from 0.2.30 onward; the cookbook has the whole of it.

Related why plain SQL behaves that way · a four-step pipeline end to end

A read is wrapped as select … from (<your statement>) t and runs with transaction_read_only on. The wrap is structural: Postgres refuses a data-modifying WITH anywhere but the top level, so with gone as (delete …) fails on its shape rather than on a keyword list. The read-only transaction catches everything else, including a volatile function that writes, which no wrapping sees. The keyword check beside them is a courtesy that turns an abort into a sentence naming write: true.

Every {{template}} becomes a bound parameter. The statement is rewritten so each reference reads $1->>'<ref>' and the values are bound alongside it, so a run input of '; drop table x; -- is a string — not because it was escaped, but because it never reaches the parser. Quotes you write yourself are consumed rather than doubled, so cik = '{{run.cik}}' and cik = {{run.cik}} mean the same thing.

This covers {{item.*}} inside a matrix: template as well, which matters because those values are rows rather than something the author typed. A JSON number binds as ::numeric and a boolean as ::boolean, so where n = {{run.n}} still works against an integer column; a text value compared against a non-text column needs your own cast, because ->> yields text and text is not coerced to the column's type for you.

The one thing you cannot template is an identifier — from {{run.table}} becomes from ($1->>'run.table') and fails with syntax error at or near "$1". That is the parameterisation working, not a gap: values can be bound and names cannot, which is where every database driver draws the same line.

What it costs. A statement runs as the engine owner, which owns every table and bypasses row-level security, and so does every p8ql: step, dialect modes included. Anyone who may define a workflow may read anything in the database. That is the trade: the alternative made every question a privileged human had to bless.

In 0.2.36 it also reaches whoever starts the run, and anyone signed in may start any workflow: a member who runs a retrieval workflow an admin wrote reads every org's documents through it. The next extension release refuses such a step when the person who started the run may not define workflows. The cookbook says what to do until then.

A deployment can also refuse the SQL an author writes —

alter database <db> set percolate.sql_policy = 'registered';

— and statements are refused at authoring time and, from 0.2.30, at dispatch too, with registered functions still running. That narrows what an author may write. It does not stop a p8ql: SEARCH reading as the owner, because p8ql is itself a registered function.

Related registering a function, and when it is worth it · what a `sql` step should return

The registry stopped being a gate and kept the jobs it was always better at. A registered function carries its own timeout_ms, and its description is the same text a bound tool takes into a model's context — so workflow.step_functions is the answer to what can a workflow cause to run for the operations you chose to bless. Under sql_policy = 'registered' it is also the whole surface.

Registration checks one thing that is easy to get wrong and expensive to discover late. execute_sql_step is SECURITY DEFINER, so your function runs as its owner rather than as you — and if that role cannot reach your schema, registration refuses with the grant that fixes it rather than succeeding and failing at every run:

ERROR:  function harbour.land_notices(jsonb) is not reachable by app_owner,
        which is the role that will execute it (execute_sql_step is SECURITY
        DEFINER). Registration would succeed and every invocation would fail
        with "permission denied for schema harbour". Run:
        grant usage on schema harbour to app_owner;

The check is on the table, not on this function, and the difference is the one that makes the inventory worth trusting. A plain insert into workflow.step_functions is refused by the same rule with the same message, so there is no door into the registry that skips it — including one you reach by reading the table before this page. Registering is owner-only either way; what the trigger adds is that an owner cannot do it by accident.

Related the spelling that needs no registration · what a `sql` step should return

The compiler copies the mapping under rest: into workflow.tasks.input verbatim, which is why url, method, body, headers, timeout_ms, jsonpath, mode, wait_ms and credential_ref are all available without the grammar enumerating them: the worker's handler is what reads them, and the compiler does not need an opinion.

credential_ref is a name, resolved by the worker from its own environment. A dump of workflow.tasks therefore never contains a secret and a task stays inspectable and replayable. {{env.X}} works the same way and is resolved only by the worker — a sql: or p8ql: step cannot read it, because the database has no business knowing the deployment's environment.

mode: async sends the request and then holds the task until the callee calls complete_task or fail_task. While it holds, the worker heartbeats the lease and writes nothing; when the callee finishes, the worker lets go. wait_ms bounds the hold, and is 15 minutes when omitted. Past it the worker fails the task terminally, because a retry would send a second request to a callee that may still be working. A wait_ms that is not a positive whole number of milliseconds is refused before anything is sent.

The hold occupies the worker. percolate worker runs one task at a time, so a worker holding an async step claims nothing else until the callee answers or wait_ms passes. Run as many http workers as you expect async steps in flight at once, plus the ones your synchronous steps need.

That is the seam a scheduled agent run uses: its step posts to /internal/run, and the runtime completes the task when the turn ends, or fails it with a verdict when the turn does. An agent: step does not use it; it is the synchronous call below.

Related which HTTP failures retry · the heartbeat and lease fence the hold relies on · every place a credential is named rather than stored

SEARCH and SEMANTIC rank against a vector, and the database makes no model calls. So the compiler expands that one step into an http_call keyed retrieve__embed plus the sql step keyed retrieve that consumes it. Your id stays on the search, so needs: [retrieve] and {{steps.retrieve.result}} mean what they look like and nothing downstream is rewired — the embed is a hidden predecessor, not a replacement.

You never write the URL. It comes from aiq.embedding_models, and USING <model> is written into the query if it was not there and refused if it disagrees with the embed step's model. A vector from one space ranked against another is not an error at runtime, only a meaningless number, so it is made a compile error instead.

queue: on the step routes the embed half; the search half runs in-database and is claimed by nobody. The hand-written pair — embed: then sql: {function: p8ql_vec, …} — is still legal and compiles to exactly the same rows.

Related the three search modes · a retrieval pipeline with an agent on the end

It compiles to POST {{env.P8_AGENT_URL}}/v1/chat/completions on the http queue with credential_ref: P8_API_KEY, and reads the answer from choices.0.message.content — the call is synchronous, so the worker holds the connection for the length of the turn. The worker needs P8_AGENT_URL, which the compose file and chart set, and no token: it signs one for the run's owner (why). Nested delegation happens inside the runtime rather than being unrolled into the workflow graph, which is why a researcher calling an analyst is one task here and a delegation tree there.

session_group asks the engine for a session id, stable for the life of the run and bound to {{run.$session}}. Steps naming the same group share a conversation, two groups are two threads, and declaring neither stays stateless — the right default for a classifier. The step sends the id as session_id, which is the key the Agent Runtime reads, and the engine creates the agentic.sessions row for it, owned by whoever started the run, the first time a step needs it, so the runtime has a conversation to continue rather than an id it answers 404 to.

output_schema is what makes the answer composable: the engine parses the model's JSON-in-a-string reply and checks it before storing, so the next step receives an object rather than prose. A shape violation is the one retryable failure in this engine — every other terminal classification exists because retrying cannot help, and here the same prompt genuinely can conform next time.

It is a shape check and not JSON Schema. Types including integer, required, properties, enum, items and additionalProperties: false are supported; $ref, allOf/anyOf/oneOf, pattern, format and numeric bounds are not, and are named here rather than ignored without a word. Declaring output_schema on a sql or matrix step is refused, because the check is for output this engine did not produce.

Related what an agent row holds · why a shape violation retries · an extractor fanned out over a backlog

There is no window in which extract is succeeded and the children do not exist yet. A controller-based fan-out has that window, and a controller that dies inside it strands the fan-out with nothing to resume from.

max_fanout is mandatory and the compiler says why rather than defaulting: a cross join with a forgotten WHERE expands to the cartesian product, and that should fail one step rather than the database. rows: takes the query itself or a registered function, exactly as sql: does — with one difference: a row source is always read-only, because it is re-run by every retry of the expansion.

continue_on: failed lets a failed child satisfy its dependents, and min_success is the floor below which the aggregate is cancelled rather than run on partial data — a fraction if it is ≤ 1, an absolute count if it is > 1. Declaring min_success without continue_on is refused by name, because the first failed child would cancel the successors and the threshold could never be reached. It is a transport floor: a 200 carrying an empty body counts as a success against it.

The fan-in reads a handle, not a payload. {{steps.extract.result.task_id}} is the matrix task and workflow.matrix_outputs(task_id) pairs every child with its row, status and output. Five hundred extractions are for aggregating inside your own function, not for passing through a step argument that caps at 64KB.

One sharp edge to check before you rely on it: a matrix template cannot declare output_schema. The compiler accepts queue, rate_key, rest, embed, agent, work, input, session and jsonpath inside a template and not that, so a fan-out child lands the model's answer as a JSON string and your fan-in parses it.

Related partial failure in depth · why the fan-in reads handles · the backlog recipe this comes from

A timer moves when workflow.promote_due_timers() runs on the pg_cron minute that already ticks the engine. A signal step then sits in waiting_external, which is the status meaning a person is the dependency, and only signal_task moves it. Both ask whether the run is finished once they have moved a step, as complete_task does, so a run whose last step is a signal or a timer ends succeeded when that step does.

signal_task checks who you are, and a run with no owner is not owned by everyone: it takes an explicit permission against the run's owner, set from the verified JWT of whoever started it. That is also why a scheduled run carries its schedule's owner — otherwise a scheduled workflow that waits for approval could never be approved, by anybody.

All three of these were legal engine kinds that no document could reach until somebody tried to write exactly this workflow and the compiler refused it. And sub_workflow had never executed at all: start_child_workflow inserted a run row without expanding the child's task graph, so composition was advertised, unreachable, and broken underneath.

Related why `pg_cron` is not optional for timers · a pipeline that waits for a person

The engine asks the predicate once, when the step's needs are done and it would otherwise become ready. True runs the step. False or null marks it skipped, which workflow.lifecycle_states records as terminal and a success, so the steps after it are released as they would be after a success, and a run of successes and skips has succeeded. A skipped step is never claimed, a skipped signal never waits for anybody, and a skipped step is never compensated, because it did nothing.

Each {{template}} in a predicate is a bound value, as in a sql: statement, and a step it names must be in needs; the definition is refused otherwise. A skipped step's result is null, so a template naming it binds NULL and a comparison with it is null, which skips too. when: "{{steps.review.result.0.r}} is null" is therefore how a step asks whether review was skipped. A gate is a when: on every step it guards: pay above runs after a skipped review because it has no condition of its own.

The predicate runs through workflow.run_sql, as a sql: step does: read-only, under percolate.sql_policy, and as the run's owner, so row policies apply to whatever it reads. One that cannot be evaluated fails its step and the run, with when: <the error> as the step's error (failure).

Related skipped, failed and cancelled · how a template binds in SQL

Attempt n of count is the ordinary step count_<n>. The first steps of an attempt wait for the last steps of the attempt before, and every step of an attempt after the first carries when: not (<until over the attempt before>). Once the check holds, the next attempt is skipped, and because a skipped step's result is null, so is every attempt after it. Each attempt is a task row a trace can read, and the bound is the number of rows written: max is required, from 1 to 20, and a repeat without it is refused.

Inside the group, {{steps.<inner>}} is this attempt's step and {{previous.<inner>}} the attempt before. On the first attempt there is none: in SQL (sql:, p8ql:, when:) the template is NULL, quotes around it included, which is why coalesce starts the count at 0; anywhere else, such as an agent's input, it is removed. until must name at least one of the attempt's steps, and an inner step's own when: still applies alongside it.

The repeat's id is a final step that waits for every attempt. The steps after the repeat read it rather than the attempts:

An attempt can be any steps, agents included: a writer and a critic, with until over the critic's grade and the writer's input reading {{previous.<critic>.result}}, is the same document with two inner steps.

satisfied false with attempts equal to max is a repeat that ran out, and a step after it can say what to do then with a when: of its own. Repeats do not nest, the group takes only id, needs and repeat, and an attempt's name must not collide with another step's id.

Related the condition each attempt carries · every template namespace

This is the rule to internalise, because getting it backwards does not raise.

args: ['{{steps.embed.result}}'] is a whole-string reference, so it resolves to the vector as an array rather than to a string that looks like one. url: 'https://x/?since={{steps.cursor.result.since}}' is embedded, so it interpolates as text and the URL works.

The failure was in this collection's own headline example. A step passing 'SEARCH "{{run.question}}" FROM chunks LIMIT 3' is not a whole-string reference, so the substitution happened nowhere, the lexical half searched for the literal text {{run.question}}, every lexical rank came back null, and the fusion degraded to semantic-only, reporting nothing. It returned rows. It looked like it worked.

Typing {{run.id}} when you meant {{run.$id}} raises with the spelling you wanted, rather than resolving to nothing and failing three tasks later.

Related the run seeing itself · why every step nests under `result`

The step kinds are the engine's and they are in the parser. The functions a sql: step may call, the queues a step may route to and the agents an agent: step may name are all rows on the database in front of you, so no document can be authoritative about them.

SCHEMA "workflow" returns both halves and runs its own examples through the compiler on every read, so a spelling this deployment no longer accepts reports itself rather than being served as correct. compiler_capabilities() probes the installed build with one canary per feature and lists what is missing, which is how version skew between the Rust parser and the SQL schema shows up as a capability report rather than as a syntax error in a document that is not wrong.

 {"accepts": {"matrix": true, "output_schema": true, "continue_on": true,
              "signal": true, "timer": true, "sub_workflow": true,
              "when": true, "repeat": true},
  "missing": []}

Related the same mechanism for the query dialect