Failure and retry

Retry is a decision rather than a default. The worker is the only thing that knows what an HTTP status actually means, so the worker decides and the engine goes with its answer.

Terminal or retryable

Here we tell the engine whether another attempt could possibly help.

fail_task(task_id, error, terminal=True)   # do not retry, whatever attempts remain
fail_task(task_id, error, terminal=False)  # requeue with exponential backoff

Failure	Class	Why
HTTP 4xx (except 408, 429)	terminal	the same request gets the same answer
HTTP 429 whose `error.code` is `insufficient_quota`, or 402	terminal	the account has no credit, and waiting does not add any
HTTP 5xx, any other 429, connection error	retryable	transient by definition
Output over the payload limit	terminal	the same response is the same size every time
Output violating `output_schema`	retryable	the same prompt really can conform next time
Unregistered step function, missing handler	terminal	`pip install` does not run itself between attempts
Missing `credential_ref`	terminal	configuration, not weather
A `$artifact` that is not a resource uuid	terminal	the same string arrives every time

Why it works — the shape violation is the only one classified against appearances

A 400 or a 404 will fail the same way every time, so retrying it five times with backoff turns a fast failure into a slow one and burns your rate limit doing it. A 5xx or a 429 is the server's problem and is worth another go, unless the 429 says the account is out of credit: that is the same status with the opposite remedy, and the worker reads the provider's error code to tell them apart. The rule is the same for an http_call step, embedding, transcription and API ingestion.

The shape violation is the interesting row, because it looks like a bug and is classified retryable anyway. Every other terminal classification exists because retrying cannot help; here the same prompt genuinely can produce conforming output on the next attempt, which turns max_attempts into a real budget for asking again. Verified: a step whose endpoint violates its schema twice and conforms on the third completes in three attempts.

The reason this decision belongs to the worker rather than to the engine is that the engine sees a failed task and the worker saw the response. Handing the classification to the engine would mean encoding HTTP semantics in PL/pgSQL and getting them wrong for every protocol that is not HTTP.

Fan-out and partial failure

A matrix over real data will hit a missing value, a rate limit or an oversized response on more or less every run.

Here we let a child failure be a result rather than an outage, while still refusing to aggregate over too little.

  - id: run
    matrix:
      rows: {function: labelled_sample, args: ['gazette', 30]}
      max_fanout: 30
      continue_on: failed
      min_success: 20
      template: {agent: extractor, input: '{{item.text}}'}

Why it works — the floor exists so a partial answer cannot report success

continue_on: failed makes a failed child satisfy its dependents. min_success is the floor below which the aggregate is cancelled rather than run on partial data, because a summary computed over 3 of 30 documents that reports success is worse than no summary at all. It takes a fraction when it is ≤ 1 and an absolute count when it is > 1.

Declaring min_success without continue_on is refused by name: without it the first failed child cancels the successors, so the threshold could never be reached and the key would be decoration.

It is a transport floor rather than a content one — a 200 carrying an empty body counts as a success against it. The engine cannot know what a good result looks like, so the threshold answers did enough calls come back and the declared shape answers the rest.

In an evaluation the polarity flips. The fraction of documents an extractor cannot parse is the measurement you wanted, so an eval sets continue_on: failed with no floor at all.

A step that did not need to run

A step has four ways to finish, and only two of them hold up the steps after it.

Status	Means	The steps after it
`succeeded`	it ran and returned	are released
`skipped`	its `when:` did not hold, so it did nothing	are released, and read its result as null
`failed`	it ran out of attempts, or failed terminally	are cancelled, unless `continue_on: failed` says otherwise
`cancelled`	something before it failed, the run failed on another branch, or the run was cancelled	are cancelled

Here we see what happens when a step's condition cannot be evaluated at all, because it compares a number with a word.

select workflow.define_yaml($y$
name: bad_gate
steps:
  - id: score
    sql: "select 0.7 as v"
  - id: review
    needs: [score]
    when: "{{steps.score.result.0.v}} > 'high'"
    sql: "select 'reviewed' as r"
  - id: pay
    needs: [review]
    sql: "select 'paid' as p"
$y$);

select workflow.start_workflow('bad_gate', '{}');

select t.step_key, t.status, left(t.error->>'error', 60) as error, r.status as run
  from workflow.runs r
  join workflow.definitions d on d.id = r.workflow_def_id
  join workflow.tasks t on t.run_id = r.id
 where d.name = 'bad_gate'
 order by t.id;

one run of bad_gate

 step_key |  status   |                        error                        |  run
----------+-----------+-----------------------------------------------------+--------
 score    | succeeded |                                                     | failed
 review   | failed    | when: invalid input syntax for type numeric: "high" | failed
 pay      | cancelled |                                                     | failed

Why it works — a condition that cannot be answered is an error, not a no

A predicate that is false or null skips its step, and that is the design: a condition over a step that was itself skipped is null, and skipping is the right answer to it. A predicate that raises is different. Reading it as false would pay a claim nobody reviewed because of a typo, so the step fails, with when: and the error as its error, and the run fails with it. The step before it had already finished, and stays finished.

skipped counts as success everywhere success is asked: a step waiting on it is released, a run of successes and skips has succeeded, and workflow.v_run_progress counts it as done. On a page or in a trace it is still told apart from succeeded, and a skipped step has no output, so the difference between "nothing to do" and "did it" survives in the record.

A failed run is over. Its other branches stop with it: a step that was still waiting or ready is cancelled, a running one is asked to stop (its heartbeat says so) and its late answer is not kept, and a running child workflow is cancelled. The one exception is compensation, which is the work a failed saga still owes, so it runs and finishes under the failed run.

A signal step that was skipped never reaches waiting_external, so nobody is asked to sign it. Whoever drives the run should read the step's status before calling signal_task: skipped means there is nothing to sign, and cancelled means a step before it failed, which the run's tasks name.

Crash recovery

Here we get work back when the process holding it stops existing.

select workflow.reap_stale_tasks();     -- one pg_cron job, once a minute

Why it works — the lease fence, and why a slow worker and a dead one look identical until one returns

A worker holds a lease it refreshes by heartbeat, and reap_stale_tasks() requeues a task whose lease has gone stale — unclaimed, and behind a backoff.

Two parts are less obvious than they look.

A reaped worker cannot publish its result. If the original worker comes back and calls complete_task, the lease fence refuses it and the refusal shows up in v_lease_violations rather than disappearing. Without that, a slow worker and a dead one are indistinguishable right up until the "dead" one returns and overwrites the retry's answer with a stale one.

A crash loop is bounded by max_attempts, fails the run, and names the worker that died.

heartbeat_task returns false for three different reasons — a cancel was requested, the task is terminal, or your lease was reaped — and all three mean stop.

Saga compensation

Here we give back the things that already succeeded, when a later step fails.

  - id: charge
    rest: {url: …, method: POST}
    compensate_with: refund
    saga_group: booking

Why it works — compensations are ordinary tasks, by design

begin_compensation() enqueues the compensating steps for completed tasks in reverse completion order. Because they are ordinary tasks they inherit retries, backoff and the audit trail, and because they stay out of the forward graph a compensation cannot accidentally satisfy a forward dependency.

The run ends failed with compensation_state = compensated, and those are two columns because they answer two questions. A saga that rolled back cleanly still did not do what it was asked.

What to watch


`workflow.v_stuck_tasks`	not moving, and not making progress
`workflow.v_lease_violations`	a reaped worker tried to publish its result
`workflow.encoding_drift()`	a producer returning double-encoded JSON
`content.check_drift()`	files no resource points at, resources never chunked

Why it works — the first one has a lesson in it worth generalising

A view of heartbeat_at alone cannot see a crash loop, because every claim refreshes it — so the failure keeps resetting the signal that is supposed to reveal it. That is worse than having no view, because somebody is watching it and drawing the wrong conclusion. v_stuck_tasks therefore pairs the two.

The general rule is to pair staleness with progress. "Not moving" and "moving and getting nowhere" are different failures, and a monitor that only detects the first will report health during the second.

Next: uploading files.