Workflows

Workflows let you chain several ProcessingSteps into a single, repeatable process. Instead of running each step by hand and moving data between them yourself, you describe what should happen once — and the platform runs it for you, in the right order, every time.

A typical workflow might: find some input data, run one or more processing steps on it, and produce new results — with independent steps running at the same time to save time.

If you have not built one yet, the Your first workflow guide walks through creating and running a simple workflow from scratch. This page covers the concepts and more advanced features in more depth.

Anatomy of a workflow

A workflow is a graph: a set of steps joined by connections. The steps declare what to do and where data comes from; the connections declare how data flows between them. Together they form a diagram that flows from start to finish, and the platform reads this diagram to decide what can run and when.

You can assemble this graph however you like — most users build it visually in the portal editor (dragging steps onto a canvas and drawing connections, no code required), but the same structure can be authored programmatically (see the workflow format).

Steps

A step is a single node in the graph, and does one job. There are four kinds:

Step	What it does
Processing step	Runs a named processing step on the platform (the actual work — analysis, conversion, computation, …).
Data query	Finds existing data by searching for it (for example, by tag, name, or date), so it can feed later steps.
Data selector	Narrows down data coming from an earlier step — for example, take only the first few items.
Explicit data	Points directly at specific data you already know, by reference.

Choosing a processing step version

A processing step can exist in several versions. For a Processing step, you can pin an exact version — or, better, give a version range and let the platform pick the newest version that matches:

You specify	The workflow uses
(nothing)	The latest available version.
1.2.4	Exactly that version (pinned).
1.2.*	The newest patch of 1.2.
1.*	The newest 1.x version.

Using a range makes a workflow less fragile: when a new version of a processing step is published, the workflow automatically picks it up on its next run — within the bounds you allow — instead of breaking or needing to be edited. Pin an exact version only when you specifically need reproducible, unchanging behavior.

Connections

A data connection joins the output of one step to the input of another. This is how results flow through the workflow, and it is also what determines the order steps run in: a step can only run once every connection feeding its inputs has delivered its data.

Variables

Variables are placeholders you can fill in each time the workflow runs — for example a search term, a threshold, or a name. You give each variable a default value when you build the workflow, and it can be overridden at run time. This lets one workflow serve many similar cases without rebuilding it.

For the file-level details — placeholder syntax, substitution rules, and how to declare a variable — see How variables are resolved.

How a workflow runs

When a workflow runs, PineXQ schedules the steps according to their data dependencies — a step runs as soon as everything it needs is ready, not in the order you happened to draw them.

When you start a workflow, the platform:

1. Looks for steps that are ready — a step is ready once all the data it needs is available.
2. Runs ready steps — including several at the same time when they don’t depend on each other.
3. Waits for results, then repeats — newly finished steps unlock the next ones.
4. Finishes when every step is done.

A few things happen automatically:

• Parallel execution — independent steps run concurrently across the available workers. Each time a step finishes, the platform re-evaluates which steps have now become ready and starts them, continuing until every step has run.
• Data resolved at execution time — rather than picking input data up front, a workflow can search for it while it runs: a data query finds matching data at that moment, and a data selector then narrows the result — for example, keeping only the newest item. This means the same workflow automatically picks up whatever data is current each time it runs.
• Variables supplied at run time — parameters can be left open in the workflow and filled in when it is started, so one workflow can be reused for many inputs without editing it.
• Timeouts — you can set a time limit for a single step and for the whole workflow. If the limit is exceeded, the run stops with an error.
• Retries — a step that fails can be retried automatically according to the workflow’s retry settings.

When a step fails

A workflow does not stop everything the moment one step fails. It runs on a best-effort basis: the platform keeps executing every step whose input is available, and only the steps that actually depend on the failed step are skipped — because the data they were waiting for never arrives. Steps on unrelated branches carry on and finish normally.

For example, in a workflow with two independent branches, a failure in one branch does not prevent the other from completing.

At the end, the run is given an overall state that reflects what happened:

State	Meaning
Completed	Every step finished successfully (or was disabled).
Partially completed	Nothing errored, but some steps could not run — for example because an upstream step they depended on produced no result.
Failed	At least one step ended in an error.

Whatever the outcome, the execution report lists every step, its result, and the details of any errors, so you can see exactly what ran and what did not.

Automatic tags

So you can always find what a run produced, the platform adds tags automatically to every job it starts and to every piece of output data — in addition to any tags you set yourself on a step.

Jobs started by a workflow are tagged with:

Tag	Meaning
workflow-job	Marks this job as one started by a workflow.
workflow-id=<id>	The id of the workflow definition. Stays the same across every run of that workflow.
workflow-run-id=workflow-run-<unique>	Identifies one specific run. A new value is generated each time the workflow starts.

Output data produced by a workflow is tagged with:

Tag	Meaning
workflow-job-workdata	Marks this data as output produced by a workflow job.
workflow-id=<id>	Same workflow id as above.
workflow-run-id=workflow-run-<unique>	Same run id as above.

Your own step tags are kept alongside these. Because both jobs and their output data share the same workflow-run-id, you can search for that one tag to retrieve everything a single run created.

The execution report

Every run produces an execution report — a JSON summary of what happened. Use it to check the outcome, trace which jobs ran, and see what failed. It contains:

Field	Description
Workflow id and run id	The same identifiers used in the tags above.
Executor job	A link to the job that ran the workflow (if applicable).
Start and end time	When the run started and ended (UTC).
Execution state	The overall outcome (for example, completed or failed).
Variables	The variable values actually used for this run.
Processing step results	For each step: its id, a link to its job, and that job’s final state.
Errors	For each problem: a message, the step that caused it, and a link to the job involved.
Report version	Lets the format evolve without breaking tools that read it.

For advanced users: the workflow format

Behind the scenes, a workflow is stored as a JSON document describing its steps, connections, and variables. You do not need to touch this to use the portal editor — but if you want to generate, validate, or version workflows programmatically, a formal JSON Schema is available.

The schema describes every field, its type, and which values are allowed, and can be used with most editors and validators for autocompletion and error checking.

• Download the workflow JSON Schema

How variables are resolved

Variables are a simple text substitution in the workflow-definition file. You reference a variable with the placeholder ${{name}}, and before the run the platform replaces every occurrence with the variable’s value. There are two ways a placeholder can appear:

• Whole value — the placeholder is the entire JSON value, including its quotes, e.g. "limit": "${{max_results}}". Here the quotes are stripped during substitution, so number and boolean variables produce JSON primitives ("limit": 10, "enabled": true) rather than strings ("10", "true").
• Embedded — the placeholder is part of a larger string, e.g. "query": "prefix=${{name}}&suffix". The surrounding quotes stay and the value is spliced in as text.

Each variable must be declared in the workflow definition’s variable list to be recognized — only declared variables appear in the portal for setting at run time, and a placeholder referencing an undeclared variable is an error.

variable_type must be one of:

• string
• number
• boolean

A declaration in the workflow’s variables list looks like this (name, value, and variable_type are required; display_name and comment are optional):

{
    "name": "max_results",
    "display_name": "Maximum results",
    "value": 10,
    "variable_type": "number",
    "comment": "How many items the data selector keeps"
}

Elsewhere in the definition you reference it with its placeholder. Because max_results is a number and the placeholder is the whole value ("${{max_results}}"), the quotes are stripped during substitution:

{
    "limit": "${{max_results}}"
}

After substitution this becomes valid JSON with a numeric value:

{
    "limit": 10
}

Definitions and reports are WorkData

In PineXQ, both the workflow definition and the execution report are just WorkData. You can view them directly in the portal or download them with the PineXQ client, like any other data. Each carries its own media type:

Artifact	Media type
Workflow definition	application/vnd.pinexq.workflow.definition+json
Execution report	application/vnd.pinexq.workflow.report+json