1 / 18
rx-controls-suite · n8n Orchestration
Same Scan.
No Python At All.
The same reactive pipeline, driven this time by a visual node graph with no in-process Python step — a genuinely different problem, and the findings it forces into the open.
Igor Khokhriakov  ·  Principal Software Engineer
github.com/scientific-software-hub/rx-controls-suite
2 / 18
00 · The Ladder A Deliberately Harder Question Each Rung Stress-testing one claim from three directions
The claim under test

rx-controls-suite doesn't compete with orchestrators — it feeds them. Bluesky, Prefect and n8n each try to break that claim from a different angle, and each reports what actually broke.

  • Bluesky — easy: Python, scientific, already loop-based. The rx pipeline drops straight into ophyd-shaped devices.
  • Prefect — harder: Python, but no scientific assumptions. The scan has to explain itself to a general-purpose task graph.
  • n8n — hardest: not Python at all. This talk.
Same core, same safety gates, same HDF5 file every time. What changes is how much of the orchestrator can hold Python state — and n8n holds none.
3 / 18
01 · What Is n8n Five Words You'll See In Every Screenshot A vocabulary primer — same shape as the Prefect talk's
  • Workflow — the node graph. Here: the guarded tomography scan, drawn as boxes and arrows instead of a Python function.
  • Node — one step: a form, an HTTP request, a Switch, a Wait.
  • Execution — one run of the workflow, with its own log of every node's input and output JSON.
  • Form Trigger — a web form that starts an execution; no API client required to launch a scan.
  • Wait — a node that parks the execution until a webhook call resumes it — how this workflow survives a beam-loss pause.
Compare to Prefect

Flow/Task/Run/Artifact/State all assumed a Python process underneath. Workflow/Node/Execution assume nothing — every node is just an HTTP call in and a JSON blob out.

What that costs

No shared Python objects between steps. Whatever state the scan needs has to live somewhere else — see slide 6.

4 / 18
01 · What Is n8n Why n8n Gets A Different Treatment Not a re-skin of the Prefect flow
The obstacle
  • n8n has no in-process Python step for this stack.
  • n8n 2.0 didn't leave Python behind — it replaced the legacy Pyodide Code node with native Python on external task runners.
  • A task-runner sandbox is still not where you run a live caproto Context and an rxtango event loop against a real beamline.
The fix
  • The rx scan is exposed as an HTTP+SSE service (scan_service.py, :8030).
  • n8n's visual node graph becomes the orchestration — every step a plain HTTP call against that service.
This isn't a workaround bolted on to make n8n fit. It's what turning an experiment into a REST-addressable resource actually requires — and the rest of this talk is about what that trade buys and costs.
5 / 18
02 · Architecture One Reused Core. A Service In Between. Same core as Prefect's talk, one more layer
Reused, Unchanged
scan_core.py
sweeps · ScanEvent stream · tier-2 watchdog · drain
New For n8n
refine.py + rx_n8n.py
quality ledger · 3 adapters
This Talk
scan_service.py :8030
FastAPI — owns the session & the cursor
n8n · :9000 — the node graph, driving the service by HTTP
Form Trigger starts a scan · Switch routes the cursor's action · Wait parks on beam loss, resumed by webhook
scan_core.py imports nothing from n8n or Prefect. Only the two layers above it are new.
6 / 18
02 · Architecture The Cost And The Buy What turning an experiment into a REST resource actually trades
Prefect could just pass objects

RxLoop, the caproto Context, the shared health observable, the open ScanRun, the sweep cursor — plain Python objects, passed between persist_result=False tasks in one process.

n8n can't

Every step is a separate HTTP call. That state has to become a server-side session (ScanSession in scan_service.py) — n8n is left holding only control flow.

Not a workaround — the cost of making the experiment a resource. The buy: any HTTP client can now drive or observe it, not just this one node graph.
7 / 18
02 · Architecture The Whole Surface, Nine Endpoints scan_service.py — everything n8n calls
Lifecycle
POST /scan
Opens the session, arms the beamline, starts the shutter supervisor.
Cursor
POST /scan/{id}/next
The one decision node — returns what to do next.
Acquire
POST /scan/{id}/sweep
Runs one pass-1 sweep to completion or a watchdog pause.
Acquire
POST /scan/{id}/refine
Re-acquires exactly the LOW-quality indices queued by /assess.
Recovery
POST /scan/{id}/wait-healthy
Arms the beam-recovery webhook, returns 202 immediately.
Decision
POST /scan/{id}/assess
Runs refine.py::assess — converged, refine, or exhausted.
Lifecycle
POST /scan/{id}/finalize
Always runs — closes the HDF5 file, writes the summary.
Test rig
POST /sim/fault
Injects a scenario — same call the "Inject Fault" form makes.
Live view
GET /events
SSE, per frame — the continuous feed n8n's own log can't give.
/sweep and /refine are sync def endpoints so they can block inside drain — the same rx-loop-thread boundary the Prefect talk covered, just crossed by FastAPI's threadpool instead of a Prefect task.
8 / 18
Part Two
The Node Graph
Fifteen nodes, one Switch, one loop back to the cursor
9 / 18
03 · The Graph · 1 of 2 Guarded Tomography Scan.json Every node, exactly as it ships
↻ the refinement cycle — slide 12
Form TriggerExperiment Form
▾
HTTPCreate Scan
▾
HTTP · cursorNext
SwitchRoute
sweeprefine assessdone abort
Sweep
Refine
Assess
↩ Next
Finalize OK
End OK
Finalize Aborted
Interlock Stop
IF · Check Aborted
true → Finalize Aborted false ↓
IF · Check Watchdog
true → Arm Resume false
↩ Next
Arm Resume
Wait Beam
↩ Next
10 / 18
03 · The Graph · 2 of 2 The Service Owns The Cursor One Switch collapses sweeps and refinement into one loop

/next returns one decision, and exactly one Switch node fans it to five branches — sweeps and refinement iterations collapse into the same loop.

{
  "action": "sweep",
  "iteration": 1,
  "sweep_index": 0,
  "sweep_count": 3,
  "index_from": 0,
  "index_to": 4,
  "remaining": 4,
  "total": 12
}
Why not Loop Over Items

n8n has a documented bug: a Wait-on-webhook node inside a Loop Over Items gets its item reprocessed on resume. One Switch around a session-owned cursor sidesteps it entirely — there's no loop node to re-enter, just a decision that gets asked again.

This is a deliberate deviation from an earlier sketch that used a Loop Over Sweeps node — changed for exactly this reason.
11 / 18
Part Three
A Cycle, Not A DAG
The retry list doesn't exist until the measurement has happened
12 / 18
04 · The Cycle · 1 of 4 Same Shape, One Structural Difference A DAG can't express a decision that doesn't exist yet
Prefect A Straight Line

prepare_beamline → run_sweep ×3 → finalize

  • Step count known before the first frame is acquired.
  • Beam loss pauses a step; it never adds one.
n8n A Loop With A Back-Edge

Assess → Next re-enters the Switch when quality is below target.

  • The retry list doesn't exist until the ring orbit has been read at the moment each frame was taken.
  • Iteration count is unknown at launch — bounded only by max_iterations.
Tango says the orbit was out of spec at the instant a frame was taken — that fact only exists after acquisition, so the graph that reacts to it can't be a DAG with a fixed step count.
13 / 18
04 · The Cycle · 2 of 4 What Iteration 2 Actually Fixes The coverage strip — the dashboard's centerpiece
Pass 1 complete · orbit_drift injected · 75% quality (9 / 12 OK)
↓  refine iteration 2  ↓
After refinement · 100% quality (12 / 12 OK) — target was 90%
unacquired OK LOW re-acquired
14 / 18
04 · The Cycle · 3 of 4 Three Honest Outcomes, Not Two refine.py :: assess
Converged
All projections meet spec, or quality % ≥ target — a few LOW frames are tolerable if the target allows it.
stop=True, converged=True
Keep Refining
Below target, iterations remain — queue the LOW indices and loop back through the Switch.
stop=False, converged=False
Exhausted
Hit the iteration cap with the beam still out of spec — reported honestly as not converged, never as a silent pass.
stop=True, converged=False
15 / 18
04 · The Cycle · 4 of 4 Five Fault Responses, Not Four The fifth row is the reason this demo exists
Injected
Handled By
Visible As
brief beam dip
rx tier-1 gate, unchanged
nothing — absorbed silently, as today
beam low > watchdog_s
sustained_low → Wait node
execution Waiting, auto-resumed on beam recovery, sweep continues mid-projection
vacuum_burst
interlock → Stop And Error
execution Error; shutter closed; partial HDF5
orbit_drift, cleared
the refinement loop
iteration 2 re-acquires LOW projections; quality % climbs to target
orbit_drift, held
max_iterations guard
finishes not converged — not a crash, not a silent pass
Row four is the one the Prefect flow has no equivalent for — and the reason this demo exists.
16 / 18
05 · The Bridge The Bridge Got Shorter, Not Just Different rx_n8n.py — three adapters, not five
Concern
rx_prefect.py
rx_n8n.py
Resuming a pause
Hops onto a private ThreadPoolScheduler(1) first — resume_flow_run's @async_dispatch sniffs a running loop and silently no-ops otherwise.
Plain GET $execution.resumeUrl — no thread-local run context to respect, fired straight from on_next.
Adapter count
5 — log_event, ProgressTracker, sweep_table, pause_until_healthy, drain
3 — event_json, EventHub, resume_on_healthy
Gotcha
async_dispatch's loop-detection trap — see the Prefect talk
The Wait node's restart hook answers GET, not POST — a POST 404s with "no waiting webhook."
The orchestrator boundary is an HTTP call, not an in-process SDK with hidden context — that difference is a finding, not an implementation accident.
17 / 18
06 · The Live-Feedback Gap Stated Plainly, Not Hidden What a general-purpose automation tool costs at a beamline
The gap
  • n8n shows one JSON blob per node execution.
  • Prefect showed a progress artifact updating twice a second.
  • Not a bug in this demo — the actual cost of choosing a general-purpose tool.
So the continuous view comes from elsewhere
  • scan_service.py serves its own instrument-panel dashboard at :8030 — the coverage strip, loop/machine/beamline panels, a per-frame SSE log.
  • live_dashboard.py (:8000) tracks an n8n-driven scan unchanged — it only ever read the TOMO:SCAN:* PVs, which this service writes exactly as the plain script did.
  • scan_report.ipynb opens the resulting HDF5 unchanged too — same compound dtype, same projections dataset.
18 / 18
Thank You
Same Reactive Pipeline.
No Python Required To Drive It.
github.com/scientific-software-hub/rx-controls-suite
Go deeper
demo/workflow-engines/README.md — run it yourself
demo/workflow-engines/n8n_workflows/*.json — both workflows, importable as-is