Jepsen: Orkes Conductor 5.3.2

Jepsen tested Orkes Conductor versions 5.3.2 through 5.5.0-rc2 using a custom harness and a single-task workflow executed by a custom Java worker across clusters approximating typical Orkes deployments (Postgres and Redis, ten nodes split between servers and workers). The system provides at-least-once delivery and promises workflows eventually reach a terminal state, but its architecture (stateless servers, a persistent DB, and a queue like Redis) permits concurrent servers and workers and intermittent failures. Tests injected process pauses, kills, and network partitions, observed state via getWorkflow()/getRunningWorkflow() calls, and enumerated anomalies including backend errors, duplicate IDs, missing executions, inconsistent inputs/outputs, null workflow IDs, and tasks that never executed. Research was funded by Orkes; Orkes published a companion blog and released patches.

Findings included frequent concurrent execution of the same task for a single workflow execution: with default lockLeaseTime (60s) Jepsen observed concurrent runs roughly once every six hours; reducing lease time to 1s raised that to every few minutes. A sweeper time-of-check-to-time-of-use bug re-enqueued tasks and amplified duplicate deliveries. Tests uncovered three serious safety bugs that caused Conductor to forget or abandon executions or mark workflows completed without running tasks, plus four minor startWorkflow errors (including null IDs). Orkes released 5.5.6 patching all serious safety issues, reducing concurrent execution and fixing two startWorkflow errors; two remaining startWorkflow issues remain under investigation. Delayed execution behavior aligns with design and requires no change.