Coming in Zeebe 0.22: Awaitable workflow outcomes

The upcoming 0.22 release of Zeebe includes a feature many users have been asking for: the ability to start a workflow and retrieve its outcome with a single command.

The new gRPC command CreateWorkflowInstanceWithResult is available for testing in the current SNAPSHOT Docker image of Zeebe and the zeebe-node-next version of the Node.js client.

This command starts a workflow instance and returns the outcome when the workflow completes.

Use-cases

A common scenario is to start a workflow in response to a REST request, and send back the outcome from the workflow in the REST response. Previous implementations of this relied on a polling worker to retrieve the outcome, and the use of subscriptions to correlate it with the REST request context (See the “Zeebe Workflows Inside a REST Request/Response” blog post and the zeebe-node-affinity package). This scenario makes sense when the workflow is short-running.

A Tour of Related Issues and PRs

The initial feature request for awaitable workflow outcomes – #2896 – got a lot of traction and input from the community. This is the best way to get something implemented in Zeebe: open a feature request, make a compelling use-case with a good description, and get wider community support for it.

The work to implement the functionality in the broker has been done by Deepthi Akkoorath, and is in this pull request. If you work with Zeebe and are interested in this feature, I recommend you read through the changes. I can’t overstate the value of reading the source code to build up your understanding of using Zeebe. All the answers are in there. You may not understand everything, but you never will if you don’t start somewhere, so dive in!

Awaiting workflow instance results is implemented in an upcoming release of the Java and Go client (#3286), and in the testing version of the Node.js client, zeebe-node-next (example).

The work to support it in zbctl is still to be done, and is being tracked here.

An example using Node.js

Here is how you use the feature with the current SNAPSHOT Zeebe server and zeebe-node-next:

import {} from 'zeebe-node-next'

async function main() {
    const zbc = new ZBClient()
    await zbc.deployWorkflow('./test.bpmn)

    const initialVariables = {
        clientId: 1
    }

    const outcome = await zbc.CreateWorkflowInstanceWithResult(
        'test-process-id',
        initialVariables,
    )

    console.log(outcome.variables)
}

main()

Dealing with gRPC request timeout

An important consideration here is the gRPC request timeout. By default, this is the request timeout configured on the Zeebe gateway (15 seconds, out of the box). If your gRPC request times out before the workflow completes, the client will receive a request timeout error back from the gateway. If this happens, you will not have a workflow instance Id for the workflow that you just created.

You can retry the request – note that each retry creates a new workflow instance, so you should ensure that any side-effects in the workflow are idempotent. If the broker timeout is due to load, it may succeed on a subsequent execution. But if the workflow takes intrinsically longer than the timeout, you will never get a result back. So you need to configure the request timeout to be of sufficient length.

You can override the gateway request timeout by using a different signature for the createWorkflowInstanceWithResult call in the Node client (the C#, Go, and Java clients use a builder pattern, so constructing the command differs in those clients):

async function main() {
    const result = await zbc.createWorkflowInstanceWithResult({
        bpmnProcessId: processId,
        variables: {
            sourceValue: 5,
            otherValue: 'rome',
        },
        requestTimeout: 25000,
    })
    return result
}

main()

Also to note: since no workflow id is returned until the workflow completes, if your system experiences a failure while the request is pending (either client-side or on the broker), you will have no way to correlate the request.

Constraining the returned variables

A further feature (#3253), that hasn’t been merged at the time I wrote this blog post, is the ability to constrain the variables returned by the call.

When this lands, you’ll be able to do this:

async function main() {
    const result = await zbc.createWorkflowInstanceWithResult({
        bpmnProcessId: processId,
        variables: {
            sourceValue: 5,
            otherValue: 'rome',
        },
        fetchVariables: ['finalBalance']
        requestTimeout: 25000,
    })
    return result.variables.finalBalance
}

main().then(console.log)

Summary

The new CreateWorkflowInstanceWithResult command allows you to “synchronously” execute workflows and receive the outcome. It’s a popular feature request in the community and will reduce the complexity of Zeebe systems in production that have been using complex patterns to achieve this functionality.

The feature is ideally suited to workflows that retrieve information in response to a request, and return it in the response. If your workflow mutates system state, or further operations rely on the workflow outcome response to the client, take care to consider and design your system for failure states and retries.

Bonus

I live stream coding with Zeebe and Node.js, weekly on Twitch.

Here are a couple of videos from this week’s stream, implementing support for this feature in the Node client.