Building Pipelines

Learn how to build pipelines using the application.

The application provides a drag-and-drop pipeline builder that accelerates the process of creating pipelines.

Drag step modules from the sidebar on the left into the design area to add steps to a pipeline.

Step modules

The action performed by a step is determined by its type, or step module. The list of built-in modules can be seen in the left-hand sidebar of the pipeline builder.

Each step module has its own set of inputs, or properties. To see a step’s properties, drag a step into the design area and then select it.

For a full list of step modules and their properties, see the step module reference.

Pipeline include steps

In addition to the built-in step modules, a pipeline can include another pipeline as one of its steps. The variables from the inner pipeline are exposed to the outer pipeline as step properties, which facilitates passing data from one pipeline to the next.

Pipeline includes are a powerful tool for creating pipelines that act as reusable modules.

The step properties for an included pipeline contain the included pipeline’s IDs (pipeline_id and revision), as well as a variables field which contains the input variable values.

Using dynamic values in step properties

All step properties, and a few additional step fields, can contain expressions rather than a value. Expressions can be used to compute values dynamically at runtime, as well as read and transform data from previous steps.

In the pipeline builder, fields that support expressions have an toggle button on the right side of the field. Toggling to expression mode switches the input to a code editor where an expression can be entered directly.

For more information about expressions, see the expression reference.

Accessing variables from steps

To access a variable from an expression, assuming the variable key is my_var, we can use:

vars.my_var

vars.<var key> is available from any expression, and contains all variables available in the current scope. Here are all variable scopes:

  1. Project variables
  2. Pipeline variables
  3. Variables set from the Set Variables step module.

Variables lower in this list will override those higher up if they have the same key.

For each included pipeline, a new scope is created. You cannot access parent variables directly from an included pipeline. Instead, you must define input variables for the child pipeline, and pass values through from the parent pipeline’s step properties.

Passing data between steps

Expressions evaluating within step properties have access to all other steps in the pipeline. To get another step, use the following expression:

steps.<step id>

This expression evaluates to an object. All steps have a common set of fields, such as name, tags, and properties. The properties field differs for each step module. Properties fields can be seen in the step module reference.

For example, to retrieve the method of an HTTP Request step with ID http, we can use the this expression:

steps.http.properties.method

After a step executes, it has an additional result field that contains data about the step execution result. The fields available on the result object differs for each step module, however all step modules will have a status field that indicates whether the step execution succeeded or failed.

For example, the following expression evaluates to true if the step with ID http succeeded:

steps.http.result.status == 'Succeeded'

Pipeline include steps have one additional field on their result object containing the evaluated outputs of the pipeline. To retrieve an output with key ipaddress on a pipeline include step with ID my_include, we can use this expression:

steps.my_include.result.outputs.ipaddress

Accessing credentials

Credentials can be retrieved dynamically by expressions, which is useful when the built-in credential step module integrations don’t quite do what you need (for example, when using the HTTP module with custom Authorization token headers).

To get credential data, pass the credential ID to the credential() helper function, like this:

credential('my_cred_id')

This expression evaluates to an object containing one or more data fields associated with the credential type. For example, to retrieve the password field of a Username/Password type of credential, use the following expression:

credential('my_cred_id').password

Because credentials are also a built-in variable type, it’s common to create a top-level variable in a pipeline and then pull the credential data out from an expression, which would look like this:

credential(vars.my_cred_var).password

Skip a step conditionally

All steps have a condition field which must be an expression. If this field evaluates to false, then the step will be skipped.

Step conditions combined with pipeline includes can be very powerful for advanced pipeline control flow. When a pipeline include step is skipped, all steps within that pipeline are also skipped.

Ignoring errors in a step

Errors in step execution can be ignored by setting the ignore_errors field to true. The result field of the step can be used by subsequent steps to make decisions based on whether the step succeeded or not. Most other result fields are also available, such as the stdout and stderr of script steps.

Restoring old revisions of a pipeline

When you save a pipeline, a new revision is created. Revisions are a linear history, and each revision is given a number. The revision history allows you to go back in time and restore previous states of a pipeline, as long as the pipeline is not deleted. The revision history is limited to the pipeline itself, so it does not include changes to the pipeline name or image.

To restore an old version of a pipeline, open the pipeline in the builder and then open the “Revisions” sidebar. Find your desired revision and open it. Then save the pipeline again, bringing that revision to the top of the history.

It’s also sometimes useful to clear a pipeline’s revision history. To do this we can simply clone the pipeline into the same project and then delete the old pipeline.

Installing tools used by pipeline steps

Many built-in step modules require an underlying tool, such as a programming language runtime or a CLI program. Supported tools can be automatically installed at a specified version during pipeline execution.

There are two ways to install a tool from a pipeline:

  1. Using the “Tool Version” field on a supported pipeline step. Using this method, the tool is only installed for the subprocess run by the step, and will not propagate to other operating system sessions.
  2. Using a tool installer step module. The step modules install the tool globally on the runner machine by modifying the PATH and environment.

Tool installations are cached on the runner for each version of the tool.