Running Pipelines

Learn the ways that pipelines can be triggered, how runs receive inputs and produce outputs, and how to view and manage run history.

A pipeline can be run from the application by simply clicking the “Run” button from the pipeline builder. Running a pipeline queues up the run and makes it available to runners, which acquire the run, execute it, and report detailed log information back to the application.

Specifying inputs for a run

If your pipeline has input variables, these will appear in the run dialog when a user runs your pipeline so that they can override the run parameters.

To add a variable to a pipeline, open the pipeline builder and select the “Variables” tab in the top-right, and then click “Add Variable”.

Running a pipeline on a schedule

To run a pipeline on a schedule, create a job. A job is a “run template” for a given pipeline. It defines the pipeline revision, input variables, and runtime settings for a pipeline run.

To create a job, visit to the “Jobs” page from the main navigation and then click “New Job”.

For a scheduled job to run, it needs an available runner assigned to the associated project. If your project doesn’t have any runners, jobs will fail to create runs. This prevents the scheduler from creating a backlog of queued runs which are flushed all at once when the runner is started.

Disabling jobs

When a job is disabled, it will never run, even if it has a scheduled trigger type. The Run button will be disabled in the application, and calls to the API will return an error code.

Scheduled jobs can also be configured to be automatically disabled when any runs fail.

Providing default values for run inputs

Pipeline variables may be marked as hidden, which has two effects:

  1. The variables are not displayed as form fields in run dialogs. Users running your pipeline will not see these variables.
  2. When a run is created from the application or API, hidden variable values are merged into the final variables for the run. This allows you to provide default values for a pipeline run that don’t need to be provided at creation time.

Suppressing output from a run

If you’d like to prevent the runner from sending any detailed data about the run back to the application, you can configure a job to suppress its reporting.

  • Suppress variables: The input variables are only stored temporarily until the runner begins executing the run, and then they are deleted. Run variables will not be available from the run page.
  • Suppress outputs: Pipeline outputs will not be sent to the application by the runner.
  • Suppress events: The detailed event log from the run will not be sent to the application by the runner. Note that this can make it difficult to debug pipeline runs, but it also ensures that no step output logs leave the runner.

With all three of these settings enabled, the runner will only send metadata about the run to the application, which creates a high degree of data isolation on the runner. This is useful when your pipelines are working with highly sensitive data, and you don’t want this data ever reaching the Refactr servers.

Getting outputs from a run

After a pipeline is finished executing, its evaluated outputs are available from the run history page as well as from the API.

To view outputs from the application, open the run from the “Run History” page, then select “Outputs” from the dropdown at the top.

Masking secrets in run events

All credential and “SecureString” variable values will be automatically masked in any run events. If the secret appears in run event logs, it will be replaced by *****. This helps prevents sensitive data from leaking into your run history.

When a secret value contains multiple lines of text, each line is treated as a single secret. While this reduces the chances of a multiline secret avoiding the masking routine, it can also result in over-masking. For example, if your secret contains formatted JSON, then the first line will be {, which causes all instances of { to be replaced with *****, which might not be what you want. To work around this problem, simply enter JSON secrets as a single line.

Queueing up many runs

When many runs are created quickly, they are placed into a queue to be processed by available runners. When no runners are assigned to a project, or there are not enough runners to process the runs, the maximum run queue length will eventually be reached. When the queue is full, you will no longer be able to create new runs from the application or API.