Running Pipelines

Learn the ways that pipelines can be triggered, how runs receive inputs and produce outputs, and how to view and manage run history.

A pipeline can be run from the application by simply clicking the “Run” button from the pipeline builder. Running a pipeline queues up the run and makes it available to runners, which acquire the run, execute it, and report detailed log information back to the application.

About jobs

While pipelines can be run directly, you can also create one or more jobs, which enable more sophisticated control over how and when a pipeline runs. A job is like a “run configuration” for a pipeline. It defines a pipeline revision, input variables, trigger type, and runtime settings for a pipeline run.

Specifying inputs for a run

If your pipeline has input variables, these will appear in the run dialog when a user runs your pipeline so that they can override the run parameters. Variables marked as “hidden” will not appear in run dialogs, but their values will be filled in at runtime. If a pipeline variable is marked as “default,” then the value specified in the pipeline will fill in for values omitted at runtime.

Running a pipeline on a schedule

To run a pipeline on a schedule, create a job. To create a job, visit to the “Jobs” page from the main navigation and then click “New Job”. In the dialog, select “Scheduled” for the trigger type, and then fill out the schedule fields.

For a scheduled job to run, it needs an available runner assigned to the associated project. If your project doesn’t have any runners, jobs will fail to create runs. This prevents the scheduler from creating a backlog of queued runs which are flushed all at once when the runner is started.

Incoming webhook job trigger

Jobs can be configured with an “incoming webhook” trigger type, which enables powerful integrations with other systems that fire webhook events. For example, you can configure a job to run a pipeline every time a commit is made to a git repository on GitHub or GitLab.

See here for more information about webhooks.

To create an incoming webhook job, visit to the “Jobs” page from the main navigation and then click “New Job”. In the dialog, select “Incoming Webhook” for the trigger type, and then optionally fill out the webhook fields. These fields can be changed after the job is created.

Incoming webhook jobs have several powerful features for transforming and validating the input request.

Transforming input variables

The Variables Transform field is an expression which can be used to dynamically create the input variables for a pipeline run from the webhook request. For example, for a pipeline with a single string variable called my_string, we could use this expression for the variables transform:

{
  "my_string": "some literal value"
}

This example “hard codes” the value of the variable my_string. Often we’ll want to instead compute the value of this variable from the incoming request data. For example, if the external system sends a JSON body like this:

{
  "company": {
    "id": "5"
  }
}

We can extract the company id into the my_string variable using this expression:

{
  "my_string": body.company.id
}

In addition to body, we can also use the headers object to access the incoming request headers. Headers are converted to lowercase, so if the company id is instead provided in a X-Company-Id header, we can access it using this expression:

{
  "my_string": headers["x-company-id"]
}
Validating incoming requests

Credentials and project variables are also available in webhook expressions, which is useful for performing custom authentication. Let’s say our external system adds an X-Auth-Token header to webhook requests, and we want to validate that this token equals a secret value.

  1. Add a credential to the project with type “API Token”, and enter the secret token in the “Token” field. Let’s say the credential ID is my_cred.
  2. In the webhook job, use the following expression in the “Validator” field to check that the X-Auth-Token header matches the credential value:
credential("my_cred").token == headers["x-auth-token"]

Project variables can be accessed using the vars. syntax, so an equivalent validator can be created by using a project variable instead:

vars.my_cred == headers["x-auth-token"]

Some webhook systems use basic authentication. To validate these requests, use the Credential field of a webhook job. This field should be a username/password type of credential. The parsing of the username and password is performed automatically behind the scenes, so you don’t need to write an expression for this case.

Finally, for systems that don’t provide an authentication mechanism, you can configure an IP whitelist for a webhook job. Any requests not matching the whitelisted IPs will be rejected with a 401 status code.

Controlling the webhook response

Since some systems require that webhook endpoints return specific status codes, you can override the success status code in the job. This code will be returned if the webhook successfully executed, which may not mean that the request created a run, for example if the Condition expression evaluated to false. To force an incoming webhook to always return a specific status code, even when errors occur, you can turn on the Ignore Errors field.

Disabling jobs

When a job is disabled, it will never run, even if it has a scheduled trigger type. The Run button will be disabled in the application, and calls to the API will return an error code.

Scheduled jobs can also be configured to be automatically disabled when any runs fail.

Suppressing output from a run

If you’d like to prevent the runner from sending any detailed data about the run back to the application, you can configure a job to suppress its reporting.

  • Suppress variables: The input variables are only stored temporarily until the runner begins executing the run, and then they are deleted. Run variables will not be available from the run page.
  • Suppress outputs: Pipeline outputs will not be sent to the application by the runner.
  • Suppress events: The detailed event log from the run will not be sent to the application by the runner. Note that this can make it difficult to debug pipeline runs, but it also ensures that no step output logs leave the runner.

With all three of these settings enabled, the runner will only send metadata about the run to the application, which creates a high degree of data isolation on the runner. This is useful when your pipelines are working with highly sensitive data, and you don’t want this data ever reaching the Refactr servers.

Getting outputs from a run

After a pipeline is finished executing, its evaluated outputs are available from the run history page as well as from the API.

To view outputs from the application, open the run from the “Run History” page, then select “Outputs” from the dropdown at the top.

Masking secrets in run events

All credential and “SecureString” variable values will be automatically masked in any run events. If the secret appears in run event logs, it will be replaced by *****. This helps prevent sensitive data from leaking into your run history.

When a secret value contains multiple lines of text, each line is treated as a single secret. While this reduces the chances of a multiline secret avoiding the masking routine, it can also result in over-masking. For example, if your secret contains formatted JSON, then the first line will be {, which causes all instances of { to be replaced with *****, which might not be what you want. To work around this problem, simply enter JSON secrets as a single line.

Queueing up many runs

When many runs are created quickly, they are placed into a queue to be processed by available runners. When no runners are assigned to a project, or there are not enough runners to process the runs, the maximum run queue length will eventually be reached. When the queue is full, you will no longer be able to create new runs from the application or API.