Loops, lookups, queries, and performance

This article explains how loops, lookups, and queries work in Pipelines, with a focus on performance.

Loops

A loop repeats a sequence of actions until a specific condition is met.

In this example, a For each loop is used to send a Slack message for each record in a list:

For each loop step in the pipeline designer, iterating over records to send a Slack message

Process nested lists in a loop

Some steps return structured data where the items you want to loop over are nested inside an object. For example, a custom action might return a company object that contains a list of departments, and each department might contain a list of employees.

You can point a loop directly at a nested list within a step's output—no Jinja required to extract it first.

To configure a loop over a nested list:

Add a Loop step after the step that outputs your structured data.
In the For each item in field, select the upstream step, then select the nested list path within its output (for example, Custom action | departments).
In the Refer to each as field, enter a name for each item in the list (for example, department).

Inside the loop body, use {{department.name}} to reference the current item's fields. The original step output remains accessible under its default variable (for example, {{ae.company}}).

Scope: A name defined in Refer to each as is only available inside the loop body where it is defined. Steps outside the loop cannot reference it. Steps nested inside the loop can reference names from any enclosing loop.

Renaming the variable for the selected reference name will update all Jinjas, branch conditions, step filters, and loop/step target references.

Example: Loop over departments from a custom action

A custom action returns a structured response containing a company name and a list of departments. Each department has a name, and contains a list of employees.

Configure the outer loop:

For each item in: Custom action | departments
Refer to each as: department

Inside that loop body, you can reference:

{{department.name}} — the current department's name
{{ae.company}} — the company name from the original step output

To process employees within each department, add a second loop inside the first:

For each item in: department | employees
Refer to each as: employee

Inside that inner loop, all levels remain accessible:

{{employee.name}}, {{employee.role}} — the current employee
{{department.name}} — the department this employee belongs to
{{ae.company}} — the company from the original step output

Example: Loop over a sibling list

If the custom action output contains two lists at the same level — for example, departments and offices — you can loop over each independently. One loop iterates over departments (Refer to each as: department). A separate loop at the same level iterates over offices (Refer to each as: office).

Configure the offices loop:

For each item in: Custom action | offices
Refer to each as: office

Because each loop uses a distinct name, the original step output remains available for both.

Limits

You can process as many levels of nesting as needed, as long as nested loops are used. You can't use a single loop to unnest multiple levels at once.
Names in Refer to each as cannot duplicate an existing step variable in the same pipeline.

Lookups and queries

Lookups and queries both retrieve data from connected systems, but they differ in scope, performance, and best use.

Lookup

A lookup searches for an item based on specific criteria and returns one result. They can use any unique field.

Lookups are easier to set up and perform faster. When possible, use a lookup to improve performance.

Query

A query searches all records in a selected account and returns a list of results. Queries are more flexible than lookups because they can filter on any field and return multiple results.

Queries are better suited for bulk operations that process or transfer groups of records. Although they use more resources and often require a loop, Jinja, or Insert to Quickbase step to process the results that are returned, they are more efficient overall when you're handling many records at once.

Treat a query like a report—limit it to only the fields and records you need. For best performance, run the query first, then loop through the results or pass them to a bulk operation step.

Bulk data processing

When you need to move data at a higher volume, use one of these bulk data processing methods:

If you need to process a large number of incoming changes in real time, use a bulk trigger.

Documentation Index