HTTP channel

  • 14 minute(s) read
Prev Next

Purpose

The HTTP channel allows you to connect to APIs, working on HTTP. Those may be APIs we don’t have native support for, or APIs we provide dedicated channels for, but with missing specific rarely-used endpoints. 

Generic fields

There is some basic information you should provide:

  • Account name—Unique name to distinguish this account from all other accounts you may use with the channel.
    The unique identifier of your account is constructed from the account name, authentication type and base URL. This lets you create different accounts for the same API but with different credentials by choosing a specific account name for each of them. If you try creating a new account using an existing name, base URL, and authentication type, it updates the existing account instead.

  • Base URL—Allows the account users to connect only to endpoints, derived from this base URL. This way your data and credentials won’t leak to any other service. The base URL should be a valid URL, containing at least the scheme and host. But it could contain path and even query params.
    The endpoint paths you define in your steps may contain whole URLs as well. In this case, the scheme and host should be identical to the base URL scheme and host, and the path should at least begin with the path set in the base URL. In this situation, the path’s query params override the Base URL’s params with the same key.
    This way you may provide some meaningful defaults through the query params and override them only by need for the specific endpoint.

  • HTTP Headers—Contains a list of key value pairs, that are converted to proper HTTP headers with each request made with the given account. Use this to provide additional information through the headers if your API requires it. It is convenient to put them in the connection, as the headers set here are used for each request. For security reasons, the headers set in the connection cannot be overridden by the step. So, if some of your endpoints expect different values for the header, you should put it in the step instead.
    Although rarely used, a header with an empty value is a valid construct, so you may provide a key with an empty value and Pipelines accepts it.
    You can specify a header with the same key multiple times as well. As per standard, the values are joined together, separated with a comma, like this:

       Header-Key: value1, value2, value…  

Connection

Most of the APIs require authentication to be used. In our connection dialog, you can provide information about the authentication type, as well as the needed credentials and Pipelines handles the authentication for you.

As with all other channels, we store your credentials in a secure way.

The channel supports the following authentication types:

  • No Authentication

  • Basic Authentication

  • Bearer Token

  • API Key

  • JWT

  • OAuth 2.0

No Authentication

Select this if your API doesn’t require authentication. You should still create an account, providing at least base URL for the API and optionally any HTTP headers your API may require.

Basic Authentication

The basic access authentication is a simple authentication method, using a username and password.

The produced header looks like this:

Authorization: Basic <credentials>, where <credentials> is the Base64 encoding of username and password joined by a single colon.

You need to provide your username and password. Quickbase automatically updates the format.

Bearer Token

The bearer token authentication’s header looks like this:

Authorization: Bearer <token>

Where <token> is the bearer token you provide in the connection dialog, without any transformations.

API Key

Use this option if your API requires a custom header for authorization or integrates the authorization key in the query parameters of the URL.

The resulting header looks like this:

<API Key Name>: <API Key Value> without any transformations.

By default, the provided name and value pair is converted to the header, but if your API requires otherwise, there is an option to attach them as query parameters to the request URL. The API Key query param cannot be overridden by the path or query parameters field in the step. In this situation, the pipeline returns a validation error and won’t make a request to the remote endpoint.

JWT

The JWT authentication method is similar to the Bearer token authentication, but in this case pipelines generates the JSON Web Token for you, based on the provided parameters.

The final result is a header like the following:

Authorization: Bearer <token> where <token> is the generated JWT.

You may provide parameters to allow Pipelines to generate the JWT for you including:

  • Signing algorithm—This is the algorithm used to sign the JWT key. We support both symmetric and asymmetric algorithms. What you use depends on the requirements of your API. It is valid to create a not signed JWT, so this field is optional, not recommended though.

  • Signing key—When a signing algorithm is selected, this field is required. The format depends on the type of the algorithm you selected above. For asymmetric algorithms, Pipelines expects a private key in PEM format, and for the symmetric algorithms, the key should be just a sufficiently long string.

  • Additional JWT headers—Provide additional JWT headers as a JSON object. These may be headers like:

    • jku— URL, referring to a resource for a set of JSON-encoded public keys, one of which corresponds to the key used to digitally sign the JWS

    • jwk— The public key that corresponds to the key used to digitally sign the JWS

    • kid— Key ID is a hint indicating which key was used to secure the JWS

Headers like those specified above are defined in the standards for JWT /RFC 7519: JSON Web Token (JWT) , JWS /RFC 7515: JSON Web Signature (JWS) and JWE /RFC 7516: JSON Web Encryption (JWE) .

The additional headers may also be used to replicate some of the claims in an unencrypted form. This might be used from the application to determine how to process the JWT before it is decrypted. Detailed information about this usage may be found here /RFC 7519: JSON Web Token (JWT)

  • JWT Claims—You can provide JWT claims as a JSON object. Refer to the detailed documentation for the allowed claims here /RFC 7519: JSON Web Token (JWT) or check the requirements of your API. Keep in mind that Pipelines adds expiration (exp) and Issued At (iat) claims for you if you fulfill the corresponding fields in the form.

  • Use Issued At (iat) claim?—If this option is checked, pipelines adds iat claim to the payload for you. The token is generated prior to each request, so iat points to a time shortly before the request itself.

  • Expiration time (exp) claim—Set the expiration time of the generated token. The default is 5 minutes. If you put 0 in the field, an exp claim is not added and the token never expires (not recommended). Otherwise, you may put the number of seconds until expiration like 3600 for an hour or a string like 2m / 4h / 2h30m / 1d3h / 1d3h20m.

OAuth 2.0

Pipelines guides you through the whole flow, prompting you to authenticate against your identity provider (when using the authorization code grant) and stores the access token and the refresh token, when present. If a refresh token is present, Pipelines authenticates you with each request with the stored access token and refreshes the token, when needed.

For this to work, you need to provide the account with the following information:

  • Authorization grant type—One of Client credentials and Authorization code grants.The one you use depends on the requirements of the API you’re trying to connect.

  • Authorization Endpoint—This is needed only for the authorization code grant. Pipelines forwards you to this endpoint, where you need to authenticate and grant access to pipelines.

  • Access Token Endpoint—This is the endpoint pipelines uses to obtain an access and, optionally, a refresh token.

  • Client identifier—You should register pipelines as a client at your API’s authorization server and populate the generated client identifier here. This way pipelines can identify itself properly against your identity provider. The callback URL to add to the registered client is: https://www.pipelines.quickbase.com/authorize 

  • Client Secret—Among the client identifiers, client secret is also generated.Copy it into this field, so pipelines can identify itself against your identity provider. This field may not be required by your API.

  • Scope—Scope of the access request. The value should be a list of space-delimited, case-sensitive strings, defined by your authorization server. This field is optional. 

Make request step

Use the Make a request step to make a request to a chosen API endpoint. This section explains configuration options, error handling, and limits.

HTTP Method

This should be one of GET / POST / PUT / PATCH / DELETE / OPTIONS / HEAD. If nothing is provided, GET is set by default. The documentation of your API documents what method requires the given endpoint. If the API is following the REST conventions, GET is used to obtain a resource, POST is used to create a resource, PUT and PATCH are used to update an existing resource. PUT expects you to provide all properties including ones that are not altered, while PATCH allows you to send only the changes made. DELETE is used to delete a resource. OPTIONS and HEAD are rarely used in this context.

PATH

This is the path to your endpoint. It may be relative to the Base URL, set in the connection or a full URL. If you provide a full URL it should conform with the base URL. This means that the schema and host should be identical, while the path should be prefixed or identical to the path in the Base URL. If query params are present in the path, they are joined with those from the base URL overriding params with the same name.

HTTP Headers

This is a list of key value pairs, transformed into HTTP headers and attached to the request. You cannot override the headers set in the connection. The headers defined here are added to those defined in the connection and are sent only for this specific step.

Like in the connection, you may set headers with empty values, as well as multiple values for the same header. Additionally, you may use jinja expressions in the step’s HTTP headers field.

Query Params

This is a list of key value pairs. Pipelines attaches them as query parameters to the request’s URL. The step query parameters take precedence over the path and the base URL. They replace parameters with the same key and attach everything else to the already present params. You may use jinja here as well.

Expected payload type

Quickbase supports natively JSON and YAML payloads in the response and parses them for you. If the response contains textual data that isn't in any of those formats, you should select TEXT from the select field instead. We return the result as is without any additional parsing. If nothing is selected, JSON is inferred.

Schema sample type

Use this field to advise pipelines about the response’s schema. This field is not present if you’ve chosen TEXT above. You could achieve this by providing an example response (sample) or directly using json / yaml schema from the documentation of your API. The value selected here guides pipelines on how to treat the input in the next field.

The format of the schema / sample doesn’t influence the expected payload type. There are APIs that may return different formats based on configuration, so it works to provide a YAML schema here and then to receive a response in JSON.

To enhance usability, Pipelines utilizes a non-restrictive schema. This is particularly helpful in the Quick Reference, where all schema fields with their type are listed, enabling their use in subsequent steps.

Pipelines does not enforce strict adherence to the schema. This flexibility allows you to work with responses that may include additional properties or return data types that differ from those defined in your sample.

The default expectation, if you don’t select a value, is JSON sample.

Schema sample

This is the actual schema or sample if the expected payload type is not TEXT. Pipelines parses it for you and creates a best-guess structured schema of the expected response. Even if a field is not present in the resulting schema, or has a different type, Pipelines continues to work. 

Validate response payload type

Given the expected payload type is not TEXT, the response payload type is validated by default. If this option is checked or left empty, Pipelines results in an error if the response is not a valid JSON or YAML. Otherwise, it proceeds with the execution even if the response is not a valid object.

This is validating only the format and not the content. Any violations of the schema won’t result in an error, even if this option is selected.

Body

If the request method is one of POST / PUT / PATCH / DELETE, you may provide request body as well. The contents of the body are up to you and the requirements of the API you are connecting. Although it is permitted to have a body attached to a DELETE request, it is not recommended and should be used only if explicitly required by the API.

Error handling

Error handling policy is specified for each step individually and applied only for the execution of that step.

Three options to select the error handling mechanism for the step:

  • Automatic—This is the default built-in error handling.

  • Custom—List only those statuses that need to be excluded from the standard error handling and handled separately. The list contains all standard status codes in the range 4xx and 5xx. If your API uses any of the unassigned status codes in those ranges, you can add them manually - custom errors.
    When returned by the API, the statuses selected in the field are treated as success - the pipeline does not terminate, proceeds to the next step the same way it does with a successful status. You may introduce logic, using the status code and message to decide how to handle the result.

  • None—Choosing this option completely turns off the automatic behavior and allows your pipeline to continue no matter what status code returns your API, every code is regarded as success.

This option changes the behavior based on the returned response from your API. Even if you have chosen None, the pipeline fails if there is another internal error.  

There are certain types of errors which are always be handled by the built-in mechanism e.g. connectivity, timeout, payload size and payload type kind of errors. For these errors, there is no way to have custom error handling inside the pipeline. The pipeline terminates if the response has a status within the 4xx range and retries the request on responses in the 5xx range, as well for 429 status (Too Many Requests).

Make request step limits

  • The Make request step works only with text responses, not with binary data

  • The maximum response content size is 1 MB

  • The maximum request content size is 1 MB

  • Timeouts are 10 seconds for connection and 5 minutes for read

  • Setting the following headers manually is not allowed: Transfer-Encoding, Content-Encoding, or Content-Length

Fetch items step

The Fetch Items step extends the capabilities of the Make Request step. It’s designed for scenarios where an API response contains a list of items (an array), and you need to iterate over each one.

Most configuration options and behaviors are the same as the Make Request step, with a few key differences described in this section.

Response handling

Unlike the Make Request step, the Fetch Items step always parses the response to get the list of items to iterate over. Because of this, there is no “Validate response payload type” option—validation is always performed automatically.

If the Fetch Items step encounters an error, the resulting list in the loop is empty. However, the HTTP status code remains available in the step’s metadata, allowing you to build additional logic to handle the error manually.

All other error handling options available in the Make Request step apply to this step as well.

Fetch items step limits

All limits from the Make Request step apply, except for the response size limit:

  • The full response of the Fetch items step may be up to 32 MB

  • Each individual item in the list must be less than 1 MB

JSON records path

The JSON records path field specifies the location of the array you want to iterate over in the response body. It accepts a path-like syntax similar to XPath.

If the field is left blank, the step assumes the entire response is a list and iterates over all elements.

Example: Basic structure

In this example, the response uses a basic structure:

{
  "data elements": [
    {
      "created_at": "<datetime>",
      "id": "1234",
      "name": "Name",
      "protected": false,
      "username": "UserName"
    }
  ],
  "errors": [
    {
      "detail": "<string>",
      "status": 123,
      "title": "<string>",
      "type": "<string>"
    }
  ]
}

To iterate over:

  • data elements, enter /data elements

  • errors, enter /errors

The leading slash is optional—both data elements and /data elements work.

Example: Nested structure

In this example, the response uses a more complex nested structure:

{
  "data": [
    {
      "created_at": "<datetime>",
      "id": "1234",
      "name": "Name",
      "items": [{...}, {...}, {...}]
    },
    {
      "created_at": "<datetime>",
      "id": "5678",
      "name": "Another name",
      "items": [{...}, {...}, {...}]
    }
  ]
}

To iterate over:

  • The items of the first element, enter /data/0/items

  • The items of the second element, enter /data/1/items

When your response contains nested arrays (for example, an items list inside each element), set the JSON records path to the outer list (/data). Then, use a loop step in your pipeline to process each nested array. You can use Jinja looping to access these inner items, but Jinja loops are limited to 100 elements.

Validation

The JSON records path is validated against the Schema sample field. If the specified path doesn’t exist in the sample schema, your pipeline won’t be saved, and an error message appears.

Output schema

After the Fetch Items step is configured, the Quick Reference widget displays the schema defined by your JSON pointer. This schema represents the output of the step, helping you understand the structure of the data available for subsequent steps in your pipeline

Additional Fields

Encoding

If your request is POST / PUT / PATCH / DELETE, this is the encoding of the request’s body. UTF-8 is set by default.