The concept of an HTTP (Web) service is an open standard (JSON, XML, SOAP, HTTP, etc.) based web application that communicates with other web applications for the purpose of exchanging data. In practice, using a protocol such as HTTP to transfer data in machine-readable file formats such as XML and JSON is a standardized way of providing interoperability between disparate applications.
In Synatic, the HTTP Service Step provides this function and generally allows the flow to send requests to read or get data from a web-based API.
Illustrated below is a flow where the HTTP Service step is used as a Source step. A JSON step converts the data to JSON format. The Buffer destination step stores the data.
The paging option is used to set up how the data is rendered and controlled as retrieved from a web request. The Paging Option is shown in the image below. Click on the Enable checkbox as shown to access the paging options.
Delay – Time to wait (in milliseconds) before fetching the next page.
Page Size – Set the number of lines to retrieve for each page. In the example above, there will be 5 for each page.
Page Start – Set the number of lines to retrieve for each page.
Paging Options
The Paging section has some options that can be used when paging is enabled. Click on an option to open its configuration. The following options are available in the paging section.
Check Length
Set the maximum number of pages received to consider as complete. It is recommended to set a higher number just in case there are a lot of records that can exceed the estimated number of pages.
Check HTTP Status Code
After each page is retrieved, check for a specific HTTP Status Code to indicate the completion of the request. Paging will stop when the set status code is identified.
For more information on HTTP Status Codes, please click the link below:
The image below shows the output from the Check HTTP Status Code Paging option.
Check Field
Enter a field name that should exist on the return value.
This option is only applicable to the JSON format.
Field Path – The path to the field.
Additional Field – Additional fields to load.
Path separator – The path separator to use. Click on one of the options to select.
The image below shows the output from the Check Field Paging option.
Check Empty Array
Check if a field exists on the return value. This is only applicable to JSON formats.
Empty JSON Array – Check if the response is an empty JSON array when selected.
Field URL
Check specified field on the response to get the next URL from. In this example, each page is set to have 100 records in the Page Size, meaning that the next URL will start paging at 101.
Path separator – The path separator to use. Click on one of the options to select.
The image below shows the results from the flow where the Next URL starts paging at 101.
