Data can be stored on our platform in a buffer which is in MongoDB which is a NoQL database. It is an unstructured database to accommodate customizable data.

A Buffer can be compared to physical storage, be it memory or disk, and temporarily hold data while moving between processes. Typically, the data is stored in a buffer as it is retrieved from a source (i.e., a database or another web service) before it is sent to another flow or device.

Buffer data is processed using a Buffer step to retrieve data from a source or use a Buffer step to submit the data to various other destinations or devices.

Illustrated below is a flow where the Buffer Destination step is used. The Record Generator is used to generate records. The Buffer Destination step is used to store the data.

The Record Generator source step is shown in the image below.

Adding a Buffer Destination Step

To add a Buffer Destination step in a flow, follow the below instructions:

1. Click on the highlighted icon as shown in the below image.

2. The below page will appear. Select or search for a Destination step. Click on the Destination tab and select Run File as shown in the image below.

Configuring the Buffer Step

Illustrated in the below image is the configuration that is available in the Buffer Destination step.

Buffer Id

The target Buffer to store the data in. Click the highlighted plus (+) icon to create a new Buffer or click the outlined arrow to select a pre-created Buffer from the dropdown list.

Operation

These operations, also known as data manipulation language (DML) operations, is used to maintain rows of data in a Buffer. The following operations are available:

Insert - Inserts a new record into the Buffer.
Upsert - Inserts a record if it does not already exist, but will update a record if it does exist.
Update - Updates an already existing record with changed data.
Delete - Performs a delete operation on the target Buffer and removes the selected record.

Insert

Select the Insert option from the Operation selection list.

There are no extra options to consider for the insert operation as records will be added to the buffer data one by one.

Upsert

Select the Upsert option from the Operation selection list.

The Upsert option has two options that you may use to update a specifically defined record/document.

Key Field - This field defines the record to be updated. This is a required field for the Buffer Destination to work.
Value Field - The value field that will be updated. Leaving it blank merges or updates the entire record.

Update

Select the Update option from the Operation selection list.

The Update option has four options that you may use to update multiple defined records.

Filter - This option uses the Query builder to identify multiple key fields to update.
Value Field - The value field that will be updated. Leaving it blank merges or updates the entire record.
Update One checkbox - Select this option to update only the first matching record encountered.
Upsert - Select this option to create the record if it does not meet the criteria.

Delete

Select the Delete option from the Operation selection list.

The Delete option has two options that you may use to update multiple defined records.

Filter - This option uses the Query builder to identify multiple key fields to delete.
Delete One - Select this option to delete only the first record encountered.

Purge before checkbox

Select this option to remove any data in the Buffer before the Flow run starts.

Batch Size

Enter the number of records to hold before inserting them into the Buffer. This is set at 1000 by default when the Buffer Destination step is added.

Useful Tutorials

Revisit the Getting Started page for a refresher.