A basic function in working with data is filtering out records you are not interested in and only working on those that you are. In SQL, this is generally achieved through a
Adding a Filter
Filters are implemented as a Step in Synatic.
Select and drag the Filter step from the Steps palette into the flow.
You can add as many filters as you need in a flow:
In the code block, you have access to the processing object, which is accessible through the
input parameter. Normally, in the case of a filter, the property of interest is the
input.record property. This is the JSON representation of the current record being processed.
At this point, you can check the fields of your input record and filter accordingly. Returning
true allows the record to pass the filter, and return
false filters out the record.
You can optionally choose when a filtered record is Discarded or Skipped through the "Discard Record" checkbox:
Selecting "Discard Record" removes the record from memory entirely. Use this option when working with huge in-memory data sets to improve performance. The record will not appear in the flow or logs etc.
Clearing marks the record as Skipped. It is not available anywhere in the processing flow but will appear in logging and error messages (if any).
Advanced Mode Options
Limitations and known issues
Revisit the Getting Started page for a refresher.
Read more about Steps here.