Integration Concepts 2 – Synchronization with REST integrations

Synchronizing records is often required when integrating two or more systems that combine to complete a business process. For the process to work smoothly the synchronization must be timely, reliable, and transactional. To ensure data integrity it is vital that any network or system errors do not result in lost or duplicated records.

This article outlines how to use features in Flowfinity to implement synchronization with external systems while ensuring reliable data consistency.

Follow this approach when doing hands-on coding with REST APIs in the programming language and API of your choice, or when using visual programming systems such as Power Automate (formerly Microsoft Flow). This technique does not apply when the synchronization logic is built into the product (e.g., Zapier).

To ensure reliable synchronization results follow these 3 steps:

1. Flowfinity apps require a specific view to be created for use when synchronizing with external apps to return all new and updated records since the last synch.
The view must use the system field 'lastSubmissionId' which contains a system-wide unique value which increases every time a record update is made so that records are returned in the correct order.
To set your synchronization view up correctly:

1. Create a new view and select the 'Layout' tab.
2. Include 'lastSubmissionID' (displayed as 'Last submission ID') as one of the columns in the view or set the view to return record objects in the view integration settings.
   
3. Click the 'Sorting' tab and chose 'Custom sorting.' Specify the sort order to be by 'Last submission ID' in ascending order.
   
4. Include a parameter into the view (e.g., 'PreviousLastSubmissionId'). Navigate to the 'Parameters' tab and click 'Add parameter.' Label it 'Previous Last Submission ID' which will generate the name 'PreviousLastSubmissionId.' Select the 'Numeric' field type. Click 'OK.'
   
5. Now access the 'Filter' tab to configure a filter that will only return records that are newer than the last processed last submission ID, as shown below.
   1. Click 'Add condition.' Select: 'Last submission ID ', 'Is Greater Than', 'Previous Last Submission ID .'
   2. Click 'Add condition.' Select: 'Previous Last Submission ID ', 'Does Not Contain Data.'
   3. Select both of the conditions and click 'Combine with OR.'
   
   Note: Steps 'b and c' account for the possibility of receiving an empty value for the parameter, and will then return all related records.

2. The external system or code must be configured to query the synchronization view described in the previous step periodically (e.g. every 5 minutes).

When sending a request through REST the code must set the parameter value to the value of the 'lastSubmissionId' from the last record of the previous request made (or no value if this is the first request).

The value of the last processed 'lastSubmissionId' should be saved in persistent storage such as a database column, or in the file system (for Power Automate this could be an Azure file).

3. When processing the synch results the external system should read the results record-by-record and update the value of the last processed 'lastSubmissionId' only when the update has been successfully applied to the second system.

By creating and configuring a dedicated synchronization view you will ensure that all new and updated records are synchronized in a reliable manner.

Allowing the synchronization of 'Deleted' records

If a record is physically deleted with the 'Delete' operation these records will not be visible when using the synchronization view approach, so we recommend the following work around to archive records rather than delete them if required.

To enable synchronization of deleted records, disable the 'Delete' operation for users of the app and configure a separate 'Deleted' (or a similar appropriate label) state for records. Create a transition operation entitled 'Delete Record' (or a similar appropriate label).

By configuring all views to not display records in the new 'Deleted' state you can still provide your users with the functionality of removing unwanted records from their views, however the hidden records will still be maintained in your database for synchronization purposes.

Now, when a non-admin user chooses to 'Delete Record' it will not actually be purged from the database, it will merely be moved in to the new 'Deleted' state and hidden from their View.

Since the record is not physically removed, the record along with its state name will be returned in the dedicated view created for synchronization. An external system may be set to remove the record from its' database based on the state name ('Deleted'). To later physically clear records in a deleted state from Flowfinity, the app designer may utilize different methods including escalations, tasks or import automation scripts.

If you have any questions about REST API synchronization please contact us.