Hi,
I want to build logic that will fetch all records from SAP system using BAPIs and store it into Outsystem Entities using timer. If new record added then only one entry should made else any update happen then respective record should update. I have around million of records in each table.
I believe looping through million of records daily using timer will be tedious task. so looking for simple but effective solution that should take less time for CUD operation.
Thanks in advance for your help.
Hello,
First of all you have to think about syncing data in batches and keep record of last record synced or using timestamp to not repeat your work.
Also you can use bulk insert rather than inset record by record.
You can check this Forge component:
https://www.outsystems.com/forge/component-overview/1117/bulkinsert-o11
First step will be to define a correct synchronisation strategy before looking into the technical implementation.
Will you be doing patching updates --> changing the details of a record that changedWill you do an overwrite of specific records --> overwriting a whole recordWill you overwrite table --> deleting all records and creating new from SAP
This all depends on how the data is used, how often it changes, what the impact is of the change, ...
If you are talking about millions of records you should definitely have a way of splitting the data in different batches.
If you are only going to read data from the SAP systems, I would recommend you to go through the 'Read-only optimized' data sync pattern via the link below. This describes how to efficiently sync bulk data from the database entity to the local entity. You can assume the database entity here as the SAP system and the local storage entity as the OutSystems entities.
https://success.outsystems.com/documentation/11/building_apps/data_management/offline/offline_data_sync_patterns/read_only_data_optimized/
You can understand the logic used here although the entities you have in your scenario is different.