Hi,
For a project I am working on we gather data via multiple API's. API Dataset A contains all information on API call (we already have this as a list) but it also contains invalid records. API Dataset B contains only valid records but data can only be pulled by using an Id from API A.
We need to create a sublist of only valid information based on API A. API B has too many levels inside of it to directly query this (we don't know where it stops since that varies).
Our problem is as follows:
If we use all data from API A and loop this to check if it exists in API B, our timer will eventually stop (time out) since it returns too many errors.
What we want to achieve: If a record exists in both A & B, it should be saved. If it does not exist in B, we want to ignore and continue to the next record in the (local) list from API A.
What would be the best way to achieve this? We tried using the OnAfterResponse already, but any input is welcome!
Max
One possible way that comes to mind is to save all the API A records into a temporary table and launch a BPT on creating each record. This BPT will check on API B if it exists and will create the data in your real table. Clear the data from temporary table on the end of the BPT.
Regards
OnAfterResponse should only be used to modify data that's received or change the HTTP status code, but never for actual data processing.
That said, if your timer runs into a time-out, you should, if possible, use partial processing, and let the timer restart itself. It depends on the capabilities of API A and the volatility of the data whether you'd want to let the API handle the partial data retrieval, or whether you'd want to store the data first as Bruno Marques suggests.
Thanks for your answers. The information coming from API A is already saved in an entity of its own, but needs to be compared to the records in API B. So basically I have a dataset (with to much inactive data) and an API to verify the data.
I know it is not a very logical process but currently the best we can do.
Isn't there anyway to make sure the timer does not stop after x amount of error while trying to retrieve data from the API? Whenever the record does not exist in the data we are collecting in API B, OutSystems receives an error and after x amount of errors, the timer stops.
Well, first, you can increase the default time-out value of 20 minutes (in Service Center). If on average you need an hour, you can set it to two hours and be done. Secondly, you can either keep track of how long the timer has been running, and check, after each iteration, whether you're running out of time, and if so store some state information (if needed, so you don't start at the beginnen in the next run), wake the timer and end. Or, if on average an iteration takes X time, limit the number of iterations to a value you know doesn't run into a time-out.
For a generic overview of timers, see this OutSystems document. For a different take on timers, see my Medium article here.