Tengo aproximadamente 16.000 registros que estoy recuperando de una API que tarda unos 15 segundos en obtener los datos. ¿Cómo puedo realizar un procesamiento por lotes?
La API utiliza POST y envía una fecha de inicio de hoy, 31 de septiembre, y una fecha de finalización de hoy, 31 de octubre.
El problema es que el temporizador tarda entre 60 y 90 minutos, y me gustaría saber cómo optimizarlo. Básicamente, mi flujo de trabajo consiste en llamar a la API, luego iterar sobre ella con un bucle `for` y una acción para crear o actualizar los registros.
Adjunto una captura de pantalla. Necesito optimizar el proceso y he visto que el procesamiento por lotes es una buena opción, pero no sé cómo implementarlo.
Hi @Felipe Guerra,
I see in your attached screenshot that you’re trying to insert records into your local database, which are retrieved from the API. You can add a small tweak to avoid timeouts within the loop itself.
First, create a local variable in the action let’s call it Timeout. Each time you create a record in the database, use the Commit Transaction server action to prevent data loss or inconsistency in case of a failure. You can flag successfully inserted records with an attribute like IsInserted.
Next, check if the duration has reached the timeout limit. If it has, wake up the timer again to continue processing the remaining records.
For example, suppose you initially set the timeout to 5 minutes. If the timer runs for 6 minutes, it stops there. You then add another 5 minutes to the Timeout value and wake up the timer again, repeating this process until all records are processed.
Condition:
CurrDateTime()>= Timeout
If True:
Set Timeout = AddMinutes(CurrDateTime(), 10)
Wake up the timer again
If False:
Continue processing the loop
Here is a screenshot which can help you understand better
One additional tip: you used a separate action for creating records, which is good practice. However, I noticed you’re using an aggregate inside that action to retrieve the document by ID to determine whether it’s an update or a create. Doing this on every iteration can lead to performance degradation, since you’re retrieving the record each time you enter the action. Imagine having 1,000 records this would put unnecessary load on the backend and could easily become a bottleneck.
A better approach is to move that aggregate outside the action just before the loop in your main logic. Then, use the CreateOrUpdate method instead. It will automatically determine whether the record is new or existing, improving performance and simplifying your logic. You can flag it using the previously mentioned attribute IsInserted, without needing to return each record individually.
Todavía tengo una pregunta: ¿cómo uso CreateOrUpdate? Con la función de agregación, ¿a qué acción corresponde? ¿Y cómo identifico si está en la base de datos, para actualizar o crear?
How do I identify if it is in the database, to update or create?
How do I use CreateOrUpdate with the aggregation function?
Which action does it correspond to?