Hi,

I have a requirement to create a screen that allows user to upload ~80,000 records from excel with around 120 columns. This would all be uploaded in a single entity. I am planning to use the bulk insert forge component to achieve this. Have a couple of queries related to the same.

1. In the Bulk insert server action, does the table name attribute refers to the Entity name or the physical table name?

2. Should I use timers at any stage of the upload, possibly to ensure that the function executes without any issues? I have read some articles about it but I am new to timers so haven't worked on it.

Thanks!

Hello Anuj,

I have one question regarding your problem, 120 columns for one single entity sounds too much. Is there a specific reason for that?

Regarding your timers question, yes you should use them to process batch jobs, ensuring no time outs and that all your data is processed. (https://www.outsystems.com/evaluation-guide/can-outsystems-handle-heavy-batch-processing/)

Ruben Bonito wrote:

Hello Anuj,

I have one question regarding your problem, 120 columns for one single entity sounds too much. Is there a specific reason for that?

Regarding your timers question, yes you should use them to process batch jobs, ensuring no time outs and that all your data is processed. (https://www.outsystems.com/evaluation-guide/can-outsystems-handle-heavy-batch-processing/)

Thanks, Ruben. Appreciate your prompt response.

Yes, it is a business requirement. Just rechecked, there are 91 columns in total.

1. Will the proposed timer in question execute a server action in which I should configure the bulk insert server action?

2. The default timeout period is 20 mins. If the upload doesn't completes in 20 minutes, should I write a logic to execute from the point of the last inserted record. Or should I increase the timeout period.


Anuj Malani wrote:

Ruben Bonito wrote:

Hello Anuj,

I have one question regarding your problem, 120 columns for one single entity sounds too much. Is there a specific reason for that?

Regarding your timers question, yes you should use them to process batch jobs, ensuring no time outs and that all your data is processed. (https://www.outsystems.com/evaluation-guide/can-outsystems-handle-heavy-batch-processing/)

Thanks, Ruben. Appreciate your prompt response.

Yes, it is a business requirement. Just rechecked, there are 91 columns in total.

1. Will the proposed timer in question execute a server action in which I should configure the bulk insert server action?

2. The default timeout period is 20 mins. If the upload doesn't completes in 20 minutes, should I write a logic to execute from the point of the last inserted record. Or should I increase the timeout period.



u can use set timeout function with value -1

Leaving a timer running without control is not good for performance, Keerthi. It blocks the processing of other timers.
It should end and restart, leaving space for smaller timers to run in the meanwhile.


Anuj, split the task into two parts.

First, upload the file and save each row into a similar Entity where all cells are text.

Second, take n rows from the text-Entity and process each row to the final Entity. Here you can do your data validation peacefully because time is not an issue.
If you still have time, repeat. Else, recall the timer and start from where you left.

I also would suggest to get some Entity Table to handle/report the errors of the batch. You also need to implement some logic to avoid to "stop" the timer in a line with error and keep retrying it...

Anuj,

Nunos has a good idea.  I recommend you go further.

I recommend you upload the whole file into an entity with a binary data attribute.  Best practice is to use 2 entities for this, one for metadata (filename, file size, etc) and a second entity linked in a 1 to 1 relationship which holds only the binary data.  If you open the file as part of the upload operation, it increases the chances of it timing out.

Then you might create a background task (timer) to convert the file into entities with text fields.  Depending on your requirements you may want to store intermediate values like Nunos suggested.  Otherwise each time you process a batch of records you will need to open the whole file.

When creating a timer that may take a long time, you need check if it has been running for over a time period, say 10 minutes for example.  Then persist the state of the timer and trigger it to start again.  This will allow other timers to run, and also prevent you timer from timing out at 20 minutes (the default setting).

I hope this helps!

Kind regards,

Stuart

Never let a timer run without a timeout. It's always better to do a records batch process, for example 5000 records each timer execution or check elapsed time if close to timeout. Then wake the timer again.

Light BPT can be a good solution for this scenario, since you might need to process 80k records.

https://www.outsystems.com/forums/discussion/41063/light-processes-under-the-hood/