Hi Community,I have a requirement where I need to upload an Excel with large number of data using Upload widget.I want to upload 1.3 million records from excel.
So when uploading it is showing "Connection Timeout Error" after changing the Server timeout the it is showing "Request failed with status 413".
I have went through various posts of forum but did't got the solution.I have even attached the .oml file for reference.
Regards,Sanji San.
Hello @Sanji San ,I did replicated your scenario and I did created a sample, though from performance perspective it take some time to load the data, however we can skip that by not assigning binary file anywhere.
So for now I did created a server action with the flow that will bootstrap the bulk data to your table with the use of timer,
I have checked this with 10 lakhs records a kind of same scenario the old difference is we were doing data sync and getting data from External DB,
However you can take this as a reference and create a logic that can help you with the data upload.
Check the OML attached
ThanksTousif Khan
Hello
You can create a timer and run the asynchronous job there,
You can create a create a entity which will hold your file until all your records get created and then you can remove the file, it will temporary hold your file.
I have used the same approach to do bulk insert into Db table,
You can refer to below links, hope it might help -
https://itnext.io/how-to-manage-heavy-timers-in-outsystems-6c954789783e#:~:text=It%20is%20a%20very%20common,processes%20is%20their%20timeout%20management.
https://www.outsystems.com/forums/discussion/81684/best-solution-of-timer/
Hi Tousif,
Thank you for replying but i have already used Timer you can check the .oml file that I have shared.
As I am new to outsystems .It would be great if you edit the .oml file that I have shared. For my understanding.
Hi @Tousif Khan,
Thank you for your reply.
I will check the .oml file and let you know.
Hello,
For the type of error it seems to be a limitation for the size of the files you can upload to the server. If your enviroment is on premise, you can check this limitations. If you are in cloud maybe you can open a ticket to Outsystems, and they can help you.
Other possible work around is to split this large file in serveral files with less amount of data, and try to upload one by one.
Best regards
Hello Sanji,
Like you said you are already using timers and still facing time-out issues. In such a scenario, we have another alternative where we can use Light BPT for uploading bulk records.
I went through your OML. You are uploading everything in FileBinary and then after processing you are trying to insert data into SalesRecordExcel. What you can see here is after upload in FileBinary is done then for processing call Light BPt where you can insert into the Second entity one row at a time.
I tried editing your OML but I don't have your Excel. I can also test the logic at my end if you can share.
In case you get the explanation above I am sharing my reference OML where I am trying to do the same thing on different entities. this will be helpful for you.
I hope you find this helpful.
Regards,
Neha
Hi Neha,Thank you for replying I tried by your way but still showing same error "Connection Timeout".
I have attached the excel file that I am uploading.
I cannot upload the exact excel file because of the upload size limit the size of the excel that I am uploading is 103 MB.
Hi Sanji San,
I have checked your oml. If you upoad from your web screen , it is a sever call then you will get the error as mentioned.
Since this is a temporary file, you can add it to resource folder then use it for your timer and remove it after finished.
I have edit your oml so that it will only run in server only. Please check Timer_ImportTableExcel_V2 (assuming that your import logic is correct).
What do you need to do next is trigger timer on the server :
After Publishing the module, you go to Service Center -> Factory -> Your Appication -> Your Module (that store the timer) -> Timer Tab -> Run Timer_ImportTableExcel_V2You can check log at Timer Monitoring. Cheers,Nam
Hi Nam,
I don't want to upload a file temporary.
The user will be uploading the file with that size at that time only it is throwing the error.
I have shared the excel file above that type of file user will be uploading.
Hello Sanji San,So there is a concept of wake the timer.When the Wake<Timer Name> action is executed, the NextRun property of that Timer is updated to the present time.
So you can even use this.
ThanksNeha
Hi Neha,
Thank you for your reply,
But I am doing that but it takes time to load data. The data loads as the timer runs.
The data is too much it will take time to execute you have to wait until data is uploaded in the database and also if you make it faster you need to refactor the code accordingly.
Hope this helps
Thanks
hi,
You can process the data once the uploaded file has been saved to the database.
For uploading a larger file you can refer to this discussion.
Once the file is uploaded you can save the data using bulk insert by waking the timer. Bulk insert works faster.
I have modified your OML.
Wasimkhan S