Hello everyone,
I'm facing a challenging scenario and would greatly appreciate your insights and solutions.
Here's the situation: I've implemented a button that triggers a timer to upload Excel files to the database, all while processing them based on customer requirements. The primary issue stems from the size of these Excel files, some of which contain over 800,000 records. As you can imagine, uploading such large files naturally takes time. However, when a file is already being uploaded, and another user attempts to upload a file, the second user's file joins a queue, waiting its turn for processing. This can be confusing for users, as they may wonder why their file isn't processed immediately.
I'd welcome any ideas or suggestions on how to improve the upload time or enhance the user experience. Currently, I'm encountering situations where a single file upload can run for several days (I've implemented a process to capture and process each 10,000 records before calling the timer again).
And Is there any way to detect if the timer has timed out? I'm considering implementing a button that will appear on the user interface and become clickable when the timer times out, allowing users to manually trigger the timer.
Your assistance in resolving this issue is highly valued.
Hi Cuong Su
You can create a mechanism that would work best on large data sets you need to create a logic flow that will update your flags after committing a transaction you can loop 1000-2000 records
Create a variable with dateTime data type and add 10 minutes to it and create records
Keep checking on your logic that time has passed or not the, if passed then Wake the timer
This is image is just an example
Best regards
Hi João Martins Pereira,
Thank you for your guidance. I'll try implementing according to this approach. However, I have one question: if a user uploads a new file while another file is currently processing, will their file still need to wait its turn to be processed?
No, processing the files via a timer is a serial capability. One at a time. By using BPT you could process them in parallel.
Hi Cuong Su,
Have you tried a light BPT process yet? They work similarly as a Timer but can work concurrent. It does require some management, just like timers, but for user initiated async actions that is almost always my go-to solution.
You can read a bit about in this blog: https://www.outsystems.com/blog/posts/light-bpt/
As for you UX requirement. Why not create a block that will show all active excel processing jobs, the current state (percentage?) and where their file are in the queue. This block can then be show next to the user info (upper right) for example so they can always see the latest information. Updating the information can be done via a javascript repeating timer or via a server side events.
Hope this helps,
Vincent
Hi Vincent Koning,
Thank you for your support. Let me go through this. I have a question regarding its capability to handle long-running tasks, such as timers. Can it effectively manage processes that take an extended amount of time to complete?
Additionally, could you please provide me with some sample UX requirements? That would be highly appreciated.
Hi Cuong
If you want to go the path of process they will run asynchronous so the catch is you need to notify your users on the UI when the process is finish
And process can run in pararel, processes are incredibly powerful. Processes can stop and resume execution depending on database events (Launch On, Close On, Conditional Start), they can wait for a set date/time (Wait activity), and can wait for a human interaction (Human Activity).
So it opens your possibilities they are more complex to implement and manage.
Also check this training material https://learn.outsystems.com/training/journeys/async-data-processing-light-processes-584 and https://learn.outsystems.com/training/journeys/async-capabilities-649