Hi, all.Is there any possible ways to catch and handle exception when server action is timeout?
Here's detail:
I want to do some bulk operations in a timer.
The timer consists of several records to manipulate, so like described in best practice, I've set up forEach to process records one by one, and abort the timer and wake it again when timeout is closing.
However, sometimes single loop (= single record manipulation) is expected to be longer than timeout(currently 20min), and here comes my problem.
Each records has "status" column to let users know how their operations going (e.g. it is waiting for execurion, in progress, or encountered some error.).
Initially, status is set as "not yet processed" and in the first part of the loop, the status will be set as "In progress" so that the user would know it is being processed and he just have to wait.
So, if the timer was aborted because of timeout, it has to be set as "error" to let him know the record was too large to be processed in time.
Adjusting the timer's timeout longer would be an option, but that's against my team's rule and OutSystems' best practice, so I'd rather not do that.
Thanks in advance.
Hi Bourdry.Thank you for your answers.
You are right. Most of cases it takes way less than 20 minutes and it's working fine, but since amount of process may vary depends on user's request, it could go beyond 20 minutes if the user sent a large amount of request at once. Of course it is exceptional so I know such kind of records cannot be processed.
About preventing retryng such record, it's already implemented just as you said.
Only thing I'm concerning is the status to be stuck in <In progress> and let user wait forever.
And this is why I'm trying to catch exception on timeout and mark the record as failed. Now it seems that it's not a reasonable way to deal with it, and collecting records with <In Progress> at first mark them as failed seems be a good way.
Thanks again for your ideas.
Hi Shinichiro
If you know that some timers do take a longer time to process, how about processing them in a separate timer of their own?
Hi, Ravi. Thank you for your answer.
Yeah, I tried that approach, and already separated them into loops (and at end of each loop, the timer will check erapsed time to abort the execution and wake itself again if needed). However the process I am concerning is a single loop, which cannot be separated anymore.
The loop itself is very simple. The main table has a key to search data, and whole process is to gather data by that key.
Maybe storing data being processed to some kind of temporary table could be an option, but as I mentioned above, it is single flow of process and very hard to separate into multiple parts. I believe it would be complete mess if I forcibly separate this timer into multiple parts.
Keeping them in a loop, is different to executing them in a separate timer of itself.
You've a main timer that processes a list of items. I'm suggesting to use this timer, but for those long running loop, create a separate timer that only deals with 1 request (not a list), and process it separate there.
Make sure you don't call each of the item in your loop to go for a separate timer, because that is going to delay your process a lot, and may keep your server busy all day. Keep smaller tasks processed inside the loop only, and only for larger tasks call a separate timer so they can have their own lifecycle.
Hi,
Agree with Ravi. But still if single record is taking more than 20 minutes in processing so its a too much time. You should optimize it for performance. You can check in logs where its taking time. As you mentioned loop itself is very simple and you are just searching data based on key. Create proper index in the entity will help in increase the speed of searching.
Apart from that, If still you think it will take time then you also can start a BPT from the timer for each record if don't want to use data further. So its entirely depend on the use case.
regards
Thank you for your answers, Ravi and Vikas!
Ravi, I understand what your idea, and my current logic is technically impremented that way.However, as Vikas said, I'm concerning about the case if a single record takes more than 20 minutes. It's less likely to happen, but I could happen as I cannot ignore the risk.
Vikas,
I'm sorry that I haven't completely understood about BPT (only idea I have about that is it's asynchronous like timer, and it can have input parameters) but are you telling me that using BPT makes timeout less concerning?
My point is like if its taking more than 20 minutes in a processing of a single record then need to check for performance like we have followed all best practices already or not.
Apart from from BPT, my point was like if still it takes more than 20 minutes. Then from Timer where you are executing the loop. Shift all logic of loop into the BPT , means we want to shift all processing from Timer to BPT. By this way timer will iterate only those records and BPT will execute them in actual.
Sorry, but still not able to understand that why a single record processing may take more than 20 minutes. Because 20 minutes is more than sufficient. What kind of processing you are doing which may take more than 20 minutes.
Hi, thank you again for your answer.
Those data to be searched by a key can be hundreds thousands of records in certain condition and those records need to be manipulated and organised into single file and that is because I assume it can take more than 20 minutes.
Number of records (that comes out of one key) can vary from 1000 to several hundreds thousands.
I've already tried optimising those queries and am still working on it, so that kind of approach is still an option.
Yet I need other approach just in case optimization won't save enough time.
Therefore allow me go back to my initial question as we are drifting off the point.
In case of timeout, are there any ways to catch that timeout and execute any process (like mark a record as failed)?
Hi @Shinichiro Endo ,
first of all, completely agree with Ravi and Vikas. What on earth are you doing that takes 20 minutes ?? And the excuse of "well it's complex and really can't be split into smaller or simpler steps" sounds like a symptom of a badly engineered process in the first place, so of course re-engineering is painful and expensive, but not impossible.
But let's say you just want to do minimal work to at least be able to process the ones that are not too large. You say it can't be split into smaller parts AND some cases still take longer than 20 minutes AND you don't want to change the 20 minutes, so in that case, you must accept that some cases will NEVER be processed by your timer, right ? IF that is the case, what you are looking for is to change your timer in such a way, that it just skips those cases, never retries them, and just goes on with the next one?
All you have to do, is change status at beginning to <in progress>, and immediately do a commit transaction, than also at the end of succesfull processing, change status to <completed> and again do a commit transaction.
In your query of which to work on, just pick up the ones with status <new>.
At the beginning of each run of your timer, you could first run a query on those that are in status <In Progress>, update them to status <Failed> and maybe mail somebody with the sad news that your complex and unsplittable software wasn't able to deal with them.
Dorine
Hi all, coming very late to the party here, but is there any answer to the initial question about handling Timer and Process timeouts?
I am looking more for a way to handle Process timeouts as sometimes a user has a lot of data and 5 mins just isn't enough. I am going to swith to a timer because of this, but I am still curious to know if there is a mechanism OutSystems provides to catch these Timer/Process timeouts? It seems like something that should be possible...
Hi, Nicholas.
From what I've learned so far, unfortunately the answer is no.
I'm not sure if I'm getting it right, but Process Timeouts for Timer is caused by IIS killing a processing thread. In other words, Process Timeout means entire OutSystems process (related to the Timer) is being shut down by outside (and higher level) of OutSystems.
Therefore, OutSystems can't catch process timeout.
By the way, as far as I remember, sometimes (perhaps not always) Exception Handler catches the process timeouts. The message is always "the thread was being aborted" or something like that,(please make sure to check it on your own logs) so if you use IF widget inside Exception Handler and make logic to handle process timeout, maybe you can make it work.
However, I haven't tried the workaround above yet, so I'm not sure if it even works.
So I recommend you to split your process as much as it surely ends within timeout period (default 20min), and make it self re-starting Timer.
Good luck!
Hi Shinichiro thanks so much for getting back to me. I'm going to try both options.