We need help to understand the issue of timer which started running on multiple frontend servers at same time.
There are three front end servers configured in our environment and as scheduled timer started, and we observed below points from service center logs
1. Timer started on front end server(FM03) twice at same time 10/15/2023 12:15:08 AM, first instance executed for 24 minutes and throwed the timeout error and second instance completed in 24 minutes without any error.
2. But through logs we observed that same timer started again after completion of it's execution in point one i.e after 24 minutes on same front end server(FM03) as mentioned in point one and also on another front end server(FM02).
3. And after it's completion as mentioned in second point it started again on new front end server(FM01)
4. Due to multiple and parallel execution of same timer, we faced the issue of updating same DB records multiple times which leads to incorrect values in DB as we were applying some business rules on that data before updating in DB
We need your help to understand why same timer was started on multiple frontend servers at same time in case of it’s first failure.
As per documentation and on different forums we know that same timer at a time can run on only one front end server..
Hi Archana,
It is correct to state that the timer, with the same name, can only have one instance actively running per front end server. With 3 Front end servers, you can have a timer with the same name running actively running 3 instances, one on each front end server.
So what could have happened, you can have:
If you gave more information about your timer and how it can be started, it would be easier to answer why your case happened.
Furthermore, I would like to advise that your timer logic should be robust to avoid, unwanted (double) execution of logic. You can do this, for example with a status attribute on the data you process, so it only will process once.
Regards,
Daniel
Thank you Daniel for valuable inputs!!
Please find below my response
As mentioned in point 1, we are still trying to find why it started twice on same front end server exactly at same time. This might had caused the issue of deadlock on DB side for DB records updates and end up in timeout error after 24 minutes.
But yes as a best practice and as you suggested we are planning to capture the success status for processed records and try to avoid the double execution.
"Logic that wakes a timer multiple times. : yes we do have logic to wake it up after 15 minutes and one more business condition so it started again on all three front end servers as mentioned in point 2 and 3 of original question"
Correction to your statement in bold. When it starts again, it will start on 1of the 3 FE's not on all 3.
"As mentioned in point 1, we are still trying to find why it started twice on same front end server exactly at same time. This might had caused the issue of deadlock on DB side for DB records updates and end up in timeout error after 24 minutes."
Might be, might be coincidence too. You could also ask this to OutSystems support, maybe they have more data to draw a conclusion.
Thank you!!
We are already in process of connecting with OutSystems support team.
Will update this post if we get any clarification from OutSystems team.
Hi Archana, May I know if the Timer module is Single Tenant or Multi Tenant? If Single tenant, we may need OutSystems support to support.
Hi Nghia,
Timer module is single tenant.
Yes we are already in process of connecting with support team. Thank You!!