126
Views
12
Comments
Solved
[BPT] Is there a way to clean up BPT Light processes?

Hi All,

Today I did some load testing with a scenario where a few BPT Light processes making a lot of webservice calls. As expected with a certain load, the bpt light processes started to fail (webservice call resulting in 503's) what is reflected in the logs and environment health:

I know that the BPTL events will retry with an increasing interval, but in this scenario (testing) I just want to clean up the errors and redo my test with a clean sheet.

Following Cláudio Oliveira's post I think I could clear the errors just editing/removing some data directly from the database, but I would only do such a thing as a last resort.


Any advice on this?


Regards,
Dennis






mvp_badge
MVP
Rank: #17
Solution

Hi Dennis,

If a BPTL process fails, it will be retried and retried and so on.

BPT Terminator is of no use here.

You have to make your BPT light process robust to handle this gracefully.

Few options you could implement, in the automatic activity of your BPTL

  • Build in a max retry count functionality. After 3 retries, end the BPTL(no exception), without performing the action.
  • use a 'kill switch' a boolean site property, for normal processing always false. In the situation, you are facing now set the site property to true. In the BPTL process, the first check should be to exit if the site property is true. 

The first option avoids you having BPTL process retrying until god knows when.

The second option can cancel them if you are already in the situation you have. So, for now, build in the kill switch and deploy. Then will kill'em.

Regards,

Daniel

mvp_badge
MVP
Rank: #72

Hi Dennis,

Did you try to use the BPT Utils Forge component to clean up the BPT Light processes?

I know it works for normal BPT processes, but I’m not sure about BPT Light. I’ve never tried it myself.

Let us know if it works.

Regards,

Nordin

Rank: #1573

Nordin Ahdi wrote:

Hi Dennis,

Did you try to use the BPT Utils Forge component to clean up the BPT Light processes?

I know it works for normal BPT processes, but I’m not sure about BPT Light. I’ve never tried it myself.

Let us know if it works.

Regards,

Nordin

Hi Nordin,

I tried using that tool, but the process itself does not need to be terminated since it is not running (it's just waiting for the next run). Besides that you can't use the bulk terminate anyway because only running processes show up there.

In the ossys_BPM_Event table I have these records:

If I change the dequeue_time to 'now' the counter in environment health is set to 0, but the event table still shows these records (even tried to run the BPT_Cleanup timer) so this is not a good solution.




mvp_badge
MVP
Rank: #72

Dennis de Jager wrote:

Hi Nordin,

I tried using that tool, but the process itself does not need to be terminated since it is not running (it's just waiting for the next run). Besides that you can't use the bulk terminate anyway because only running processes show up there.

In the ossys_BPM_Event table I have these records:

If I change the dequeue_time to 'now' the counter in environment health is set to 0, but the event table still shows these records (even tried to run the BPT_Cleanup timer) so this is not a good solution.

Hi Dennis,

Your problem is clear to me now. The title of the post was a bit misleading, which set me on the wrong foot.

I think Daniël’s suggestions are good ways of building some sort of a mechanism that gives your BPT Light process an “out” from the endless retries.

Regards,

Nordin


mvp_badge
MVP
Rank: #17
Solution

Hi Dennis,

If a BPTL process fails, it will be retried and retried and so on.

BPT Terminator is of no use here.

You have to make your BPT light process robust to handle this gracefully.

Few options you could implement, in the automatic activity of your BPTL

  • Build in a max retry count functionality. After 3 retries, end the BPTL(no exception), without performing the action.
  • use a 'kill switch' a boolean site property, for normal processing always false. In the situation, you are facing now set the site property to true. In the BPTL process, the first check should be to exit if the site property is true. 

The first option avoids you having BPTL process retrying until god knows when.

The second option can cancel them if you are already in the situation you have. So, for now, build in the kill switch and deploy. Then will kill'em.

Regards,

Daniel

Rank: #1573

Hi

By implementing the killswitch and change the next run into 'right now'  in the db (otherwise the next run would be tomorrow) I managed to get rid of all errors swiftly.

Good and sound advice to add some retry functionality, too bad I didn't do that right away because I realized the need of this before I even started;)

Thank you!


Rank: #1305

Hey Dennis

 How you updated  'right now' ? I mean in which table and its by process id?

mvp_badge
MVP
Rank: #17

Dennis is referring to the "Right now" button on the timer schedule in Service Center I believe, that is not related to the reply I made that is marked as solution. 

You can change your LBPT process as outlined in the solution and publish. That will end the remaining LBPT processes.

Rank: #1573

Not really, at the time my processes had been retrying for so long that it would take a day before the kill-switch kicked in.
Therefore I changed the Next_Run by connecting to the db directly on the ossys_BPM_Event table, so I could get rid of the processes right away.
However, I would not recommend this (in general please try not to change data directly in the database) unless you are really 100% sure of what you are doing. This article helped me a lot with the latter.

mvp_badge
MVP
Rank: #17

Hello Dennis,

My fault, thanks for the extra information!

Regards,

Daniel

mvp_badge
MVP
Rank: #17

We all have been there, we keep on learning.

Rank: #1305

Hey Daniël ,

Thanks for reply . I have another question related to your solution, For max retry count functionality ,do we need to keep process try count somewhere or its avaible in system table? 

mvp_badge
MVP
Rank: #17

For LBPT there is no max retry count. It keeps retrying with a larger time window every time until it tries it once a day.