Service timeout issue causing IIS server slow when called using BPM process

Service timeout issue causing IIS server slow when called using BPM process

  


Setup:

 Two web applications:

  1. ServerApp

  2. TestClient1


 ServerApp has two different REST APIs exposed:

  1. Hello:

    Method: GET

    GetHello: Returns simple text response.


  2. TimeOut:

    Method: GET

    GetTimeOut: Service goes to to wait state for 'SleepSeconds' and returns a text message.


 TestClient has one HomeScreen with two main buttons:

  1. Call Hello: Calls 'GetHello' from ServerApp and displays message on screen.


  2. Call Timeout:

     Loops for 'LoopCount' times. So if LoopCount =3, iteration index is like 3,2,1...

     For each iteration, launches 'CallTimeOutProcess' process.

     CallTimeOutProcess has one input parameter- 'SleepSeconds'.

     Process contains one Automatic Activity 'CallTimeOutService' which in turn calls 'GetTimeout' service and logs the response.


Note: 'GetTimeout' client has been set with Timeout of 5 seconds.


Steps:

  Open TestClient1 in browser.

  Set a loopCount value (like 4)

  Click 'Call Timeout'

  'Action End' message is displayed.

  In service center, go to Monitoring -> General logs. 'GetTimeout' responses are recorded.

  In service center, go to Monitoring -> Processes. Open 'CallTimeOutProcess'. Multiple process instances are created as per the loopCount value.

  Next, click on 'Call Hello'. Should work.


  Now, Set a loopCount value more than 4( like 5 ) and hit 'Call Timeout'. (because we have a timeout set for 5 seconds)

  In service center, go to Monitoring -> Error logs. Operation timeout message is logged.

  In service center, go to Monitoring -> General logs. 'GetTimeout' responses are recorded for 4 REST calls.

  In service center, go to Monitoring -> Processes. Open 'CallTimeOutProcess'. Multiple process instances are created as per the loopCount value. Some instances have errors.

  Next, click on 'Call Hello'. Should work.


Issues:

  Set a very high loopCount value (like 100 or 200).

  Multiple process instances are created with most of them in 'Active with Error' status.

  Click on 'Call Hello'. Does not works. Service takes too long to respond.

  Open any other application in environment. Response is slow.

  Check Outsystems service in 'TaskManager' (on-premise environment). Memory usage is too high.


Questions:

  1. Is there a limit to number of process instances spawned at a time for a BPM process?

  2. If yes, Is there a difference in this limit in a Development and Production server. i.e. would the performance be better for a production server?

  3. Is there a way to overcome above situation. I need to send a high amount of request asynchronously to an exposed REST service. The request might timeout but execution in OS should not stop.

The client should not wait for a response thats why I went for a BPM activity. But it seems to fail for a large amount of requests.


Applications attached.


Thanks!



Attaching TesClient1 here

First of all standard HTTP responce timeout is 30 seconds so you should ensure your REST service responds with something within that time so do not put "Sleeps" within a REST endpoint. 

Next if you are going to have long running async connections then get your REST endpoint to launch an "Automatic Activity" from inside a process and return immediatly. "Automatic Activity" Processes then have a 5 minute timeout, I don't think there is any limit to the number of processes however there might be a paralel limit, not certain though. If you expect more than 5 minute then you need to either break your process down into multiple batched "Automatic Activitys" or use a timer. 

For timers the timeout is around 30 minutes by default I think but is changable, you can have up to 3 different timers running at a time but only one of any of any particular timer. 

John Williams wrote:

First of all standard HTTP responce timeout is 30 seconds so you should ensure your REST service responds with something within that time so do not put "Sleeps" within a REST endpoint. 

Next if you are going to have long running async connections then get your REST endpoint to launch an "Automatic Activity" from inside a process and return immediatly. "Automatic Activity" Processes then have a 5 minute timeout, I don't think there is any limit to the number of processes however there might be a paralel limit, not certain though. If you expect more than 5 minute then you need to either break your process down into multiple batched "Automatic Activitys" or use a timer. 

For timers the timeout is around 30 minutes by default I think but is changable, you can have up to 3 different timers running at a time but only one of any of any particular timer. 

Thanks for your reply!

The sample application I shared is a mock up to the actual scenario we are facing. In reality, timeout period is high and there is no sleep. Sleep is added to interpret a scenario where service may take longer time to process and cause a timeout at client.

The REST i am calling is outside OutSystems environment so i cannot change it.


Do you have any idea what could be the parallel limit for number of process instances? Our threshold requirement is more than 150 asynchronous REST calls at a time. If there is any configuration change required to up the number of threads in IIS or increase the RAM to handle such scenario then please let me know. 



Ok sorry I misunderstood I thought you were producing the REST service not consuming it. 

I haven't had to create a huge number of concurrent processes so don't know the limits sorry. There may be some details in the masterclasses on Modeling Business Processes or Best Practices and Timers otherwise you might need to check with OutSystems support. I know timers can be run on a dedicated front end server and the number of timers can be increased, not certain if this is the same for processes, again I'd suggest checking in with Outsystems support on that.

If you get completely stuck you always have the option to create an extension. You can write your own .net extension and through that should be able to have complete control over timeouts, threading etc.