Scheduler Service: Error executing request

Scheduler Service: Error executing request

  

We are encountering issues with regards to running multiple processes in our staging environment.

This is the scenario:
1. We have a process that calculates data. This process is running multiple advance queries and updates entities.
2. This process is triggered when we insert a row in an entity.
3. We have tested this working in our development and test environments and didn;t encounter any major issues. Note that we ran at least 2000 instances of this process at the same time.

In our UAT environment, we've triggered multiple instances of this process and experiencing a weird error we didn't encounter before. This is the error we are getting for every instance of this process:

10-13-2016 15:16:36Scheduler Service: Error executing request http://127.0.0.1/Report for Activity Recalculate. Request duration = 335 secs.

Is this somehow related to this issue : https://www.outsystems.com/forums/discussion/7539/scheduler-service-error-executing-request/

If so, how can we resolve this issue? Thank you very much in advance and hoping to hear from you soon.


Full Error Log Below:


Message:Scheduler Service: Error executing request http://127.0.0.1/Report/ for Activity Recalculate. Request duration = 332 secs. 
Environment InformationeSpaceVer: 0 (Id=0, PubId=0, CompiledWith=9.1.301.0)
RequestUrl:  (Method: )
AppDomain: Scheduler.exe
Path: C:\...\PS\
Locale:
DateFormat: yyyy-M-d
PID: 3200 ('Scheduler', Started='5/3/2016 11:45:19 AM', Priv=58Mb, Virt=654Mb)
TID: 24
Thread Name: Activity Processor #6
.NET\: 4.0.30319.34209
Stack:System.InvalidOperationException: Client found response content type of 'text/html; charset=utf-8', but expected 'text/xml'.
The request failed with the error message:
--
<!DOCTYPE html>
<html>
    <head>
        <title>Request timed out.</title>
        <meta name="viewport" content="width=device-width" />
        <style>
         body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;}
         p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
         b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
         H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
         H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
         pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
         .marker {font-weight: bold; color: black;text-decoration: none;}
         .version {color: gray;}
         .error {margin-bottom: 10px;}
         .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
         @media screen and (max-width: 639px) {
          pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
         }
         @media screen and (max-width: 479px) {
          pre { width: 280px; }
         }
        </style>
    </head>

    <body bgcolor="white">

            <span><H1>Server Error in '/Report' Application.<hr width=100% size=1 color=silver></H1>

            <h2> <i>Request timed out.</i> </h2></span>

            <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">

            <b> Description: </b>An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

            <br><br>

            <b> Exception Details: </b>System.Web.HttpException: Request timed out.<br><br>

            <b>Source Error:</b> <br><br>

            <table width=100% bgcolor="#ffffcc">
               <tr>
                  <td>
                      <code>

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.</code>

                  </td>
               </tr>
            </table>

            <br>

            <b>Stack Trace:</b> <br><br>

            <table width=100% bgcolor="#ffffcc">
               <tr>
                  <td>
                      <code><pre>

[HttpException (0x80004005): Request timed out.]
</pre></code>

                  </td>
               </tr>
            </table>

            <br>

            <hr width=100% size=1 color=silver>

            <b>Version Information:</b> Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.34274

            </font>

    </body>
</html>
<!--
[HttpException]: Request timed out.
-->
--.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at OutSystems.HubEdition.RuntimePlatform.Processes.ActivityHandler.ExecuteActivityActionv2(String ssKey, Int32 activityId, Int32 processId, Int32 tenantId, ActivityStatus forInitialStatus, DateTime previousNextRun)
   at #oof.#nof.Execute()
   at OutSystems.HubEdition.Scheduler.SchedulerProducerConsumer`1.<>c__DisplayClassa.<Execute>b__9()

Well, the request timed out. This can have any number of reasons (e.g. database lock), so you'll have to take a closer look. You can look in Service Center to check which instance causes the timeout.