Timeout errors in Timers

Timeout errors in Timers

  
Hi All,
 
I have a Timer which runs every  5mins.Under the action of this timer I have a logic which fetches the data from underlying database view.After fetching the data from view, I am trying to insert the same Record into outsystem table if it is new record or updating if it is existing one.Here I am looping each record from view and inserting  or updating into outsystem table.Since the view has millions of records looping each record using foreach taking long time and there by timer is timing out without completing.I would like to know is there any better or alternate way to insert or update the outsystem table instead of looping each record.I mean can we do a bulk insert/updates into outsystem table?

Thanks,
Prashanth 
 
I don't know anything about your application but in my opinion anything that is attempting to create/update millions of records every five minutes will perform poorly, if it doesn't fail completely, even if you resolve this time out issue.

I would recommend taking a step back and try and determine why you need to do this and look at alternatives that don't require moving all this data around.  My first question, again not knowing your application, is that if you have access to the data why can't you just use Outsystems SQL statements (simple or advanced as needed) to view it?  If you take this approach, what is it in your application that you need to do that you can't?

Hope this helps.
basically what curt says. if you are moving/updating millions of records, you should think of a better strategy first.
You have a rather small execution-window to do everything.


Anyhow,

1. with advanced queries you can do more, so that might be a solution.
2. you really should chop the time into partial timers. so, for example. do 1000 records, then call the timer again. this way the execution will be kept alive.



Statler & Waldorf and the amazing flappy tweedles! wrote:
basically what curt says. if you are moving/updating millions of records, you should think of a better strategy first.
You have a rather small execution-window to do everything.


Anyhow,

1. with advanced queries you can do more, so that might be a solution.
2. you really should chop the time into partial timers. so, for example. do 1000 records, then call the timer again. this way the execution will be kept alive.


 
 Hi,

I am trying to do the same thing as you said i.e processing 1000 records or more per run.Now the timer is not throwing any errors related to timeout issues.But I am seeing an different error today as shown below.This error is not occuring for every run but rarely I am seeing this new error.Is it an issue related to scheduler service installed in server or differnt one?.

 Scheduler Service: Error executing request http://127.0.0.1/Application2/_TimerHandler.asmx for Timer TestSummaryTable. Request duration = 108 secs.  [retry 3 of 3 scheduled]

The underlying connection was closed: An unexpected error occurred on a receive.
   at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
   at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at OutSystems.HubEdition.Scheduler.TimerHandler.ExecuteTimer(String ssKey, Int32 timeout, Int32 tenantId)
   at OutSystems.HubEdition.Scheduler.TimerJob.#alc()
   at #xMc.#WOc.#alc(#3kc job)

Thanks,
Prashanth 
Curt Raddatz wrote:
I don't know anything about your application but in my opinion anything that is attempting to create/update millions of records every five minutes will perform poorly, if it doesn't fail completely, even if you resolve this time out issue.

I would recommend taking a step back and try and determine why you need to do this and look at alternatives that don't require moving all this data around.  My first question, again not knowing your application, is that if you have access to the data why can't you just use Outsystems SQL statements (simple or advanced as needed) to view it?  If you take this approach, what is it in your application that you need to do that you can't?

Hope this helps.
 Hi Curt,

Thanks for response.I am trying to split the records and processing them as statler said.Now I am not seeing any time out issues but a differnt error as mentioned in Statler's reply which occurs rarely.

Thanks,
Prashanth Reddy
 
Unfortunately I don't know enough to know what this error is trying to tell you but the information displayed is still cause for concern.  The message indicates it took 108 seconds to process the 1000 records (actually less than that since it failed) for a rate of 10 records/per second.  At that rate it would take more than a day to process the millions of records that you are trying to move.  Certainly part of the reason is because you are using web services.  There is some overhead converting the record to XML and then back to a record when using web services.  Why are you using web services instead of accessing the data using SQL queries?  Is the data in a different database?
Curt, this error is related to the execution of the timer itself - not related to the migration code developed by Prashanth.

Prashanth (/Curt), 

the OutSystems Scheduler Service calls a web service that in turn executes the developer's code. Common "web executions" will be killed by IIS after 100 or 110 seconds (unless "global default" value is changed or an explicit different timeout is defined on a per-execution basis). 

The Web Service changes it's timeout (to a value based on the Timer's "Timeout in Minutes" defined in Service Studio) in order to allow executions that span several minutes. I'd say that for some reason this timeout is not being followed. Anti-virus? "Messed up" server?

Also check http://www.outsystems.com/forums/discussion/8733/timers-not-running-right/