FAQ - W3WP Crash

  

Symptom

 

When navigating in your OutSystems application, suddenly your browsers begin an endless request, until it eventually times out on the browser side (connection closed).

 

When you access your OutSystems server and check the Event log, in the Application group, you see an error similar to the following:

 

Source: .NET Runtime 2.0 Error

Type: Error

Description:

Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d6968e, faulting module kernel32.dll, version 5.2.3790.4480, stamp 49c51f0a, debug? 0, fault address 0x0000bef7.

 

 

If you look in the System group, you might see a warning similar to the following:

 

Source: W3SVC

Type: Warning

Description:

A process serving application pool 'OutSystemsApplications' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '1234'. The data field contains the error number

 

 

Cause

 

This events mean that you experienced a worker process crash. This means that a severe error in one of your applications has caused the worker process (w3wp) to crash. This means that somewhere in your application(s) there is logic that is crashing.

 

When using OutSystems to develop your applications, the most common causes for this are: infinite loops somewhere in your logic and misbehaved integration (extensions).

 

 

Resolution

 

To solve this type of errors, you first need to identify where it is occurring. Since this problem can occur anywhere in your logic, the process should follow this line of though:

  • In a top-bottom approach, identify where (eSpace) it is ocurring;
  • Determine usage patterns of that eSpace that trigger the behavior;
  • Inspect the logic.

To start with this, a good approach is to search for a candidate eSpace or set of eSpaces. If you can trivially replicate the crash (e.g. navigating in your application, after a given sequence of clicks, always causes the problem) see the logic in the eSpace of this last click before the crash, and go back to the beginning.

 

If the crash is more random, you will need to first detect where it is occurring. Some guidelines for this:

  • Try stopping the Scheduler Service and leave it like that for some hours. If the crash stops occurring, that means that the problem is with a timer in one of your applications, and now you have narrowed it;
  • Create a new application pool in IIS and put one (or a set of) eSpaces there. When the next crash occurs, check the System group in Event Log for the warning mentioned in Causes: if you now see the name of your newly created application pool instead of OutSystemsApplications, it means that you have found the guilty eSpace, and can start digging from there.

As a last resort, you may want to use DebugDiag and attach it to the w3wp.exe and get a stack of the crash, but most of the times the above procedure will allow you to identify the problem. We frequently get reports for this sort of problems at Support, and I am yet to see one that is not caused by a simple infinite loop in the eSpace logic.

 

 

Further considerations

 

The above troubleshoot is also useful for identifying infinite loops in your logic that, instead of a worker process crash, cause 100% cpu usage. In that case, when isolating the eSpaces in a separate application pool, simply forcefully kill the w3wp.exe that is using 100% CPU and check the Event Log to see which application pool it was serving.

 

Feel free to post your remarks or corrections to what is said above.

Hey All

While the DebugDiag 1.1 works rather well for Windows 2003 server 32-bit and 64 bit worker processes, there's yet to be released a version that works with Windows 2008 servers.

Meanwhile, the solution is to use built-in tools of the operating system, depending on the type of problem:
  • for worker process hangs, just generate hang dumps through Task Manager tool
  • for worker process crashes you can use the Windows Error Reporting (WER), which is enabled by default, to collect crash dumps
Information about how to perform these tasks, and which other tools are available is present on the IIS blog article http://blogs.iis.net/webtopics/archive/2009/11/25/how-to-collect-a-crash-dump-of-an-iis-worker-process-on-iis-7-0-and-above.aspx

Hope this is helpful.

Cheers
Greetings,

For the record, these type of crashes mentioned by Acácio on the top post, in Windows Server 2008 R2 are logged similar to:

Source: Application Error

Type: Error

Description:
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17514, time stamp: 0x4ce7c78c
Exception code: 0xe053534f
Fault offset: 0x000000000000a49d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

Cheers,
When we need to get to the part of actually dumping the Worker Process memory, this post by Tess Fernandez provides a pretty good overview of how to obtain the crash and do an initial analysis.

This is very useful if after isolating the problem (to confirm the actual stack of the error) or if your attempts to isolate the error are inconclusive.
The information you will obtain is very similar to what you get in a stack trace in the Error log. Particularly, the bottom lines should identify the eSpace where the request is running - therefore telling you which eSpace to move to the separate pool.

Cheers,
Ricardo
Hi,

I finished installing the latest version of the platform server on a new server and after setting up the OsISAPIFilterx64 I have this error:

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: filter.dll, version: 8.5.9600.16384, time stamp: 0x5215e17d
Exception code: 0xc0000005
Fault offset: 0x000000000000c579
Faulting process id: 0xfa8
Faulting application start time: 0x01cfb8e690730772
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\System32\inetsrv\filter.dll
Report Id: 449c28ec-24db-11e4-80bd-005056032dc5
Faulting package full name: 
Faulting package-relative application ID: 

This error only appears when I click on the page "SEO URLs" in Service Center.
Any suggestions to this issue?
 
Thanks,
Pedro Coelho
Hi Pedro

It seems that you'r using IIS 8.5 which is available on Windows 2012 R2 and Windows 8.1, and neither of them are supported by the OutSystems Platform yet.

That's probably why the SEO filter is failling on that worker process.

 Cheers
Pedro Coelho wrote:
Hi,

I finished installing the latest version of the platform server on a new server and after setting up the OsISAPIFilterx64 I have this error:

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: filter.dll, version: 8.5.9600.16384, time stamp: 0x5215e17d
Exception code: 0xc0000005
Fault offset: 0x000000000000c579
Faulting process id: 0xfa8
Faulting application start time: 0x01cfb8e690730772
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\System32\inetsrv\filter.dll
Report Id: 449c28ec-24db-11e4-80bd-005056032dc5
Faulting package full name: 
Faulting package-relative application ID: 

This error only appears when I click on the page "SEO URLs" in Service Center.
Any suggestions to this issue?
 
Thanks,
Pedro Coelho
 
 Did you manage to get this working?
Hi,

This happened on a new server with Windows server 2012 R2, as Outsystems does not support this version of windows I opted to reinstall windows with windows server 2008.

Cheers,
Pedro Coelho
That's not really a solution but ok.
In the company I work for, we have parternship with Microsoft and we must install the last release of windows server, which in this case is 2012 :/
Hi Henrique

You have to analyze the crash dump to determine the cause of the IIS Worker process crash. Olny by analyzing the crash point stack, you can assert if the error comes from the system, platform or application.

@Pedro, support for Windows 2012 R2 has been added to version 9.0.

Cheers
@Miguel João
I have a eSpace that was developed for the version 7.0, is it possible to know if it is compatible?

Thanks for the tip, I will try to find out what is happening ;)
Hi Henrique

Upgrading the espace to version 9, means upgrading an environment to version 9. Although the Platform will upgrade the espace, it's possible that it might find some patterns that hit on identified breaking changes.

To perform an environment upgrade, you should evaluate the impacts after analyzing the the Installation checklist for version 9, as well as system requirements document, and breaking changes document.

Cheers
Is there a community license for version 9?
Hi Henrique,

No, the community edition evolved to the Personal Environment in the cloud, so there's no community edition licenses to 9.0.

You can use the Personal Environment in the cloud instead, whic is already in 9.0, so no need for upgrade.

Cheers
Just for the record, the SEO issue was one of the problems identified and fixed during the Windows 2012 R2 certification. So it already works correctly on the latest release of the platform.

Regards,
João Rosado
@Miguel - Can you tell me if it is possible to associate a DNS record to the cloud server?
@João - So... is it possible to use the new DLL for version 9? :P
Hi Henrique

Currently, is not possible to associate a custom DNS record to the Personal Environment.

Cheers
Hi

Debug Diag v2 from Microsoft is available for newer stacks of the platform.
https://www.microsoft.com/en-us/download/details.aspx?id=49924

Debug Diag v1.2 might be useful as well in older versions (e.g. it works on .NET 2.0 and Windows 2008R2)
https://www.microsoft.com/en-us/download/details.aspx?id=26798


Other useful readings:
Microsoft Support KB: https://support.microsoft.com/en-us/kb/2580960
DebugDiag blog: http://blogs.msdn.com/b/debugdiag/

Regards,
Acácio