20100318

Debugging Deadlocks in ASP .NET

Deadlock detection

Deadlock is reported in Event Viewer

Section: Application
Source: W3SVC-WP
ID: 2262
Desc: ISAPI 'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.

and

Section: System
Source: W3SVC
ID: 1013
Desc: A process serving application pool 'xxx' exceeded time limits during shut down. The process id was 'xxx'.

Server configuration

After detecting deadlock IIS kills old and creates new process.
To prevent it enable the Orphan Worker Process feature.

WARNING: Orphaned Worker Processes must be killed manually to free resources and memory.

cd \Inetpub\adminscripts
adsutil.vbs SET W3SVC/AppPools/DefaultAppPool/OrphanWorkerProcess TRUE

where DefaultAppPool should be replaced with proper Application Pool name
List of Application Pools together with Process Ids can be obtained by IISAPP tool (type iisapp in cmd).

You can use automatic dump collector but it seems that better get it by hand.

Dump collecting

Best way to get dump is using DebugDiag tool.
Go to Processes tab right-click on selected process and select Create Full Userdump.

Quick dump analysis

Using DebugDiag you can quickly get info from created dump.
Right-click on .dmp file and select Analyze Crash/Hang Issue it will show you

Stack trace of each Thread
All current requests
But it won't load symbols for your application.
Better way is to use WINDBG described below.

Detailed dump analysis

Procedure is described here.

1. Install Debugging Tools.
2. Run C:\Program Files\Debugging Tools for Windows\windbg.exe.
3. Open Crash Dump (form File menu)
4. Important commands:
.symfix binpath
.reload
.loadby sos mscorwks
.cordll -ve -u -l
!threads
~* e !clrstack
Last line should display all threads with call stacks and symbols resolved.

System.Threading.Monitor.Enter means that thread waits on lock (...) {...} statement.

What can possibly go wrong?

Wrong CLR version

!threads command says:

CLRDLL: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.3053 f:0
doesn't match desired version 2.0.50727.3082 f:0
CLRDLL: Unable to find mscordacwks_x86_x86_2.0.50727.3082.dll by mscorwks search
CLRDLL: Unable to find 'mscordacwks_x86_x86_2.0.50727.3082.dll' on the path
CLRDLL: Unable to find mscorwks.dll by search
CLRDLL: ERROR: Unable to load DLL mscordacwks_x86_x86_2.0.50727.3082.dll, Win32 error 0n2
Failed to load data access DLL, 0x80004005

Solution

Install all Service Packs from here.

.NET Framework 2.0 Service Pack 2
.NET Framework 3.0 Service Pack 2
.NET Framework 3.5 Service Pack 1

1 comment:

Anonymous said...

When debugging a rich client I too found the problem with WinDbg not being able to find a particular version of the mscordacwks assembly.

I was able to make progress by copying the assembly from the machine that I had created the dmp file on to my PC's C:\Windows\Microsoft.NET\Framework\v2.0.50727 directory and naming it correctly. i.e. the version of mscordacwks.dll on my PC is 2.0.50727.4927. I made a copy of the assembly on the other PC, which was version 2.0.50727.3082, and named it mscordacwks_x86_x86_2.0.50727.3082.dll. I then copied the assembly to my PC's C:\Windows\Microsoft.NET\Framework\v2.0.50727 directory.