Lock from ISAPI_Rewrite!TerminateFilter

ISAPI_Rewrite is Apache mod_rewrite compatible URL rewriter for Microsoft IIS
User avatar
Posts: 8
Joined: 11 Nov 2016, 20:29

Lock from ISAPI_Rewrite!TerminateFilter

20 Oct 2017, 20:36

Hi

We see sporadic spikes on execution of even simplest pages (all of them covered by ISAPI_REWRITE) and on analysis we found that spikes are very likely caused by locking on ISAPI_Rewrite!TerminateFilter. Is this a known issue, is there a fix or workaround?
Below is about 32 bit version. We made attempt to switch to 64bit, spikes were even larger, but dumps from 64bit were not collected yet.

Checking the callstack *ONLY* for the concerned long running page /maintenance/index.aspx running threads which you were concerned about:


Address Completed Timeout Time (secs) ThreadId SessionId PipelineStep ReturnCode Verb Url
88516428 no 110 327 158 200 GET /maintenance/index.aspx
97bdf060 no 110 248 94 200 GET /maintenance/index.aspx


Callstack of the long running thread looks like below:


0:094> kL
# ChildEBP RetAddr
00 84d9e36c 7700ebae ntdll!ZwWaitForSingleObject+0x15
01 84d9e3d0 7700ea92 ntdll!RtlpWaitOnCriticalSection+0x13e
02 84d9e3f8 10002550 ntdll!RtlEnterCriticalSection+0x150
03 84d9e4f8 10006d32 ISAPI_Rewrite+0x2550
04 84d9e5a0 1000d6de ISAPI_Rewrite+0x6d32
05 84d9e614 1000d35f ISAPI_Rewrite!TerminateFilter+0xf0e
06 84d9e65c 100776c6 ISAPI_Rewrite!TerminateFilter+0xb8f
07 84d9e698 730a9c64 ISAPI_Rewrite!TerminateFilter+0x6aef6
08 84d9e958 73d36965 filter!W3_FILTER_CONTEXT::NotifyFilters+0x153
09 84d9edfc 7353aaa2 iiscore!__SEH_epilog4+0x18
0a 84d9ee28 7349878f clr!UM2MThunk_Wrapper+0x22
0b 84d9eef8 739245a0 clr!NestedHandlerExRecord::Init+0x2
0c 84d9ef5c 00fad8ff clr! ?? ::FNODOBFM::`string'+0x90db
0d 84d9ef90 73c2ac86 CLRStub[StubLinkStub]@4f5988f100fad8ff
0e 84d9f004 73979554 webengine4!W3_MGD_HANDLER::ProcessNotification+0x62
0f 84d9f094 73485dd1 clr! ?? ::FNODOBFM::`string'+0x43a5e
10 84d9fa68 770358c4 clr!Thread::intermediateThreadProc+0x55
11 84d9fa88 770875cb ntdll!SQM_SETIFMIN_DWORD+0x4
12 84d9fa8c 00000000 ntdll!RtlInvalidHandlerDetected+0xa1


From the callstack, it’s pretty clear that your Third party isapi filter is interfering with the request and it is waiting for the critical section to be released.

Dumping out more Info about the Critical section, I see below:


Detected possible blocking or leaked critical section at ISAPI_Rewrite!TerminateFilter+93658 owned by thread 193 in w3wp__acclipse.com__PID__9812__Date__10_20_2017__Time_04_09_43PM__74__iisfreb!FREB_LOG_FILE_MANAGER__WriteLogFile.dmp

Impact of this lock

47.09% of threads blocked

(Threads 3 4 5 6 7 8 9 10 11 12 13 14 15 17 18 72 73 77 78 80 81 83 84 85 86 92 94 95 96 97 101 102 105 110 111 113 115 117 118 119 120 123 124 125 126 127 128 129 131 133 136 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 162 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 182 184 185 186 189 190)

Callstacks of owning thread 193:


0:193> kL
# ChildEBP RetAddr
00 3607dd54 1005e803 KERNELBASE!RaiseException+0x58
01 3607dd8c 100391a7 ISAPI_Rewrite!TerminateFilter+0x52033
02 3607de10 1000283e ISAPI_Rewrite!TerminateFilter+0x2c9d7
03 00000000 00000000 ISAPI_Rewrite+0x283e

There has been an exception being triggered here. Not sure if that is related but worth addressing.


More Information:

0:193> lmvm ISAPI_Rewrite
Browse full module list
start end module name
10000000 100ab000 ISAPI_Rewrite (export symbols) ISAPI_Rewrite.dll
Loaded symbol image file: ISAPI_Rewrite.dll
Image path: F:\iFirm-Live\URLRewrite\ISAPI_Rewrite.dll
Image name: ISAPI_Rewrite.dll
Browse all global symbols functions data
Timestamp: Tue Apr 05 14:23:47 2011 (4D9AD89B)
CheckSum: 000AE822
ImageSize: 000AB000
File version: 3.1.0.82
Product version: 3.1.0.82
File flags: 0 (Mask 17)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0000.04b0
CompanyName: Helicon Tech
ProductName: ISAPI_Rewrite3
InternalName: ISAPI_Rewrite3
OriginalFilename: ISAPI_Rewrite3.dll
ProductVersion: 3, 1, 0, 82
FileVersion: 3, 1, 0, 82
FileDescription: ISAPI_Rewrite3 filter dll
LegalCopyright: Copyright (C) 2002 - 2010 Helicon Tech. All rights reserved.
Comments: ISAPI_Rewrite3 filter dll

User avatar
Posts: 8
Joined: 11 Nov 2016, 20:29

Re: Lock from ISAPI_Rewrite!TerminateFilter

21 Oct 2017, 13:09

Below is thread which initiated TerminateFilter and busy with work while blocking all others.
The concern is that decision to call TerminateFilter is made by ISAPI_Rewrite!HttpFilterProc, not by IIS as it shouldbe.

34e9edb4 1000c762 ISAPI_Rewrite!HttpFilterProc+0x122, calling ISAPI_Rewrite!TerminateFilter+0xa90
34e9ede8 730a9c64 filter!W3_FILTER_CONTEXT::NotifyFilters+0x153


---------------------------------------------
Thread 164
Current frame: nativerd!CONFIG_ELEMENT::GetCollectionKey+0x39
ChildEBP RetAddr Caller, Callee
34e9ba48 73d7eb9a nativerd!CONFIG_COLLECTION::CalculateEffectiveCollection+0x198, calling nativerd!CONFIG_ELEMENT::GetCollectionKey
34e9ba78 74ce14ad kernel32!HeapFree+0x14, calling ntdll!RtlFreeHeap
34e9ba8c 73d5704a nativerd!ARRAY_LIST::~ARRAY_LIST+0x20, calling kernel32!HeapFree
34e9babc 73d78f32 nativerd!SCHEMA_ELEMENT::CreateElementObject+0x966, calling nativerd!CONFIG_COLLECTION::CalculateEffectiveCollection
34e9bb40 740b06ba iisutil!STRU::STRU+0x18, calling iisutil!BUFFER::BUFFER
34e9bb58 73d66c0a nativerd!ATTRIBUTE_VALUE::IsEqual+0x14b, calling msvcrt!_wcsicmp
34e9bb60 73d66b56 nativerd!ATTRIBUTE_VALUE::IsEqual+0x97, calling iisutil!STRA::~STRA
34e9bb6c 73d66b65 nativerd!ATTRIBUTE_VALUE::IsEqual+0xa6, calling nativerd!__security_check_cookie
34e9bb9c 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9bbf0 73d69773 nativerd!ATTRIBUTE_VALUE::SetString+0x95, calling msvcrt!memcpy
34e9bc10 73d719f2 nativerd!SCHEMA_ATTRIBUTE::CopyString+0x363, calling nativerd!ATTRIBUTE_VALUE::SetString
34e9bc30 73d66c0a nativerd!ATTRIBUTE_VALUE::IsEqual+0x14b, calling msvcrt!_wcsicmp
34e9bc38 73d66b56 nativerd!ATTRIBUTE_VALUE::IsEqual+0x97, calling iisutil!STRA::~STRA
34e9bc44 73d66b65 nativerd!ATTRIBUTE_VALUE::IsEqual+0xa6, calling nativerd!__security_check_cookie
34e9bc50 75264f82 KERNELBASE!LocalFree+0x27, calling ntdll!RtlFreeHeap
34e9bc78 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9bc98 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9bd04 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9bd20 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9bd24 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9bd58 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9bd5c 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9bd78 75265a52 KERNELBASE!LocalAlloc+0x5f, calling ntdll!RtlAllocateHeap
34e9bd98 75265a52 KERNELBASE!LocalAlloc+0x5f, calling ntdll!RtlAllocateHeap
34e9bda4 75265b8d KERNELBASE!LocalAlloc+0x19a, calling KERNELBASE!_SEH_epilog4
34e9bde0 740b9c71 iisutil!BUFFER::ReallocStorage+0x58, calling msvcrt!memcpy
34e9bdf0 740b06ba iisutil!STRU::STRU+0x18, calling iisutil!BUFFER::BUFFER
34e9bdf4 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9be10 73d66e70 nativerd!ATTRIBUTE_VALUE::CalculateKey+0x130, calling iisutil!STRA::~STRA
34e9be1c 73d66e83 nativerd!ATTRIBUTE_VALUE::CalculateKey+0x143, calling nativerd!__security_check_cookie
34e9be30 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9be34 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9be60 73d56e6a nativerd!INDEXED_ARRAY_LIST::FindEntry+0x2b
34e9bebc 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9bec0 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9bed4 73d8b696 nativerd!LOCKING_ATTRIBUTES::InitializeFromDomNode+0x278, calling nativerd!__security_check_cookie
34e9bf1c 76fee0a3 ntdll!RtlFreeHeap+0x105, calling ntdll!RtlpLowFragHeapFree
34e9bf34 76ffc96f ntdll!RtlpReAllocateHeap+0x190, calling ntdll!RtlFreeHeap
34e9bf40 76ffc983 ntdll!RtlpReAllocateHeap+0xaf7, calling ntdll!_SEH_epilog4
34e9bfb0 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9bfc0 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9bfc4 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9bfe4 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9bff8 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c048 76ffc983 ntdll!RtlpReAllocateHeap+0xaf7, calling ntdll!_SEH_epilog4
34e9c064 76ff42ff ntdll!RtlAllocateHeap+0x21d, calling ntdll!memset
34e9c078 76ff42ff ntdll!RtlAllocateHeap+0x21d, calling ntdll!memset
34e9c0c0 73d5670a nativerd!ARRAY_LIST::AllocateStorage+0x39, calling ntdll!RtlReAllocateHeap
34e9c0e0 73d5707d nativerd!ARRAY_LIST::AddEntry+0x29, calling nativerd!ARRAY_LIST::AllocateStorage
34e9c138 76fee0a3 ntdll!RtlFreeHeap+0x105, calling ntdll!RtlpLowFragHeapFree
34e9c150 75fe98cd msvcrt!free+0xcd, calling ntdll!RtlFreeHeap
34e9c15c 75fe98da msvcrt!free+0xec, calling msvcrt!_SEH_epilog4
34e9c170 740b0786 iisutil!STRU::AuxAppend+0x87, calling msvcrt!memcpy
34e9c190 740b2457 iisutil!STRU::Copy+0x1b, calling iisutil!STRU::AuxAppend
34e9c1a0 73d78e49 nativerd!SCHEMA_ELEMENT::CreateElementObject+0x87c, calling nativerd!SCHEMA_ELEMENT::CreateElementObject
34e9c238 73d80000 nativerd!CONFIG_COLLECTION::ApplyDifferenceInCollections+0x22f, calling nativerd!CONFIG_ELEMENT::QueryHasElementChanged
34e9c260 73d9b7b4 nativerd!CONFIG_SECTION_TABLE::InsertSection+0x84, calling nativerd!CONFIG_HASH<SECTION_GROUP_TABLE>::Insert
34e9c284 73d5750b nativerd!READ_ONLY_DOM_NODE::DereferenceConfigDomNode+0xf, calling kernel32!InterlockedDecrementStub
34e9c290 73d66066 nativerd!SECTION_GROUP_TABLE::Parse+0x449, calling iisutil!STRA::~STRA
34e9c29c 73d66075 nativerd!SECTION_GROUP_TABLE::Parse+0x458, calling nativerd!__security_check_cookie
34e9c350 6d0921c1 xmllite!XmlReader::ParseEndElement+0x94, calling xmllite!XmlReader::ParseQName
34e9c39c 76fee0a3 ntdll!RtlFreeHeap+0x105, calling ntdll!RtlpLowFragHeapFree
34e9c3b4 75fe98cd msvcrt!free+0xcd, calling ntdll!RtlFreeHeap
34e9c3c0 75fe98da msvcrt!free+0xec, calling msvcrt!_SEH_epilog4
34e9c3d4 74ce14ad kernel32!HeapFree+0x14, calling ntdll!RtlFreeHeap
34e9c3ec 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c408 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c414 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c434 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c450 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c480 73d56e6a nativerd!INDEXED_ARRAY_LIST::FindEntry+0x2b
34e9c494 73d56fa6 nativerd!DICTIONARY_LIST::GetValueByName+0x5a, calling nativerd!INDEXED_ARRAY_LIST::FindEntry
34e9c49c 73d56f92 nativerd!DICTIONARY_LIST::GetValueByName+0x46, calling nativerd!NAMED_ENTRY_KEY::~NAMED_ENTRY_KEY
34e9c4c4 73d572d8 nativerd!DICTIONARY_LIST::QueryValue+0x11, calling nativerd!DICTIONARY_LIST::GetValueByName
34e9c4d4 73d5759e nativerd!READ_ONLY_DOM_NODE::GetAttributeValue+0x1c, calling nativerd!DICTIONARY_LIST::QueryValue
34e9c4e0 73d8b5b9 nativerd!LOCKING_ATTRIBUTES::InitializeFromDomNode+0x19b
34e9c4e8 73d8b687 nativerd!LOCKING_ATTRIBUTES::InitializeFromDomNode+0x269, calling iisutil!STRA::~STRA
34e9c4f4 73d8b696 nativerd!LOCKING_ATTRIBUTES::InitializeFromDomNode+0x278, calling nativerd!__security_check_cookie
34e9c50c 73d66a47 nativerd!ATTRIBUTE_VALUE::ReferenceAttributeValue+0xf, calling kernel32!InterlockedIncrementStub
34e9c544 73d773dd nativerd!SCHEMA_ELEMENT::ApplySetAttributes+0x116
34e9c548 73d778a9 nativerd!SCHEMA_ELEMENT::ApplySetAttributes+0x5e2, calling iisutil!STRA::~STRA
34e9c5a4 75265099 KERNELBASE!LocalFree+0x13e, calling KERNELBASE!_SEH_epilog4
34e9c5a8 740b9ca1 iisutil!BUFFER::FreeMemory+0x14, calling kernel32!LocalFreeStub
34e9c5b4 73d65a3f nativerd!SECTION_GROUP_TABLE::ParseNewSectionGroupTable+0x3a8, calling iisutil!STRA::~STRA
34e9c5c8 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c5cc 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c654 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c658 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9c69c 76ff7009 ntdll!LdrUnlockLoaderLock+0x36, calling ntdll!RtlLeaveCriticalSection
34e9c6a0 76ff7010 ntdll!LdrUnlockLoaderLock+0xb4, calling ntdll!_SEH_epilog4
34e9c6ec 76fee0a3 ntdll!RtlFreeHeap+0x105, calling ntdll!RtlpLowFragHeapFree
34e9c708 76fee0a3 ntdll!RtlFreeHeap+0x105, calling ntdll!RtlpLowFragHeapFree
34e9c720 75262fab KERNELBASE!GetModuleFileNameA+0x9b, calling ntdll!RtlFreeHeap
34e9c734 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c7bc 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c7c0 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9c7d8 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c7dc 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9c834 75265a52 KERNELBASE!LocalAlloc+0x5f, calling ntdll!RtlAllocateHeap
34e9c840 75265b8d KERNELBASE!LocalAlloc+0x19a, calling KERNELBASE!_SEH_epilog4
34e9c868 740b61bd iisutil!CLKRLinearHashTable::_FindKey+0xb7, calling iisutil!CReaderWriterLock2::ReadUnlock
34e9c88c 740b6253 iisutil!CLKRHashTable::FindKey+0x6b, calling iisutil!CLKRLinearHashTable::_FindKey
34e9c898 73d79b02 nativerd!SCHEMA_ENTRY::CreateConfigElement+0x99, calling nativerd!SCHEMA_ELEMENT::CreateElementObject
34e9c8c8 73d63977 nativerd!CONFIG_SECTION::ParseConfigSetting+0x2ad
34e9c938 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9c93c 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9c990 73d9788a nativerd!CONFIG_HASH_NODE::Find+0x29, calling nativerd!CONFIG_HASH_KEY::IsEqual
34e9c9d0 73d6496e nativerd!CONFIG_HASH<CONFIG_ELEMENT>::ReadUnlock+0x14, calling iisutil!CReaderWriterLock2::ReadUnlock
34e9c9d8 73d64c33 nativerd!CONFIG_HASH<CONFIG_SECTION>::Find+0x6f, calling nativerd!CONFIG_HASH<CONFIG_ELEMENT>::ReadUnlock
34e9c9f8 73d64f75 nativerd!CONFIG_HASH<CONFIG_ELEMENT>::Find+0x36, calling nativerd!CONFIG_HASH<CONFIG_SECTION>::Find
34e9ca24 73d65583 nativerd!SECTION_GROUP_TABLE::ParseConfigSetting+0x292, calling nativerd!CONFIG_SECTION::ParseConfigSetting
34e9ca7c 740b0786 iisutil!STRU::AuxAppend+0x87, calling msvcrt!memcpy
34e9cac0 73d9788a nativerd!CONFIG_HASH_NODE::Find+0x29, calling nativerd!CONFIG_HASH_KEY::IsEqual
34e9cae4 73d978f5 nativerd!CONFIG_HASH_NODE::Insert+0x25, calling nativerd!CONFIG_HASH_NODE::Find
34e9cb0c 73d843bd nativerd!CONFIG_HASH<CONFIG_ELEMENT>::Insert+0x110, calling iisutil!CReaderWriterLock3::WriteUnlock
34e9cb54 73d9b7b4 nativerd!CONFIG_SECTION_TABLE::InsertSection+0x84, calling nativerd!CONFIG_HASH<SECTION_GROUP_TABLE>::Insert
34e9cb78 73d5750b nativerd!READ_ONLY_DOM_NODE::DereferenceConfigDomNode+0xf, calling kernel32!InterlockedDecrementStub
34e9cb84 73d66066 nativerd!SECTION_GROUP_TABLE::Parse+0x449, calling iisutil!STRA::~STRA
34e9cb90 73d66075 nativerd!SECTION_GROUP_TABLE::Parse+0x458, calling nativerd!__security_check_cookie
34e9ccd8 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9ccec 76ff71cd ntdll!RtlpLowFragHeapAllocFromContext+0x882, calling ntdll!RtlpSubSegmentInitialize
34e9ccfc 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9cd28 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9cd60 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9cd64 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9cdc0 76fee0a3 ntdll!RtlFreeHeap+0x105, calling ntdll!RtlpLowFragHeapFree
34e9cdd8 76ffc96f ntdll!RtlpReAllocateHeap+0x190, calling ntdll!RtlFreeHeap
34e9cde4 76ffc983 ntdll!RtlpReAllocateHeap+0xaf7, calling ntdll!_SEH_epilog4
34e9cdf8 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9cdfc 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9ce38 7525de95 KERNELBASE!ReadFile+0x118, calling ntdll!ZwReadFile
34e9ce50 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9ce68 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9ce88 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9ce9c 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9ceec 76ffc983 ntdll!RtlpReAllocateHeap+0xaf7, calling ntdll!_SEH_epilog4
34e9cf08 76ff42ff ntdll!RtlAllocateHeap+0x21d, calling ntdll!memset
34e9cf1c 76ff42ff ntdll!RtlAllocateHeap+0x21d, calling ntdll!memset
34e9cf64 73d5670a nativerd!ARRAY_LIST::AllocateStorage+0x39, calling ntdll!RtlReAllocateHeap
34e9cf84 73d5707d nativerd!ARRAY_LIST::AddEntry+0x29, calling nativerd!ARRAY_LIST::AllocateStorage
34e9cf9c 73d56954 nativerd!VALUE_ENTRY::SetValue+0x63, calling msvcrt!memcpy
34e9cfc0 73d573a6 nativerd!DICTIONARY_LIST::AddValue+0xba, calling nativerd!NAMED_ENTRY_KEY::~NAMED_ENTRY_KEY
34e9cfdc 76fee0a3 ntdll!RtlFreeHeap+0x105, calling ntdll!RtlpLowFragHeapFree
34e9cff4 75fe98cd msvcrt!free+0xcd, calling ntdll!RtlFreeHeap
34e9d000 75fe98da msvcrt!free+0xec, calling msvcrt!_SEH_epilog4
34e9d014 740b0786 iisutil!STRU::AuxAppend+0x87, calling msvcrt!memcpy
34e9d034 740b2457 iisutil!STRU::Copy+0x1b, calling iisutil!STRU::AuxAppend
34e9d040 73d654fe nativerd!SECTION_GROUP_TABLE::ParseConfigSetting+0x20c, calling nativerd!SECTION_GROUP_TABLE::ParseConfigSetting
34e9d0d8 76ffa1eb ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x17d, calling ntdll!memcpy
34e9d134 76ff71cd ntdll!RtlpLowFragHeapAllocFromContext+0x882, calling ntdll!RtlpSubSegmentInitialize
34e9d144 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d1ac 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d234 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d238 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9d2a4 7701d111 ntdll!RtlpWin32NTNameToNtPathName_U+0x6e, calling ntdll!memcpy
34e9d2c8 77019a13 ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x79, calling ntdll!RtlpWin32NTNameToNtPathName_U
34e9d2e0 76ffa263 ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x330, calling ntdll!__security_check_cookie
34e9d2fc 75265099 KERNELBASE!LocalFree+0x13e, calling KERNELBASE!_SEH_epilog4
34e9d35c 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d360 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9d38c 740b06ba iisutil!STRU::STRU+0x18, calling iisutil!BUFFER::BUFFER
34e9d400 73d843bd nativerd!CONFIG_HASH<CONFIG_ELEMENT>::Insert+0x110, calling iisutil!CReaderWriterLock3::WriteUnlock
34e9d430 740b0786 iisutil!STRU::AuxAppend+0x87, calling msvcrt!memcpy
34e9d438 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d450 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d45c 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d4dc 76ff71cd ntdll!RtlpLowFragHeapAllocFromContext+0x882, calling ntdll!RtlpSubSegmentInitialize
34e9d4ec 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d514 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d538 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d574 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d578 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9d5b8 76ff42ff ntdll!RtlAllocateHeap+0x21d, calling ntdll!memset
34e9d5d4 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9d5e8 73d56e6a nativerd!INDEXED_ARRAY_LIST::FindEntry+0x2b
34e9d5fc 73d868fc nativerd!LOCATION_TABLE::FindTableInternal+0xe4, calling nativerd!INDEXED_ARRAY_LIST::FindEntry
34e9d604 73d86920 nativerd!LOCATION_TABLE::FindTableInternal+0x108, calling nativerd!NAMED_ENTRY_KEY::~NAMED_ENTRY_KEY
34e9d634 73d8bce3 nativerd!LOCATION_CONTEXT::Reference+0xd, calling kernel32!InterlockedIncrementStub
34e9d640 73d5cdd3 nativerd!InvasivePtr<LOCATION_CONTEXT>::Attach<LOCATION_CONTEXT>+0xd, calling nativerd!InvasivePtr<LOCATION_CONTEXT>::Reset
34e9d64c 73d5d8ec nativerd!InvasivePtr<LOCATION_CONTEXT>::operator=<LOCATION_CONTEXT>+0x1e, calling nativerd!InvasivePtr<LOCATION_CONTEXT>::Attach<LOCATION_CONTEXT>
34e9d660 73d5e78a nativerd!CONFIG_FILE::ParseLocation+0x25e, calling nativerd!SECTION_GROUP_TABLE::ParseConfigSetting
34e9d6d8 6d092e65 xmllite!XmlReader::ResolveElementNamespace+0x12, calling xmllite!NamespaceManager::LookupNamespaceDeclaration
34e9d71c 76fee0a3 ntdll!RtlFreeHeap+0x105, calling ntdll!RtlpLowFragHeapFree
34e9d734 75fe98cd msvcrt!free+0xcd, calling ntdll!RtlFreeHeap
34e9d740 75fe98da msvcrt!free+0xec, calling msvcrt!_SEH_epilog4
34e9d754 740b0786 iisutil!STRU::AuxAppend+0x87, calling msvcrt!memcpy
34e9d774 740b2457 iisutil!STRU::Copy+0x1b, calling iisutil!STRU::AuxAppend
34e9d788 73d61e48 nativerd!CONFIG_FILE::ParseTopLevelElements+0x4a7, calling nativerd!CONFIG_FILE::ParseLocation
34e9d7e8 612c3d53 mlang!CMultiLanguage2::_InternalQueryInterface+0x16, calling mlang!ATL::AtlInternalQueryInterface
34e9d800 612c3d1e mlang!ATL::CComInternalCreator<ATL::CComTearOffObject<CMultiLanguage2> >::CreateInstance+0x5a, calling mlang!CMultiLanguage2::_InternalQueryInterface
34e9d830 612c24a7 mlang!ATL::AtlInternalQueryInterface+0x76
34e9d854 740b0786 iisutil!STRU::AuxAppend+0x87, calling msvcrt!memcpy
34e9d874 740b2457 iisutil!STRU::Copy+0x1b, calling iisutil!STRU::AuxAppend
34e9d890 73d62172 nativerd!CONFIG_FILE::ParseDom+0xe5, calling nativerd!CONFIG_FILE::ParseTopLevelElements
34e9d8c8 7659a06b ole32!CreateInprocInstanceHelper+0x1c
34e9d908 76599e37 ole32!ICoCreateInstanceEx+0x65d, calling ole32!__security_check_cookie
34e9d97c 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d984 6d093237 xmllite!SNodeData::OnCharacterSourceShift+0x1eb, calling xmllite!StringBuilder::OnCharacterSourceShift
34e9d994 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9d9c4 6d0961c8 xmllite!CharacterSource::SetEncoding+0x5a, calling xmllite!EncodingFactory::IsIdentificationEqual
34e9da1c 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9da20 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9da38 6d092e30 xmllite!XmlReader::ParseElement+0x36b, calling xmllite!XmlReader::ResolveElementNamespace
34e9da64 740b0786 iisutil!STRU::AuxAppend+0x87, calling msvcrt!memcpy
34e9da84 740a13d2 iisutil!STRU::Copy+0x24, calling iisutil!STRU::AuxAppend
34e9daa0 73d57644 nativerd!READ_ONLY_DOM_NODE::Create+0x81
34e9dab0 612c39fd mlang!ATL::CComTearOffObject<CMultiLanguage2>::Release+0x14, calling kernel32!InterlockedDecrementStub
34e9dac4 73d57bd8 nativerd!READ_ONLY_DOM::ParseXmlFile+0x195
34e9dae0 73d88ea5 nativerd!PROVIDER_ENTRY::ReferenceProviderEntry+0xa, calling kernel32!InterlockedIncrementStub
34e9daec 73d62755 nativerd!CONFIG_FILE::ParseConfigFile+0x1c7, calling nativerd!CONFIG_FILE::ParseDom
34e9db18 73d8e796 nativerd!CONFIG_CACHE::ParseConfigFile+0x23, calling nativerd!CONFIG_FILE::ParseConfigFile
34e9db38 73d9182c nativerd!CONFIG_CACHE::CacheMiss+0x57a, calling nativerd!CONFIG_CACHE::ParseConfigFile
34e9de94 74ce400e kernel32!BaseIsThisAConsoleName+0xc9
34e9dec8 74ce3f46 kernel32!CreateFileWImplementation+0x69, calling KERNELBASE!CreateFileW
34e9def0 74cfd8c4 kernel32!ReadDirectoryChangesW+0x93, calling ntdll!NtNotifyChangeDirectoryFile
34e9df18 76fee0a3 ntdll!RtlFreeHeap+0x105, calling ntdll!RtlpLowFragHeapFree
34e9df30 75264f82 KERNELBASE!LocalFree+0x27, calling ntdll!RtlFreeHeap
34e9df3c 75265099 KERNELBASE!LocalFree+0x13e, calling KERNELBASE!_SEH_epilog4
34e9e000 7659b157 ole32!CClassCache::CDllFnPtrMoniker::~CDllFnPtrMoniker+0x16, calling KERNELBASE!InterlockedDecrement
34e9e020 75fe98da msvcrt!free+0xec, calling msvcrt!_SEH_epilog4
34e9e024 73d581f3 nativerd!CONFIG_SYSTEM_FACTORY::`scalar deleting destructor'+0x1f, calling nativerd!operator delete[]
34e9e040 76599ee0 ole32!ICoCreateInstanceEx+0x64a, calling ole32!UpdateResultsArray
34e9e058 76599e37 ole32!ICoCreateInstanceEx+0x65d, calling ole32!__security_check_cookie
34e9e114 76ff71cd ntdll!RtlpLowFragHeapAllocFromContext+0x882, calling ntdll!RtlpSubSegmentInitialize
34e9e140 73d8c850 nativerd!PATH_CACHE::GetNode+0x26, calling nativerd!PATH_CACHE::GetNodeInternal
34e9e164 73d8e6ea nativerd!CONFIG_CACHE::CacheLookup+0x53, calling nativerd!PATH_CACHE::GetNode
34e9e194 73d925e9 nativerd!CONFIG_CACHE::TryGetDeepestCacheNode+0x124, calling nativerd!CONFIG_CACHE::CacheLookup
34e9e1b8 73d91b3b nativerd!CONFIG_CACHE::ProcessConfigPath+0x96, calling nativerd!CONFIG_CACHE::CacheMiss
34e9e1ec 73d91b96 nativerd!CONFIG_CACHE::GetConfigFileList+0x21, calling nativerd!CONFIG_CACHE::ProcessConfigPath
34e9e20c 73d7bcbb nativerd!CONFIG_SYSTEM::GetConfigSectionInternal+0x10d, calling nativerd!CONFIG_CACHE::GetConfigFileList
34e9e268 10067c15 ISAPI_Rewrite!TerminateFilter+0x5b445, calling ISAPI_Rewrite!TerminateFilter+0x609c4
34e9e280 1006716a ISAPI_Rewrite!TerminateFilter+0x5a99a, calling ISAPI_Rewrite!TerminateFilter+0x52069
34e9e288 10067176 ISAPI_Rewrite!TerminateFilter+0x5a9a6, calling ISAPI_Rewrite!TerminateFilter+0x56185
34e9e2b4 10067176 ISAPI_Rewrite!TerminateFilter+0x5a9a6, calling ISAPI_Rewrite!TerminateFilter+0x56185
34e9e2b8 1006740f ISAPI_Rewrite!TerminateFilter+0x5ac3f, calling ISAPI_Rewrite!TerminateFilter+0x5a964
34e9e2c4 10067390 ISAPI_Rewrite!TerminateFilter+0x5abc0, calling ISAPI_Rewrite!TerminateFilter+0x56185
34e9e2f4 100670e7 ISAPI_Rewrite!TerminateFilter+0x5a917, calling ISAPI_Rewrite!TerminateFilter+0x550e4
34e9e314 10067390 ISAPI_Rewrite!TerminateFilter+0x5abc0, calling ISAPI_Rewrite!TerminateFilter+0x56185
34e9e318 10067681 ISAPI_Rewrite!TerminateFilter+0x5aeb1, calling ISAPI_Rewrite!TerminateFilter+0x5aaa6
34e9e338 1006768f ISAPI_Rewrite!TerminateFilter+0x5aebf, calling ISAPI_Rewrite!TerminateFilter+0x52037
34e9e3a4 10067bc4 ISAPI_Rewrite!TerminateFilter+0x5b3f4, calling ISAPI_Rewrite!TerminateFilter+0x5afb5
34e9e3d8 1005e8ba ISAPI_Rewrite!TerminateFilter+0x520ea, calling ISAPI_Rewrite!TerminateFilter+0x5b319
34e9e45c 7702349c ntdll!RtlDispatchException+0x127, calling ntdll!RtlpExecuteHandlerForException
34e9e488 73d85dac nativerd!PATHMAP_TABLE::Create+0x351, calling nativerd!__security_check_cookie
34e9e508 7525c54f KERNELBASE!RaiseException+0x58, calling ntdll!RtlRaiseException
34e9e51c 76599cb2 ole32!CoCreateInstanceEx+0x75, calling ole32!CComActivator::DoCreateInstance
34e9e534 10006616 ISAPI_Rewrite+0x6616, calling ISAPI_Rewrite+0x6ab0
34e9e604 7525c54f KERNELBASE!RaiseException+0x58, calling ntdll!RtlRaiseException
34e9e78c 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9e7ec 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9e814 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9e818 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9e874 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9e878 76fee172 ntdll!RtlAllocateHeap+0x206, calling ntdll!RtlpLowFragHeapAllocFromContext
34e9e88c 1005e1c2 ISAPI_Rewrite!TerminateFilter+0x519f2
34e9e8ac 1005be7b ISAPI_Rewrite!TerminateFilter+0x4f6ab, calling ISAPI_Rewrite!TerminateFilter+0x51979
34e9e8c8 76fee0a3 ntdll!RtlFreeHeap+0x105, calling ntdll!RtlpLowFragHeapFree
34e9e8ec 7659e9f3 ole32!CRetailMalloc_Alloc+0x16, calling ntdll!RtlAllocateHeap
34e9e8fc 73d7ce81 nativerd!CONFIG_SYSTEM::GetAdminSection+0x9e, calling nativerd!CONFIG_SYSTEM::GetConfigSectionInternal
34e9e934 10036ddc ISAPI_Rewrite!TerminateFilter+0x2a60c
34e9e958 1005d1d7 ISAPI_Rewrite!TerminateFilter+0x50a07, calling ISAPI_Rewrite!TerminateFilter+0x59577
34e9e990 1005d3a3 ISAPI_Rewrite!TerminateFilter+0x50bd3, calling ISAPI_Rewrite!TerminateFilter+0x50987
34e9e9ac 1003855d ISAPI_Rewrite!TerminateFilter+0x2bd8d, calling ISAPI_Rewrite!TerminateFilter+0x2a580
34e9e9e8 10006616 ISAPI_Rewrite+0x6616, calling ISAPI_Rewrite+0x6ab0
34e9ea04 1005e803 ISAPI_Rewrite!TerminateFilter+0x52033, calling kernel32!RaiseExceptionStub
34e9ea3c 100391a7 ISAPI_Rewrite!TerminateFilter+0x2c9d7, calling ISAPI_Rewrite!TerminateFilter+0x51feb
34e9ea88 10036c11 ISAPI_Rewrite!TerminateFilter+0x2a441, calling ISAPI_Rewrite!TerminateFilter+0x2c960
34e9eab8 10002863 ISAPI_Rewrite+0x2863
34e9eb08 10001df5 ISAPI_Rewrite+0x1df5, calling ntdll!RtlLeaveCriticalSection
34e9eb48 10002632 ISAPI_Rewrite+0x2632, calling ISAPI_Rewrite+0x2790
34e9ebd0 10007a51 ISAPI_Rewrite+0x7a51, calling ISAPI_Rewrite+0x2460
34e9ec48 10006d32 ISAPI_Rewrite+0x6d32, calling ISAPI_Rewrite!TerminateFilter+0x4f8d1
34e9ec6c 1000de0e ISAPI_Rewrite!TerminateFilter+0x163e, calling ISAPI_Rewrite+0x7840
34e9ec9c 1003678d ISAPI_Rewrite!TerminateFilter+0x29fbd, calling ISAPI_Rewrite!TerminateFilter+0x4f5da
34e9ecf0 1000d6de ISAPI_Rewrite!TerminateFilter+0xf0e, calling ISAPI_Rewrite!TerminateFilter+0x1520
34e9ed64 1000d35f ISAPI_Rewrite!TerminateFilter+0xb8f, calling ISAPI_Rewrite!TerminateFilter+0xcc0
34e9ed88 75265099 KERNELBASE!LocalFree+0x13e, calling KERNELBASE!_SEH_epilog4
34e9edb4 1000c762 ISAPI_Rewrite!HttpFilterProc+0x122, calling ISAPI_Rewrite!TerminateFilter+0xa90
34e9ede8 730a9c64 filter!W3_FILTER_CONTEXT::NotifyFilters+0x153
34e9eeb0 73434c86 clr!COMNlsInfo::InternalGetCaseInsHash+0x100, calling clr!COMNlsHashProvider::HashiStringKnownLower80
34e9eebc 73434c86 clr!COMNlsInfo::InternalGetCaseInsHash+0x100, calling clr!COMNlsHashProvider::HashiStringKnownLower80
34e9eecc 733c882a clr!_EH_epilog3_GS+0xa, calling clr!__security_check_cookie
34e9eed0 73434ca6 clr!COMNlsInfo::InternalGetCaseInsHash+0x275, calling clr!_EH_epilog3_GS
34e9eee8 73434bbe clr!COMNlsHashProvider::HashiStringKnownLower80+0x1c4, calling clr!HashiStringKnownLower80
34e9eeec 76fee40c ntdll!RtlpLowFragHeapAllocFromContext+0xaec, calling ntdll!_SEH_epilog4
34e9ef08 731c45b1 authanon!WWWServerTraceProvider::CheckTracingEnabled+0x1a
34e9ef24 731c6963 authanon!IISAuthenticationEvents::AUTH_REQUEST_AUTH_TYPE::IsEnabled+0x11, calling authanon!WWWServerTraceProvider::CheckTracingEnabled
34e9ef2c 73d193c2 iiscore!WWWServerTraceProvider::CheckTracingEnabled+0x1a
34e9ef38 731c6fb1 authanon!TraceAuthSucceeded+0x24, calling authanon!IISAuthenticationEvents::AUTH_REQUEST_AUTH_TYPE::IsEnabled
34e9ef48 731c45b1 authanon!WWWServerTraceProvider::CheckTracingEnabled+0x1a
34e9ef5c 730aa664 filter!W3_FILTER_CONTEXT::HandleAuthComplete+0x85, calling filter!W3_FILTER_CONTEXT::NotifyFilters
34e9ef9c 730a4eaa filter!RequestDoWork+0x149, calling filter!W3_FILTER_CONTEXT::HandleAuthComplete
34e9efc8 730a4f78 filter!CIISHttpModule::OnPostAuthenticateRequest+0x16, calling filter!RequestDoWork
34e9efe4 73d3034b iiscore!NOTIFICATION_CONTEXT::RequestDoWork+0xbe
34e9eff8 73d311c0 iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+0x305, calling iiscore!NOTIFICATION_CONTEXT::RequestDoWork
34e9f04c 734892f1 clr!SVR::gc_heap::allocate_more_space+0x35, calling clr!SVR::gc_heap::try_allocate_more_space
34e9f070 73d1df13 iiscore!NOTIFICATION_CONTEXT::CallModules+0x28, calling iiscore!NOTIFICATION_CONTEXT::CallModulesInternal
34e9f0a0 73d2f53f iiscore!SITE_PERF_COUNTERS::IncrementCounter+0x25, calling kernel32!InterlockedIncrementStub
34e9f0b8 73d2257d iiscore!W3_CONTEXT::DoStateRequestAuthenticate+0x5a, calling iiscore!NOTIFICATION_CONTEXT::CallModules
34e9f0dc 73d22620 iiscore!W3_CONTEXT::DoWork+0x7c, calling iiscore!W3_CONTEXT::DoStateRequestAuthenticate
34e9f10c 73487e72 clr!Buffer::MemMove+0x12, calling MSVCR120_CLR0400!memmove
34e9f114 73485429 clr!SecurityStackWalk::MethodIsAnonymouslyHostedDynamicMethodWithCSToEvaluate+0x33, calling clr!MethodDesc::IsLCGMethod
34e9f14c 734856ba clr!SecurityDeclarative::DoDeclarativeActions+0x8c, calling clr!SecurityDeclarative::InvokeDeclarativeActions
34e9f168 733c6a9f clr!Frame::Pop+0x8, calling clr!GetThread
34e9f170 734855ff clr!JIT_Security_Prolog_Framed+0x15c, calling clr!_EH_epilog3
34e9f190 733e8179 clr!COMNlsInfo::InternalCompareStringOrdinalIgnoreCase+0xc5, calling clr!_EH_epilog3
34e9f1a0 73d193c2 iiscore!WWWServerTraceProvider::CheckTracingEnabled+0x1a
34e9f1bc 73d19917 iiscore!IISRequestNotificationEvents::NOTIFY_MODULE_START::IsEnabled+0x14, calling iiscore!WWWServerTraceProvider::CheckTracingEnabled
34e9f1cc 73d30b71 iiscore!NOTIFICATION_CONTEXT::GetNextNotification+0x44, calling iiscore!NOTIFICATION_CONTEXT::NotificationNumber
34e9f1d4 73d30c12 iiscore!NOTIFICATION_CONTEXT::GetNextNotification+0xe5
34e9f1f4 73d21d84 iiscore!W3_CONTEXT::GetNextNotification+0x41, calling iiscore!NOTIFICATION_CONTEXT::GetNextNotification
34e9f21c 73c2bb04 webengine4!W3_MGD_HANDLER::GetNextNotification+0x42
34e9f230 6f6ec805 (MethodDesc 6f4f2ef8 +0x25 System.Web.HttpRuntime.OnRequestNotificationCompletionHelper(System.IAsyncResult)), calling 017a88d6
34e9f264 6f6ec7b9 (MethodDesc 6f51fc98 +0x19 System.Web.HttpRuntime.OnRequestNotificationCompletion(System.IAsyncResult)), calling (MethodDesc 6f4f2ef8 +0 System.Web.HttpRuntime.OnRequestNotificationCompletionHelper(System.IAsyncResult))
34e9f268 720ae836 (MethodDesc 71d349fc +0x16 System.Globalization.CultureInfo.get_UserDefaultUICulture()), calling (JitHelp: CORINFO_HELP_GETSHARED_GCSTATIC_BASE)
34e9f284 73c2bba9 webengine4!MgdGetResponseChunks+0x57
34e9f29c 6f71fae9 (MethodDesc 6f538920 +0x91 DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, Int32 ByRef, IntPtr[], Int32[], Int32[]))
34e9f2ac 73d30d91 iiscore!NOTIFICATION_CONTEXT::GetIsLastNotification+0x89
34e9f2c8 73d21e18 iiscore!W3_CONTEXT::GetIsLastNotification+0x6d, calling iiscore!NOTIFICATION_CONTEXT::GetIsLastNotification
34e9f2f0 73c2b529 webengine4!W3_MGD_HANDLER::GetLastNotification+0x1c
34e9f304 73c2b879 webengine4!W3_MGD_HANDLER::CanDisposeManagedContext+0x14, calling webengine4!W3_MGD_HANDLER::GetLastNotification
34e9f314 73c2b85d webengine4!MgdCanDisposeManagedContext+0x1d, calling webengine4!W3_MGD_HANDLER::CanDisposeManagedContext
34e9f320 6f71fa1d (MethodDesc 6f538900 +0x35 DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus))
34e9f344 73d262e4 iiscore!W3_MAIN_CONTEXT::ContinueNotificationLoop+0x1f
34e9f364 73d2633d iiscore!W3_MAIN_CONTEXT::ProcessIndicateCompletion+0x1f, calling iiscore!W3_MAIN_CONTEXT::ContinueNotificationLoop
34e9f378 73d207da iiscore!W3_CONTEXT::IndicateCompletion+0x75, calling iiscore!W3_MAIN_CONTEXT::ProcessIndicateCompletion
34e9f388 73c85afd webengine4!W3_MGD_HANDLER::IndicateCompletion+0x45
34e9f39c 73c87ea2 webengine4!MgdIndicateCompletion+0x22, calling webengine4!W3_MGD_HANDLER::IndicateCompletion
34e9f3ac 6f71d7e1 (MethodDesc 6f5390f8 +0x3d DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef))
34e9f3cc 6f71d7e1 (MethodDesc 6f5390f8 +0x3d DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef))
34e9f3ec 6f6dbf72 (MethodDesc 6f521384 +0x4d2 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)), calling 6f6541c8
34e9f4e0 6f6dba7f (MethodDesc 6f521378 +0x1f System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)), calling (MethodDesc 6f521384 +0 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32))
34e9f508 00fad8a6 00fad8a6
34e9f51c 733bec5c clr!UM2MThunk_WrapperHelper+0x10
34e9f528 7353aaf1 clr!UM2MThunk_Wrapper+0x76, calling clr!UM2MThunk_WrapperHelper
34e9f54c 7353aaa2 clr!UM2MThunk_Wrapper+0x22, calling clr!_alloca_probe
34e9f588 7353aa22 clr!Thread::DoADCallBack+0xbc
34e9f5bc 733bf43e clr!CrstBase::Enter+0x11b, calling ntdll!RtlTryEnterCriticalSection
34e9f5e8 73d29da9 iiscore!CONTEXT_ALLOC::AllocateMemory+0x32, calling iiscore!CONTEXT_ALLOC_BUFFER::Alloc
34e9f628 733c6a9f clr!Frame::Pop+0x8, calling clr!GetThread
34e9f630 73471fdd clr!Thread::DoExtraWorkForFinalizer+0x660, calling clr!Frame::Pop
34e9f654 7353a98a clr!UM2MDoADCallBack+0x92, calling clr!Thread::DoADCallBack
34e9f6ac 00fad8ff 00fad8ff, calling clr!UM2MDoADCallBack
34e9f6e0 73c2ac86 webengine4!W3_MGD_HANDLER::ProcessNotification+0x62
34e9f704 73c2ace3 webengine4!ProcessNotificationCallback+0x33, calling webengine4!W3_MGD_HANDLER::ProcessNotification
34e9f718 73471097 clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x1d6
34e9f764 734722c3 clr!ThreadpoolMgr::ExecuteWorkRequest+0x4f
34e9f77c 73472109 clr!ThreadpoolMgr::WorkerThreadStart+0x3d3, calling clr!ThreadpoolMgr::ExecuteWorkRequest
34e9f7ac 733c822b clr!EEHeapFree+0x3b, calling kernel32!HeapFree
34e9f7e4 73485dd1 clr!Thread::intermediateThreadProc+0x55
34e9f804 76ff03d8 ntdll!LdrGetProcedureAddressEx+0xca, calling ntdll!_SEH_epilog4_GS
34e9f808 76ff0262 ntdll!LdrGetProcedureAddress+0x18, calling ntdll!LdrGetProcedureAddressEx
34e9f824 75261f7c KERNELBASE!GetProcAddress+0x44, calling ntdll!LdrGetProcedureAddress
34e9f840 63ebc0ab ABCpdf8_32!NTransform_Skew+0x4cda3b, calling ntdll!RtlLeaveCriticalSection
34e9f84c 63ebd25a ABCpdf8_32!NTransform_Skew+0x4cebea, calling ABCpdf8_32!NTransform_Skew+0x4cda28
34e9f854 63ebd252 ABCpdf8_32!NTransform_Skew+0x4cebe2, calling ABCpdf8_32!NTransform_Skew+0x4cccbd
34e9f884 63ebd252 ABCpdf8_32!NTransform_Skew+0x4cebe2, calling ABCpdf8_32!NTransform_Skew+0x4cccbd
34e9f888 63eb8128 ABCpdf8_32!NTransform_Skew+0x4c9ab8, calling ABCpdf8_32!NTransform_Skew+0x4ceb2d
34e9f890 63eb8130 ABCpdf8_32!NTransform_Skew+0x4c9ac0, calling KERNELBASE!GetCurrentThreadId
34e9f8a8 63eb81c6 ABCpdf8_32!NTransform_Skew+0x4c9b56, calling ABCpdf8_32!NTransform_Skew+0x40
34e9f8e4 63eb824f ABCpdf8_32!NTransform_Skew+0x4c9bdf, calling ABCpdf8_32!NTransform_Skew+0x4cccbd
34e9f8e8 63eb826d ABCpdf8_32!NTransform_Skew+0x4c9bfd, calling ABCpdf8_32!NTransform_Skew+0x4c9aea
34e9f8f0 76ff9364 ntdll!LdrpCallInitRoutine+0x14
34e9f918 76ff9b0d ntdll!LdrpInitializeThread+0x1f5, calling ntdll!RtlLeaveCriticalSection
34e9f920 76ff9aea ntdll!LdrpInitializeThread+0x1df, calling ntdll!_SEH_epilog4
34e9f974 76ff9b4c ntdll!LdrpInitializeThread+0x131, calling ntdll!RtlActivateActivationContextUnsafeFast
34e9f978 76ff9afa ntdll!LdrpInitializeThread+0x179, calling ntdll!RtlDeactivateActivationContextUnsafeFast
34e9f9ac 76ff9aea ntdll!LdrpInitializeThread+0x1df, calling ntdll!_SEH_epilog4
34e9f9b4 76ff9871 ntdll!_LdrpInitialize+0x244, calling ntdll!_SEH_epilog4
34e9f9fc 76ff9893 ntdll!LdrInitializeThunk+0x1a, calling ntdll!ZwContinue
34e9fc70 73485db7 clr!Thread::intermediateThreadProc+0x3b, calling clr!_alloca_probe_16
34e9fc84 74ce336a kernel32!BaseThreadInitThunk+0xe
34e9fc90 76ff9902 ntdll!__RtlUserThreadStart+0x70
34e9fcd0 76ff98d5 ntdll!_RtlUserThreadStart+0x1b, calling ntdll!__RtlUserThreadStart
---------------------------------------------

User avatar
Posts: 402
Joined: 06 Mar 2012, 11:59

Re: Lock from ISAPI_Rewrite!TerminateFilter

23 Oct 2017, 07:46

Hello.

The version of ISAPI_Rewrite you are using is 3.1.0.82. the current version available for download is 3.1.0.112. Please upgrade your ISAPI_Rewrite installation to the most current build first as we cannot debug older builds of the software. Simply download new package and hit install, it should upgrade automatically (unless you have a severely custom configuration of course). If the problem persists, please send all your ISAPI_Rewrite configuration files here for review.

User avatar
Posts: 8
Joined: 11 Nov 2016, 20:29

Re: Lock from ISAPI_Rewrite!TerminateFilter

23 Oct 2017, 09:50

Hi,
Thanks a lot for a quick reply. We've made 2 day run on latest 3.1.0.112 64bit version. We had to rollback because of other issue, but based on some indicators we had same issues as in 3.1.0.82. No debugdiag dumps however were collected on 3.1.0.112

Do you think in newer version TerminateFilter will not be called from ISAPI_Rewrite!HttpFilterProc?

Other possible concern we found is in platform we are running. Windows 2008 R2 might be affected https://support.microsoft.com/en-us/help/2618982/fix-memory-leak-in-rhs-exe-after-you-configure-the-iis-7-5-w3svc-servi which causes too long time in GetAdminSection.

We will upgrade and re-validate by collecting more dumps.

Thanks

User avatar
Posts: 402
Joined: 06 Mar 2012, 11:59

Re: Lock from ISAPI_Rewrite!TerminateFilter

24 Oct 2017, 06:29

Please consider upgrading to Helicon Ape then. Ape is a different product and based on different technology, yet it supports same URL rewriting syntax as ISAPI_Rewrite and offers much more. If you find that Ape fixes your issue you are eligible for free upgrade. Please note that Ape and ISAPI_Rewrite should not be installed together on the same server To avoid possible errors I suggest to uninstall ISAPI_Rewrite before installing Ape, thus the whole operation is better to be carried in a staging environment first.

User avatar
Posts: 8
Joined: 11 Nov 2016, 20:29

Re: Lock from ISAPI_Rewrite!TerminateFilter

05 Nov 2017, 14:37

Hi,
We've installed windows hotfix for nativerd.dll https://support.microsoft.com/en-us/help/2618982/fix-memory-leak-in-rhs-exe-after-you-configure-the-iis-7-5-w3svc-servi but our version of nativerd 22009 already beyond hotfix 21825 and isapi_rewrite is 112, but problem still here
The stack goes deep into nativerd, reading web, appool and machine.config configuration files completely on every request. If needed - we can share procmon. I've though that it's a problem of large configuration files MaxCachedFileSize
https://support.microsoft.com/en-us/help/954864/description-of-the-registry-keys-that-are-used-by-iis-7-0--iis-7-5--an
The largest - C:\inetpub\temp\appPools\poolname config is 112K and it large, contains 100+ applications so our configuration files are well below 256K

Can you please answer:

1. Why isapi_rewrite entering TerminateFilter so frequently, can it be prevented?
2. How TerminateFilter and other flows call GetAdminSection and is it possible to cache those calls and call it much less frequently?
3. Can 1-2 be achieved by configuration or new version is required and do you plan new version?
4. If you think the APE is our only option - do you have a migration guide from ISAPI_rewrite to APE, can we reuse same config file?

Please let me know if I need to email you httpd.conf (don't want to share it public).
Call stack from procmon is attached.
Attachments
Callstack_nativerd_112_2.CSV
(7.74 KiB) Downloaded 664 times

User avatar
Posts: 402
Joined: 06 Mar 2012, 11:59

Re: Lock from ISAPI_Rewrite!TerminateFilter

11 Nov 2017, 09:14

Hello.

1. TerminateFilter is never called from inside of ISAPI_Rewrite code. What you are reading is either incorrect data or memory corruption. BTW, as a possible result of memory corruption, the whole call stack may be invalid.
2. ISAPI_Rewrite calls for GetAdminSection every time it starts, to find new sites. As far as I can see it does not call it when the filter is terminated.
3. It is still not clear what to achieve and how.
4. Ape uses the completely different code, in fact, it is written in .NET. Configurations should be compatible, you will just copy your existing configuration. Please note ISAPI_Rewrite and Ape should not run on the same site side-by-side as they may conflict. You will definitely need a spare staging environment to complete this move, especially on high load sites. Although the upgrade may go completely automatic, there are just too many places where things may go awry, so don't risk doing it in the live environment.

User avatar
Posts: 8
Joined: 11 Nov 2016, 20:29

Re: Lock from ISAPI_Rewrite!TerminateFilter

22 Nov 2017, 13:20

Hi Yaroslav,
We've got very good progress with replacement of ISAPI_REwrite to the latest version of APE.

We have following questions:
1. We are using RewriteMap to the text file with slowly changing map data (10-100 times a day).
RewriteMap sitemap txt:..\..\URLRewrite\sitemapper.txt
RewriteRule ^/([^/]+)/(.*)$ /$1/$ {sitemap:%1|notfound}
When this text file sitemapper.txt is modified by concurrent threads both APE and ISAPI_Rewrite are failing with exception.
According to https://www.helicontech.com/ape/doc/mod_core.htm we've applied Options All -StopOnError -FileChangesEventedMode and now file modifications don't crash filter. In documentation it's stated that When disabled, file changes will be monitored using poll method. In Procmon we don't see calls to read configuration files. How frequent is poll interval, can it be configured, can we trigger reread of configuration files by any other means except apppool recycle?

2. How should we convert ISAPI_rewrite licenses to APE licenses, should we use same registration codes?

If short meeting required - let me know.

Thanks

User avatar
Posts: 402
Joined: 06 Mar 2012, 11:59

Re: Lock from ISAPI_Rewrite!TerminateFilter

29 Nov 2017, 02:08

Hello.

Please find answers below:

1. This is strange. I suppose product is not crashing but reports an error instead. I'd like to see that error message. Such situation may happen if the program that modifies the map file locks access to it after making modifications. Also, in such situation Ape may report access error, but will be able to read the file later, after file the lock is lifted. With evented mode disabled Ape will re-check files quite often, but it will not re-read them if they are not modified. You may need to watch for another access message in FileMon. Check if it works first, for example, modify files several times and confirm changes are getting loaded and Ape responds to the new configuration.

2. Please identify your ISAPI_Rewrite license somehow and I will provide a new license for you. For example tell your registration anme or e-mail.

Return to ISAPI_Rewrite 3.0

Who is online

Users browsing this forum: No registered users and 14 guests