- 2008.02.17 - Found this thread here. Possible condition in which the runtime not decrementing the counter.
- 2008.02.19 - This has already been reported to Microsoft on the connect site.
I've just noticed this, ironically, during a lull in our testing. I had some spare cycles and started using VSTS Team Test for load testing, trying to get a feel for it.. Not only does it allow you to collect performance counter data (as it should) it alerts you post test, if some values are outside recommended 'green' ranges.
Anyway, VSTS Team Test flagged my test runs with the information that the .NET CLR LocksAndThreads\Current Queue Length counter had a phenomenally high value...like > 200K and the fact that it kept increasing. On average it increased 1-2 every second.
The help for the .NET CLR LocksAndThreads\Current Queue Length counter states: This counter displays the total number of threads currently waiting to acquire some managed lock in the application. This counter is not an average over time; it displays the last observed value.
That tells me that it shouldn't be continually climbing unless we've got a problem. These servers were 'hot' in that the service instances had not been cycled for some time as part of stability testing, but that shouldn't account for what I was seeing.
Of course, this caused a cold lump to form in the pit of my stomach. You see, we *did* have a piece of code in a pipeline component that explicitly locks a resource. The code is relatively new, but had been performing well in so much as our throughput didn't seem impacted after deploying it.
Initially, I saw this as the Global instance being high, though it didn't take long to narrow it down to the BTSNTSvc.exe BizTalk process. The BizTalk host process in question has the send ports that leverage the WSE adapter (and the custom pipeline components).
I'm a huge fan of the "look in the mirror first" philosophy (sometimes to a fault) whenever there is a problem, so I took a memory snap of the of the BTSNTSvc.exe process.
I'm looking for anything that might indicate a problem either with a deadlock, or an abnormal number of threads. So I open the snap (*.dmp) in Windbg; !syncblk, !threads, !threadpool and !dumpheap -stat don't turn up anything unusual (to me).
Hmmm...this was originally spotted on a machine under test. So, for comparison, I checked out a my shiny new laptop that I had just finished configuring BizTalk Server 2006 Developer on. The image below shows a similar behavior. This host didn't actually have anything running in it yet (beyond core BizTalk), however, it too, had the telltale climb of the .NET CLR LocksAndThreads\Current Queue Length.
Googling hasn't turned up any fruit either. So I will shelve this one for now and review on Monday. If you've run into something similar, please feel free to smack me up side the head with it, or optionally, contact me. I'll try posting to the BizTalk forums over the weekend to see what turns up.
Configuration:
- Windows Server 2003 Standard x86, SP2
- BizTalk Server 2006 Enterprise
- CLR 2.0 (not sure of SP revision yet)
Thanks!
Z
