2009-09-23 138 views
6

自從幾個星期以來,我面臨着一個巨大的問題。我是一個在IIS7(W2008 SP1)下託管的asp.net應用程序,並且每隔幾個小時它就會開始佔用接近50%的CPU,這時可能沒有用戶連接。 這是可以理解的,因爲我們使用Quartz.net來做一些應用程序recicling,但是我們不能再現問題。ASP.NET:無負載下CPU佔用率高

這裏是JetBrains公司dotTrace 3.1製成,而CPU是高蹤:http://mycenter.info/tmp/DotTraceSnapshot.zip

通常在過程中浪費CPU是w3wp.exe的,但在過去幾天SQLSERVER(2008)和分佈式緩存(1.2。 1,並在週一更新到1.2.4測試版)也在殺死CPU。 奇怪的是有些時候memcached開始消耗100%,其統計數據表明它很安靜,但是當請求發生時它可以正常工作。

Here's故障轉儲W3WP的(或堆棧跟蹤轉儲),使用的WinDbg: (基於本指南:http://blogs.technet.com/marcelofartura/archive/2006/09/15/troubleshooting-iis-100-cpu-issues-step-by-step-intermediary.aspx

0:000> ~ 
. 0 Id: 1be4.1d3c Suspend: 1 Teb: 7ffdf000 Unfrozen 
    1 Id: 1be4.b1c Suspend: 1 Teb: 7ffde000 Unfrozen 
    2 Id: 1be4.12a0 Suspend: 1 Teb: 7ffdd000 Unfrozen 
    3 Id: 1be4.19d0 Suspend: 1 Teb: 7ffdc000 Unfrozen 
    4 Id: 1be4.1714 Suspend: 1 Teb: 7ffd7000 Unfrozen 
    5 Id: 1be4.1a18 Suspend: 1 Teb: 7ffd6000 Unfrozen 
    6 Id: 1be4.12ac Suspend: 1 Teb: 7ffd5000 Unfrozen 
    7 Id: 1be4.dec Suspend: 1 Teb: 7ffd4000 Unfrozen 
    8 Id: 1be4.1e48 Suspend: 1 Teb: 7ffd8000 Unfrozen 
    9 Id: 1be4.1ca8 Suspend: 1 Teb: 7ffd3000 Unfrozen 
    10 Id: 1be4.1508 Suspend: 1 Teb: 7ffaf000 Unfrozen 
    11 Id: 1be4.1bc0 Suspend: 1 Teb: 7ffae000 Unfrozen 
    12 Id: 1be4.1f48 Suspend: 1 Teb: 7ffad000 Unfrozen 
    13 Id: 1be4.1994 Suspend: 1 Teb: 7ffac000 Unfrozen 
    14 Id: 1be4.1a48 Suspend: 1 Teb: 7ffab000 Unfrozen 
    15 Id: 1be4.12c8 Suspend: 1 Teb: 7ffa8000 Unfrozen 
    16 Id: 1be4.e44 Suspend: 1 Teb: 7ffa7000 Unfrozen 
    17 Id: 1be4.19e0 Suspend: 1 Teb: 7ffa6000 Unfrozen 
    18 Id: 1be4.19b0 Suspend: 1 Teb: 7ffa2000 Unfrozen 
    19 Id: 1be4.1b30 Suspend: 1 Teb: 7ffd9000 Unfrozen 
    20 Id: 1be4.1bfc Suspend: 1 Teb: 7ffa3000 Unfrozen 
    21 Id: 1be4.1be8 Suspend: 1 Teb: 7ffa1000 Unfrozen 
    22 Id: 1be4.1a54 Suspend: 1 Teb: 7ffa5000 Unfrozen 
    23 Id: 1be4.b74 Suspend: 1 Teb: 7ff3d000 Unfrozen 
    24 Id: 1be4.19b4 Suspend: 1 Teb: 7ff3c000 Unfrozen 
    25 Id: 1be4.1460 Suspend: 1 Teb: 7ffdb000 Unfrozen 
    26 Id: 1be4.1eac Suspend: 1 Teb: 7ffaa000 Unfrozen 
    27 Id: 1be4.1b90 Suspend: 1 Teb: 7ffa4000 Unfrozen 


0:023> #23s 
Search address set to 77dc9a94 
*** WARNING: Unable to verify checksum for SMDiagnostics.ni.dll 
*** WARNING: Unable to verify checksum for System.Data.ni.dll 
*** ERROR: Module load completed but symbols could not be loaded for Microsoft.Web.Services3.DLL 
*** WARNING: Unable to verify checksum for System.Windows.Forms.ni.dll 
*** WARNING: Unable to verify checksum for System.Web.ni.dll 
*** WARNING: Unable to verify checksum for Ademy.UI.Web.DLL 
*** ERROR: Module load completed but symbols could not be loaded for AjaxControlToolkit.DLL 
*** ERROR: Module load completed but symbols could not be loaded for 7zSharp.DLL 
*** WARNING: Unable to verify checksum for mscorlib.ni.dll 
*** ERROR: Module load completed but symbols could not be loaded for Iesi.Collections.DLL 
*** WARNING: Unable to verify checksum for System.Design.ni.dll 
*** WARNING: Unable to verify checksum for System.Core.ni.dll 
*** WARNING: Unable to verify checksum for Ademy.Event.DLL 
*** WARNING: Unable to verify checksum for System.ServiceModel.ni.dll 
*** ERROR: Module load completed but symbols could not be loaded for System.ServiceModel.ni.dll 
*** WARNING: Unable to verify checksum for App_Theme_Ocean.wgubmrqt.dll 
*** WARNING: Unable to verify checksum for NHibernate.Burrow.AppBlock.DLL 
*** ERROR: Module load completed but symbols could not be loaded for NHibernate.Burrow.AppBlock.DLL 
*** WARNING: Unable to verify checksum for NHibernate.Caches.SysCache2.DLL 
*** ERROR: Module load completed but symbols could not be loaded for NHibernate.Caches.SysCache2.DLL 
*** WARNING: Unable to verify checksum for Ademy.UI.Web.Controls.DLL 
*** WARNING: Unable to verify checksum for Microsoft.JScript.ni.dll 
*** WARNING: Unable to verify checksum for System.Web.Mobile.ni.dll 
*** WARNING: Unable to verify checksum for System.Runtime.Serialization.ni.dll 
     ^Memory access error in '#23s' 

0:023> kb 
ChildEBP RetAddr Args to Child    
11c6ede4 77dc8ed4 766bc622 0000038c 00000000 ntdll!KiFastSystemCallRet 
11c6ede8 766bc622 0000038c 00000000 11c6ee20 ntdll!NtSetEvent+0xc 
11c6edf8 011011ef 0000038c 7f52be6e 0fda4888 kernel32!SetEvent+0x10 
WARNING: Frame IP not in any known module. Following frames may be wrong. 
11c6ee20 71b26ffe 060c5f9c 010039b0 010628a0 0x11011ef 
*** WARNING: Unable to verify checksum for System.ni.dll 
11c6ee4c 712c4b14 02528958 060c5f9c 11c6ee94 mscorlib_ni+0x216ffe 
11c6ee5c 712c4abe 060c5fb0 02528958 060c600c System_ni+0x144b14 
11c6ee94 71679260 060c5d24 7167926d 060c5d24 System_ni+0x144abe 
11c6eec8 717d8373 060c5d24 11c6f3e8 712c4ce4 System_ni+0x4f9260 
11c6ef14 712c4ce4 00000000 02528930 11c6ef74 System_ni+0x658373 
11c6ef54 7129dbcb 098b6ac4 11c6efec 72f7eff8 System_ni+0x144ce4 
11c6efa4 71b26d66 02df349c 11c6efc0 71b45681 System_ni+0x11dbcb 
11c6efb0 71b45681 00000000 0dcfd2d8 11c6efd0 mscorlib_ni+0x216d66 
11c6efc0 72f11b4c 766b45f1 00000000 11c6f050 mscorlib_ni+0x235681 
11c6efd0 72f221f9 11c6f0a0 00000000 11c6f070 mscorwks!CallDescrWorker+0x33 
11c6f050 72f36571 11c6f0a0 00000000 11c6f070 mscorwks!CallDescrWorkerWithHandler+0xa3 
11c6f194 72f365a4 71a91ff0 11c6f2c8 11c6f1e8 mscorwks!MethodDesc::CallDescr+0x19c 
11c6f1b0 72f365c2 71a91ff0 11c6f2c8 11c6f1e8 mscorwks!MethodDesc::CallTargetWorker+0x1f 
11c6f1c8 7302a471 11c6f1e8 68e9b644 0dcfd2d8 mscorwks!MethodDescCallSite::CallWithValueTypes+0x1a 
11c6f394 7302a5c6 11c6f424 68e9b194 02df34e4 mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+0x9f 
11c6f444 71b45577 11c6f3e8 02df17d0 01c177f8 mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0x10f 

預先感謝任何尖端!!

UPDATE:

Here's吊死線程的託管堆棧: I'm認爲它看起來像memcached的供應商,但還不確定我該怎麼辦。

0:023> !clrstack 
OS Thread Id: 0xb74 (23) 
ESP  EIP  
11c6ee38 77dc9a94 [NDirectMethodFrameStandaloneCleanup: 11c6ee38] Microsoft.Win32.Win32Native.SetEvent(Microsoft.Win32.SafeHandles.SafeWaitHandle) 
11c6ee48 71b26ffe System.Threading.EventWaitHandle.Set() 
11c6ee54 712c4b14 System.Net.TimerThread.Prod() 
11c6ee64 712c4abe System.Net.TimerThread+TimerQueue.CreateTimer(Callback, System.Object) 
11c6eea0 71679260 System.Net.ConnectionPool.CleanupCallbackWrapper(Timer, Int32, System.Object) 
11c6eed4 717d8373 System.Net.TimerThread+TimerNode.Fire() 
11c6ef1c 712c4ce4 System.Net.TimerThread+TimerQueue.Fire(Int32 ByRef) 
11c6ef5c 7129dbcb System.Net.TimerThread.ThreadProc() 
11c6efac 71b26d66 System.Threading.ThreadHelper.ThreadStart_Context(System.Object) 
11c6efb8 71b45681 System.Threading.ExecutionContext.runTryCode(System.Object) 
11c6f3e8 72f11b4c [HelperMethodFrame_PROTECTOBJ: 11c6f3e8] System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object) 
11c6f450 71b45577 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 
11c6f46c 71b301c5 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 
11c6f484 71b26ce4 System.Threading.ThreadHelper.ThreadStart() 
11c6f6b0 72f11b4c [GCFrame: 11c6f6b0] 
11c6f9a0 72f11b4c [ContextTransitionFrame: 11c6f9a0] 

發現的解決方案:

在Windows上運行2008年我更新到1.2.6時,這是由於爲Win32 memcached的中1.2.1的一個bug,和一切工作。我想我看到了w3wp進程,因爲我用來連接到memcached的庫有一個掛起的回收進程,即使memcached仍在響應。

SOLUTION 2 FOUND:

如果第一個解決方案不起作用,請閱讀THIS POST。我想memcached解決方案只是隱藏真正的問題,這是SmtpClient中的一個錯誤。

回答

2

在WinDbg中,問題:

〜* E clrstack

這將轉儲所有的管理線程堆棧和應該給你什麼是在這個過程中發生的事情的想法。

也試一試!失控,它會告訴你每個線程運行了多少時間。重點關注那些運行時間最長的頂級線程堆棧。

+0

謝謝! 「〜* e!clrstack」輸出「找不到導出clrstack」約20次,並且!runaway顯示線程(#23/Id:1be4.b74)運行10分鐘。我不知道這個命令,但我可以看出這是文章中描述的步驟後面的錯誤。 有沒有其他想法?我怎樣才能使〜!e!clrstack工作? – 2009-09-23 19:16:50

+0

謝謝!我已經添加了託管堆棧的問題,但它不是我的代碼是消耗CPU .. :( – 2009-09-24 13:43:24

+0

我找到了解決方案(見上文),感謝您的幫助! – 2009-09-28 13:14:32

0

這是否可能是由緩存問題引起的?例如,您是否有緩存的數據集設置爲在數據庫過期時自動重新加載?

我們曾經有過這種情況。我們有一個龐大的數據集,我們希望始終可用。數據並沒有經常改變,所以我們將其設置在緩存中,並且1小時到期,然後在我們的global.asax中,我們處理了刪除(如​​here所述,而不關注鏈接中描述的警告。我們在數小時後將數據集重新加載到緩存,這導致每小時CPU使用率和數據庫使用率都很高。

編輯 - 添加

不用說,我們很快看到了這一點,從我們的失誤教訓。