當運行一個分銷商是與5+工人比較忙碌,不時的服務停止處理消息。nServiceBus經銷商掛
NServiceBus版本是3.3.8與修復從4.x的一些反向移植(關於MessagePropertyFilters,鎖定關於ClearAvailabilityForWorkers和幾個小問題)的源代碼可以在這裏找到:
https://github.com/PeterLehmann/NServiceBus/tree/undoWfpNameChange
分析使用WinDbg掛服務顯示,大部分的線程在等待內存分配或釋放這樣的:
System.Runtime.InteropServices.GCHandle.InternalFree(IntPtr)
System.Runtime.InteropServices.GCHandle.Free()
System.Messaging.Interop.MessagePropertyVariants.Unlock()
System.Messaging.MessageQueue.ReceiveCurrent(System.TimeSpan, Int32, System.Messaging.Interop.CursorHandle, System.Messaging.MessagePropertyFilter, System.Messaging.MessageQueueTransaction, System.Messaging.MessageQueueTransactionType)
System.Messaging.MessageQueue.Peek(System.TimeSpan)
NServiceBus.Unicast.Queuing.Msmq.MsmqMessageReceiver.HasMessage()
NServiceBus.Unicast.Transport.Transactional.TransactionalTransport.HasMessage()
NServiceBus.Unicast.Transport.Transactional.TransactionalTransport.Process()
NServiceBus.Utils.WorkerThread.Loop()
System.Threading.ExecutionContext.runTryCode(System.Object)
和
System.Runtime.InteropServices.GCHandle.InternalAlloc(System.Object, System.Runtime.InteropServices.GCHandleType)
System.Runtime.InteropServices.GCHandle..ctor(System.Object, System.Runtime.InteropServices.GCHandleType)
System.Messaging.Interop.MessagePropertyVariants.Lock()
System.Messaging.MessageQueue.ReceiveCurrent(System.TimeSpan, Int32, System.Messaging.Interop.CursorHandle, System.Messaging.MessagePropertyFilter, System.Messaging.MessageQueueTransaction, System.Messaging.MessageQueueTransactionType)
System.Messaging.MessageQueue.Peek(System.TimeSpan)
NServiceBus.Unicast.Queuing.Msmq.MsmqMessageReceiver.HasMessage()
NServiceBus.Unicast.Transport.Transactional.TransactionalTransport.HasMessage()
NServiceBus.Unicast.Transport.Transactional.TransactionalTransport.Process()
NServiceBus.Utils.WorkerThread.Loop()
System.Threading.ExecutionContext.runTryCode(System.Object)
我們有多個線程與上述兩個疊似乎都掛着和消息沒有被處理。然後我們有一個線程也在等待InternalAlloc是在烏鴉通信的中間
System.Runtime.InteropServices.GCHandle.InternalAlloc(System.Object, System.Runtime.InteropServices.GCHandleType)
System.Runtime.InteropServices.GCHandle..ctor(System.Object, System.Runtime.InteropServices.GCHandleType)
System.Net.SafeDeleteContext.InitializeSecurityContext(System.Net.SecurDll, System.Net.SafeFreeCredentials ByRef, System.Net.SafeDeleteContext ByRef, System.String, System.Net.ContextFlags, System.Net.Endianness, System.Net.SecurityBuffer, System.Net.SecurityBuffer[], System.Net.SecurityBuffer, System.Net.ContextFlags ByRef)
System.Net.SSPIAuthType.InitializeSecurityContext(System.Net.SafeFreeCredentials, System.Net.SafeDeleteContext ByRef, System.String, System.Net.ContextFlags, System.Net.Endianness, System.Net.SecurityBuffer[], System.Net.SecurityBuffer, System.Net.ContextFlags ByRef)
System.Net.SSPIWrapper.InitializeSecurityContext(System.Net.SSPIInterface, System.Net.SafeFreeCredentials, System.Net.SafeDeleteContext ByRef, System.String, System.Net.ContextFlags, System.Net.Endianness, System.Net.SecurityBuffer[], System.Net.SecurityBuffer, System.Net.ContextFlags ByRef)
System.Net.NTAuthentication.GetOutgoingBlob(Byte[], Boolean, System.Net.SecurityStatus ByRef)
System.Net.NTAuthentication.GetOutgoingBlob(System.String)
System.Net.NegotiateClient.DoAuthenticate(System.String, System.Net.WebRequest, System.Net.ICredentials, Boolean)
System.Net.NegotiateClient.Authenticate(System.String, System.Net.WebRequest, System.Net.ICredentials)
System.Net.AuthenticationManager.Authenticate(System.String, System.Net.WebRequest, System.Net.ICredentials)
System.Net.AuthenticationState.AttemptAuthenticate(System.Net.HttpWebRequest, System.Net.ICredentials)
System.Net.HttpWebRequest.CheckResubmitForAuth()
System.Net.HttpWebRequest.CheckResubmit(System.Exception ByRef)
System.Net.HttpWebRequest.DoSubmitRequestProcessing(System.Exception ByRef)
System.Net.HttpWebRequest.ProcessResponse()
System.Net.HttpWebRequest.SetResponse(System.Net.CoreResponseData)
System.Net.ConnectStream.ProcessWriteCallDone(System.Net.ConnectionReturnResult)
System.Net.ConnectStream.CallDone(System.Net.ConnectionReturnResult)
System.Net.ConnectStream.WriteHeaders(Boolean)
System.Net.HttpWebRequest.EndSubmitRequest()
System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest, Boolean)
System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)
System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)
System.Net.HttpWebRequest.GetResponse()
Raven.Client.Connection.HttpJsonRequest.ReadStringInternal(System.Func`1<System.Net.WebResponse>)
Raven.Client.Connection.HttpJsonRequest.ReadResponseString()
Raven.Client.Connection.HttpJsonRequest.ReadResponseJson()
Raven.Client.Connection.ServerClient.DirectCommit(System.Guid, System.String)
Raven.Client.Connection.ServerClient+<>c__DisplayClass5b.<Commit>b__5a(System.String)
Raven.Client.Connection.ServerClient.TryOperation[[System.__Canon, mscorlib]](System.Func`2<System.String,System.__Canon>, System.String, Boolean, System.__Canon ByRef)
Raven.Client.Connection.ServerClient.ExecuteWithReplication[[System.__Canon, mscorlib]](System.String, System.Func`2<System.String,System.__Canon>)
Raven.Client.Connection.ServerClient.Commit(System.Guid)
Raven.Client.Document.DocumentSession.Commit(System.Guid)
Raven.Client.Document.RavenClientEnlistment.Commit(System.Transactions.Enlistment)
System.Transactions.Oletx.OletxEnlistment.CommitRequest()
...
我們試圖併發和Servermode垃圾收集的變化,它看起來像這個問題happend當垃圾回收器正在發生,但只有時候,我們看到GC完美運行,正常吞吐量不受影響。
我檢查是否有東西阻塞了終結器,但它似乎並不是這樣,做了一些挖掘,以發現內存分配/釋放中的某些東西是否會死鎖,但似乎無法找到任何東西(我可以在這裏深入挖掘,但首先需要輸入一些信息)。
目前業務正在重新啓動服務時出現這種情況然而即使這樣有時會失敗和服務陷入有時「停止」狀態,並已被殺害很難讓他們重新開始處理。
因此,有其他人遇到掛服務,nServiceBus 3.3.8?
它看起來像問題是圍繞閱讀消息屬性,某種競爭條件/內存泄漏... 你試過在3.3.8的發佈版本上運行這個嗎? 它只能在負載下重現嗎? 你可以在集成測試中重現這個嗎? –
它只能在負載下重現,並且不經常發生,所以我們沒有成功進行集成測試或負載測試來持續重現。我們已經使用3.3.8的發佈版本運行,具有完全相同的問題。 – jakobandersen
你可以試用最新版本的NSB嗎? –