每cf bind-service
或cf unbind-service
Servicebroker(Cloud Foundry組件)會創建新的隨機生成的憑證(請參閱cf env $APP
)。通過我們的MongoEnterprise服務產品,Service Broker連接到Ops Manager API並將新用戶部署到mongod replicaset。
當行動管理器部署用戶(createUser
)他使用WriteConcern「多數」的,這意味着至少一個次級需要以確認寫操作。
COMMAND [conn53768] command rs_$DBNAME.$cmd command: createUser { createUser: "$USERNAME", pwd: "xxx", digestPassword: false, roles: [ { role: "readWrite", db: "rs_$DBNAME" } ], writeConcern: { w: "majority" } } keyUpdates:0 writeConflicts:0 numYields:0 reslen:138 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { W: 2 } }, Collection: { acquireCount: { w: 2 } } } protocol:op_query 9811ms
由於mongorestore
,將其進行了幾個小時發生此問題之前,有一種可能性,即createUser
命令是由於從分別在前景忙建築物索引次級缺乏確認超時和背景。索引版本(默認mongorestore
選項)首先在主文件上完成,完成時將在副文件中進行。這解釋了在createUser
期間副期的滯後上升。
根據索引和數據大小,索引構建可能需要很長時間。我們聽到很多客戶抱怨索引構建時間過長。
這是來自索引構建的日誌。在Ops Manager中,客戶可以在所有副本集成員稍微延遲的情況下看到流式傳輸mongodb.log
(opsmanager.service.consul)。一切與WriteConcern:
...
2017-02-15T10:40:06.199+0000 I INDEX [repl writer worker 10] build index done. scanned 1108917 total records. 672 secs
2017-02-15T10:50:08.553+0000 I INDEX [repl writer worker 4] build index done. scanned 1108917 total records. 602 secs
2017-02-15T11:01:13.888+0000 I INDEX [repl writer worker 7] build index done. scanned 1108917 total records. 665 secs
...
2017-02-15T15:01:37.405+0000 I INDEX [repl index builder 176] build index done. scanned 1109531 total records. 659 secs
2017-02-15T15:01:37.406+0000 I INDEX [repl index builder 170] build index done. scanned 1109531 total records. 659 secs
2017-02-15T15:16:20.139+0000 I INDEX [repl index builder 170] build index done. scanned 1109699 total records. 882 secs
這些都是從Automation Agent錯誤
[2017/02/15 13:33:36.297] [.error] [cm/mongoctl/authctl.go:updateUser26Style:697] [101] <rs_$DB_NAME> [13:33:36.297] Error updating 2.6-style user = AuthUser([email protected]_$DB_NAME,roles=AuthRoles(1:{"Rolename":"readWrite","Db":"rs_$DB_NAME"}),interned=true,identity=2) in db = rs_$DB_NAME : <rs_$DB_NAME> [13:33:36.297] Timed out (timeout=-1ns) trying to runCommandWithTimeout(dbName=rs_$DB_NAME, cmd=[{"Name":"updateUser","Value":"$USER"},{"Name":"pwd","Value":"$PASSWORD"},{"Name":"digestPassword","Value":false},{"Name":"roles","Value":[{"db":"rs_$DB_NAME","role":"readWrite"}]}])
[2017/02/15 13:33:36.297] [.error] [cm/mongoctl/authctl.go:UpsertUser:516] [101] <rs_$DB_NAME> [13:33:36.297] Error upserting 2.6-style user = AuthUser([email protected]_$DB_NAME,roles=AuthRoles(1:{"Rolename":"readWrite","Db":"rs_$DB_NAME"}),interned=true,identity=2) : <rs_$DB_NAME> [13:33:36.297] Error updating 2.6-style user = AuthUser([email protected]_$DB_NAME,roles=AuthRoles(1:{"Rolename":"readWrite","Db":"rs_$DB_NAME"}),interned=true,identity=2) in db = rs_$DB_NAME : <rs_$DB_NAME> [13:33:36.297] Timed out (timeout=-1ns) trying to runCommandWithTimeout(dbName=rs_$DB_NAME, cmd=[{"Name":"updateUser","Value":"$USER"},{"Name":"pwd","Value":"$PASSWORD"},{"Name":"digestPassword","Value":false},{"Name":"roles","Value":[{"db":"rs_$DB_NAME","role":"readWrite"}]}])
[2017/02/15 13:33:36.297] [.error] [cm/action/adjustusers.go:adjustUsers:52] [101] <rs_$DB_NAME> [13:33:36.297] Error upserting user = AuthUser([email protected]_$DB_NAME,roles=AuthRoles(1:{"Rolename":"readWrite","Db":"rs_$DB_NAME"}),interned=true,identity=2). Trying to proceed though. : <rs_$DB_NAME> [13:33:36.297] Error upserting 2.6-style user = AuthUser([email protected]_$DB_NAME,roles=AuthRoles(1:{"Rolename":"readWrite","Db":"rs_$DB_NAME"}),interned=true,identity=2) : <rs_$DB_NAME> [13:33:36.297] Error updating 2.6-style user = AuthUser([email protected]_$DB_NAME,roles=AuthRoles(1:{"Rolename":"readWrite","Db":"rs_$DB_NAME"}),interned=true,identity=2) in db = rs_$DB_NAME : <rs_$DB_NAME> [13:33:36.297] Timed out (timeout=-1ns) trying to runCommandWithTimeout(dbName=rs_$DB_NAME, cmd=[{"Name":"updateUser","Value":"$USER"},{"Name":"pwd","Value":"$PASSWORD"},{"Name":"digestPassword","Value":false},{"Name":"roles","Value":[{"db":"rs_$DB_NAME","role":"readWrite"}]}])
[2017/02/15 13:33:36.381] [.error] [cm/executor/executor.go:ExecutePlan:184] <rs_$DB_NAME> [13:33:36.381] Postcondition failed for step AdjustUsers because
[The value of 'currentState.UsersRight' = false, but it should be true]. Outcome=3
摘要(操作管理員通過HTTP談到與Autoamtion代理在管理虛擬機,自動化代理,然後與原協議的mongod講)主要工作正常,但WriteConcern大多數超時,阻止索引構建。