我試圖在PBS集羣上運行GNU Parallel的許多小型串行作業,每個計算節點有16個核心,因爲我打算使用多個計算節點,因此我將選項-S $ SERVERNAME傳遞給GNUParallel,但是讓我困惑的是是使用-S $SERVERNAME
不等於我指定的就業人數時,我打算產卵超過9個工作啓動的節點上作業的數量,下面是我的意見:「在運行最大作業數」不等於在遠程服務器上使用GNU Parallel時指定的作業數量?
[[email protected] ~]$ parallel --version
GNU parallel 20160922
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.
Web site: http://www.gnu.org/software/parallel
When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.
[[email protected] ~]$ hostname # this shows my hostname
shelob001
使用時GNUParallel作爲本地主機沒有-SERVERNAME,沒有問題,我打算派生10個工作,並且GNUParallel開始了10個工作:
[[email protected] ~]$ parallel --progress echo ::: `seq 1 10`
Computers/CPU cores/Max jobs to run
1:local/16/10 # 10 jobs spawned, no problem
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
local:10/0/100%/0.0s 1
local:9/1/100%/0.0s 2
local:8/2/100%/0.0s 3
local:7/3/100%/0.0s 4
local:6/4/100%/0.0s 5
local:5/5/100%/0.0s 6
local:4/6/100%/0.0s 7
local:3/7/100%/0.0s 8
local:2/8/100%/0.0s 9
local:1/9/100%/0.0s 10
local:0/10/100%/0.0s
當我使用GNUParallel來使用-S $SERVERNAME
產生少於10個作業時,仍然沒有問題。
[[email protected] ~]$ parallel -S shelob001 --progress echo ::: `seq 1 1`
Computers/CPU cores/Max jobs to run
1:shelob001/16/1 # When the number of jobs is less than 10, no problem
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
shelob001:1/0/100%/0.0s 1
shelob001:0/1/100%/1.0s
[[email protected] ~]$ parallel -S shelob001 --progress echo ::: `seq 1 8`
Computers/CPU cores/Max jobs to run
1:shelob001/16/8 # When the number of jobs is less than 10, no problem
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
shelob001:8/0/100%/0.0s 1
shelob001:7/1/100%/1.0s 7
shelob001:6/2/100%/0.5s 3
shelob001:5/3/100%/0.3s 8
shelob001:4/4/100%/0.2s 5
shelob001:3/5/100%/0.2s 2
shelob001:2/6/100%/0.2s 6
shelob001:1/7/100%/0.1s 4
shelob001:0/8/100%/0.1s
[[email protected] ~]$ parallel -S shelob001 --progress echo ::: `seq 1 9`
Computers/CPU cores/Max jobs to run
1:shelob001/16/9 # When the number of jobs is less than 10, no problem
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
shelob001:9/0/100%/0.0s 1
shelob001:8/1/100%/1.0s 5
shelob001:7/2/100%/0.5s 8
shelob001:6/3/100%/0.3s 2
shelob001:5/4/100%/0.2s 6
shelob001:4/5/100%/0.2s 9
shelob001:3/6/100%/0.2s 3
shelob001:2/7/100%/0.1s 4
shelob001:1/8/100%/0.1s 7
shelob001:0/9/100%/0.1s
這裏是混淆了我,當我嘗試使用作業號> = 10,就業人數產生了比想總是少了一個,在這裏我要產卵10,纔開始9個作業:
[[email protected] ~]$ parallel -S shelob001 --progress echo ::: `seq 1 10` # I want to start 10 jobs
Computers/CPU cores/Max jobs to run
1:shelob001/16/9 #why here "Max jobs to run" is 9?
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
shelob001:9/0/100%/0.0s 2
shelob001:9/1/100%/3.0s 1
shelob001:8/2/100%/1.5s 7
shelob001:7/3/100%/1.0s 4
shelob001:6/4/100%/0.8s 9
shelob001:5/5/100%/0.6s 8
shelob001:4/6/100%/0.5s 3
shelob001:3/7/100%/0.4s 5
shelob001:2/8/100%/0.4s 6
shelob001:1/9/100%/0.4s 10
shelob001:0/10/100%/0.4s
[[email protected] ~]$ parallel -S shelob001 --progress echo ::: `seq 1 11`
Computers/CPU cores/Max jobs to run
1:shelob001/16/10 # it seems the jobs started is one less than I specified
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
shelob001:10/0/100%/0.0s 1
shelob001:10/1/100%/3.0s 2
shelob001:9/2/100%/1.5s 8
shelob001:8/3/100%/1.0s 3
shelob001:7/4/100%/0.8s 4
shelob001:6/5/100%/0.6s 5
shelob001:5/6/100%/0.5s 7
shelob001:4/7/100%/0.4s 10
shelob001:3/8/100%/0.4s 9
shelob001:2/9/100%/0.3s 6
shelob001:1/10/100%/0.4s 11
shelob001:0/11/100%/0.4s
[[email protected] ~]$
我使用「top」檢查了計算節點的狀態,但它表明只有9個Cpus在我使用seq 1 10
時使用。希望我已經明確了我的問題,有誰能指出這個問題的可能原因?任何建議都是值得歡迎的。
非常感謝!
感謝Ole,不過看起來-j + 1沒有幫助: –
[fchen14 @ shelob001〜] $ parallel -j + 1 -S 16/shelob001 --progress echo :::'seq 1臺10' 電腦/ CPU核心/最大作業運行 1:shelob001/16/9 電腦:工作完成運行啓動的作業/平均秒/%完成 shelob001 /職位:9/0/100% /0.0s 1 shelob001:9/1/100%/ 2.0s 2 shelob001:8/2/100%/ 1.0s 9 shelob001:7/3/100%/ 0.7s 7 shelob001:6/4/100%/ 0.5s 3 shelob001:5/5/100%/ 0.4s 8 shelob001:4/6/100%/ 0.3s 5 shelob001:3/7/100%/ 0.3s 4 shelob001:2/8/100%/ 0.2s 6 shelob001:1/9/100%/ 0.3s 10 shelob001:0/10/100%/0.3小號 –