Spark度量標準：如何訪問執行者和工作者數據？

Question

注：我使用的Spark紗線

我一直在嘗試在星火實施Metric System。我啓用了ConsoleSink和CsvSink，併爲所有四個實例（驅動程序，主設備，執行程序，工作程序）啓用了JvmSource。但是，我只有驅動程序輸出，並且在控制檯和csv目標目錄中沒有worker/executor/master數據。

在閱讀this question之後，我不知道在提交工作時是否需要向執行人發貨。

我提交命令： ./bin/spark-submit --class org.apache.spark.examples.SparkPi lib/spark-examples-1.5.0-hadoop2.6.0.jar 10

貝婁是我metric.properties文件：

# Enable JmxSink for all instances by class name 
*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink 

# Enable ConsoleSink for all instances by class name 
*.sink.console.class=org.apache.spark.metrics.sink.ConsoleSink 

# Polling period for ConsoleSink 
*.sink.console.period=10 

*.sink.console.unit=seconds 

####################################### 
# worker instance overlap polling period 
worker.sink.console.period=5 

worker.sink.console.unit=seconds 
####################################### 

# Master instance overlap polling period 
master.sink.console.period=15 

master.sink.console.unit=seconds 

# Enable CsvSink for all instances 
*.sink.csv.class=org.apache.spark.metrics.sink.CsvSink 
#driver.sink.csv.class=org.apache.spark.metrics.sink.CsvSink 

# Polling period for CsvSink 
*.sink.csv.period=10 

*.sink.csv.unit=seconds 

# Polling directory for CsvSink 
*.sink.csv.directory=/opt/spark-1.5.0-bin-hadoop2.6/csvSink/ 

# Worker instance overlap polling period 
worker.sink.csv.period=10 

worker.sink.csv.unit=second 

# Enable Slf4jSink for all instances by class name 
#*.sink.slf4j.class=org.apache.spark.metrics.sink.Slf4jSink 

# Polling period for Slf4JSink 
#*.sink.slf4j.period=1 

#*.sink.slf4j.unit=minutes 


# Enable jvm source for instance master, worker, driver and executor 
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource 

worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource 

driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource 

executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource 

這裏是由星火創建的CSV文件的列表。我期待着爲Spark執行者（也是JVM）訪問相同的數據。

app-20160812135008-0013.driver.BlockManager.disk.diskSpaceUsed_MB.csv 
app-20160812135008-0013.driver.BlockManager.memory.maxMem_MB.csv 
app-20160812135008-0013.driver.BlockManager.memory.memUsed_MB.csv 
app-20160812135008-0013.driver.BlockManager.memory.remainingMem_MB.csv 
app-20160812135008-0013.driver.jvm.heap.committed.csv 
app-20160812135008-0013.driver.jvm.heap.init.csv 
app-20160812135008-0013.driver.jvm.heap.max.csv 
app-20160812135008-0013.driver.jvm.heap.usage.csv 
app-20160812135008-0013.driver.jvm.heap.used.csv 
app-20160812135008-0013.driver.jvm.non-heap.committed.csv 
app-20160812135008-0013.driver.jvm.non-heap.init.csv 
app-20160812135008-0013.driver.jvm.non-heap.max.csv 
app-20160812135008-0013.driver.jvm.non-heap.usage.csv 
app-20160812135008-0013.driver.jvm.non-heap.used.csv 
app-20160812135008-0013.driver.jvm.pools.Code-Cache.committed.csv 
app-20160812135008-0013.driver.jvm.pools.Code-Cache.init.csv 
app-20160812135008-0013.driver.jvm.pools.Code-Cache.max.csv 
app-20160812135008-0013.driver.jvm.pools.Code-Cache.usage.csv 
app-20160812135008-0013.driver.jvm.pools.Code-Cache.used.csv 
app-20160812135008-0013.driver.jvm.pools.Compressed-Class-Space.committed.csv 
app-20160812135008-0013.driver.jvm.pools.Compressed-Class-Space.init.csv 
app-20160812135008-0013.driver.jvm.pools.Compressed-Class-Space.max.csv 
app-20160812135008-0013.driver.jvm.pools.Compressed-Class-Space.usage.csv 
app-20160812135008-0013.driver.jvm.pools.Compressed-Class-Space.used.csv 
app-20160812135008-0013.driver.jvm.pools.Metaspace.committed.csv 
app-20160812135008-0013.driver.jvm.pools.Metaspace.init.csv 
app-20160812135008-0013.driver.jvm.pools.Metaspace.max.csv 
app-20160812135008-0013.driver.jvm.pools.Metaspace.usage.csv 
app-20160812135008-0013.driver.jvm.pools.Metaspace.used.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Eden-Space.committed.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Eden-Space.init.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Eden-Space.max.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Eden-Space.usage.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Eden-Space.used.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Old-Gen.committed.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Old-Gen.init.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Old-Gen.max.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Old-Gen.usage.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Old-Gen.used.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Survivor-Space.committed.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Survivor-Space.init.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Survivor-Space.max.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Survivor-Space.usage.csv 
app-20160812135008-0013.driver.jvm.pools.PS-Survivor-Space.used.csv 
app-20160812135008-0013.driver.jvm.PS-MarkSweep.count.csv 
app-20160812135008-0013.driver.jvm.PS-MarkSweep.time.csv 
app-20160812135008-0013.driver.jvm.PS-Scavenge.count.csv 
app-20160812135008-0013.driver.jvm.PS-Scavenge.time.csv 
app-20160812135008-0013.driver.jvm.total.committed.csv 
app-20160812135008-0013.driver.jvm.total.init.csv 
app-20160812135008-0013.driver.jvm.total.max.csv 
app-20160812135008-0013.driver.jvm.total.used.csv 
DAGScheduler.job.activeJobs.csv 
DAGScheduler.job.allJobs.csv 
DAGScheduler.messageProcessingTime.csv 
DAGScheduler.stage.failedStages.csv 
DAGScheduler.stage.runningStages.csv 
DAGScheduler.stage.waitingStages.csv 

Answer 1

由於您沒有給出您嘗試過的命令，我假設您沒有傳遞metrics.properties。要通過metrics.propertis的命令應該是

spark-submit <other parameters> --files metrics.properties 
--conf spark.metrics.conf=metrics.properties 

注metrics.properties在--files到指定& --conf，該--files將轉移metrics.properties文件的執行者。既然你可以看到驅動程序的輸出，而不是執行者，我認爲你缺少--files選項。