如何將Spark輸出重定向到控制檯？

我在CHD集羣上運行spark任務，所有日誌都以文本文件的形式存儲到歷史記錄服務器中。有沒有辦法讓這些輸出在控制檯上打印？我看到的全部是如何將Spark輸出重定向到控制檯？

15/10/21 15:47:09 INFO yarn.Client: Application report for application_1445455790310_0014 (state: ACCEPTED) 
15/10/21 15:47:10 INFO yarn.Client: Application report for application_1445455790310_0014 (state: ACCEPTED) 
15/10/21 15:47:11 INFO yarn.Client: Application report for application_1445455790310_0014 (state: ACCEPTED) 
15/10/21 15:47:12 INFO yarn.Client: Application report for application_1445455790310_0014 (state: ACCEPTED) 
15/10/21 15:47:13 INFO yarn.Client: Application report for application_1445455790310_0014 (state: ACCEPTED) 
15/10/21 15:47:14 INFO yarn.Client: Application report for application_1445455790310_0014 (state: ACCEPTED) 
15/10/21 15:47:15 INFO yarn.Client: Application report for application_1445455790310_0014 (state: ACCEPTED) 
15/10/21 15:47:16 INFO yarn.Client: Application report for application_1445455790310_0014 (state: ACCEPTED) 
15/10/21 15:47:17 INFO yarn.Client: Application report for application_1445455790310_0014 (state: ACCEPTED) 
15/10/21 15:47:18 INFO yarn.Client: Application report for application_1445455790310_0014 (state: ACCEPTED) 
15/10/21 15:47:19 INFO yarn.Client: Application report for application_1445455790310_0014 (state: RUNNING)

來源

2015-10-21 Instinct

火花測井系統與log4j一起使用。

根據官方documentation，Spark提供了一個名爲log4j.properties.template的配置文件，您可以在其中指定不同的日誌記錄屬性。該文件位於主Spark目錄中的文件夾conf下。

爲了讓Spark檢測並使用此配置文件，需要對其進行重命名並從中刪除.template。

默認模板看起來像下面這樣：

# Set everything to be logged to the console 
log4j.rootCategory=INFO, console 
log4j.appender.console=org.apache.log4j.ConsoleAppender 
log4j.appender.console.target=System.err 
log4j.appender.console.layout=org.apache.log4j.PatternLayout 
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n 

# Settings to quiet third party logs that are too verbose 
log4j.logger.org.spark-project.jetty=WARN 
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR 
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO 
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO 
log4j.logger.org.apache.parquet=ERROR 
log4j.logger.parquet=ERROR 

...

在這個例子中，默認的日誌記錄方法設置爲控制檯。雖然我沒有測試過它，但您應該能夠使用此示例模板查看輸出框。

來源

2015-10-22 09:41:34

如何將Spark輸出重定向到控制檯？

回答

相關問題