入門失敗使用SparkContext.setLogLevel

Question

我使用sparkContext.setLogLevel("ERROR")改變了我的SparkContext的日誌級別，當我運行該程序，我得到時加載類「org.slf4j.impl.StaticLoggerBinder」：

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". 
SLF4J: Defaulting to no-operation (NOP) logger implementation 
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details 

一切仍然照常運行，但我想知道這是從哪裏來的，因爲我沒有手動插入任何SLF4J依賴項。 我的POM依賴部分看起來是這樣（其中scala.version是2.11.8和scala.binaty.version是2.11）：

<dependencies> 
    <dependency> 
     <groupId>org.scala-lang</groupId> 
     <artifactId>scala-library</artifactId> 
     <version>${scala.version}</version> 
     <scope>provided</scope> 
    </dependency> 
    <dependency> 
     <groupId>org.scala-lang</groupId> 
     <artifactId>scala-compiler</artifactId> 
     <version>${scala.version}</version> 
     <scope>provided</scope> 
    </dependency> 
    <dependency> 
     <groupId>org.scalatest</groupId> 
     <artifactId>scalatest_${scala.binary.version}</artifactId> 
     <version>2.2.1</version> 
     <scope>test</scope> 
    </dependency> 
    <dependency> 
     <groupId>com.amazonaws</groupId> 
     <artifactId>aws-java-sdk</artifactId> 
     <version>1.10.11</version> 
     <scope>provided</scope> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_${scala.binary.version}</artifactId> 
     <version>${spark.version}</version> 
     <scope>provided</scope> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_${scala.binary.version}</artifactId> 
     <version>${spark.version}</version> 
     <scope>provided</scope> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-hive_${scala.binary.version}</artifactId> 
     <version>${spark.version}</version> 
     <scope>provided</scope> 
    </dependency> 
    <dependency> 
     <groupId>joda-time</groupId> 
     <artifactId>joda-time</artifactId> 
     <version>2.9.4</version> 
    </dependency> 
    <dependency> 
     <groupId>org.scalaj</groupId> 
     <artifactId>scalaj-http_${scala.binary.version}</artifactId> 
     <version>2.3.0</version> 
    </dependency> 
    <dependency> 
     <groupId>net.liftweb</groupId> 
     <artifactId>lift-json_${scala.binary.version}</artifactId> 
     <version>3.0-M8</version> 
    </dependency> 
    <dependency> 
     <groupId>commons-codec</groupId> 
     <artifactId>commons-codec</artifactId> 
     <version>1.9</version> 
    </dependency> 
</dependencies> 

任何想法是什麼導致了這一點，我怎麼解決？

Answer 1

我猜你的一個依賴關係有一個特定版本的slf4j，它覆蓋了Spark所使用的版本，從而導致該錯誤。

我建議你明確地把slf4j依賴於你的pom.xml

例如：

<properties> 
    <org.slf4j.version>1.7.5</org.slf4j.version> 
</properties> 

<dependencies> 
    <!-- Logging --> 
    <dependency> 
     <groupId>org.slf4j</groupId> 
     <artifactId>slf4j-api</artifactId> 
     <version>${org.slf4j.version}</version> 
    </dependency> 
    <dependency> 
     <groupId>org.slf4j</groupId> 
     <artifactId>jcl-over-slf4j</artifactId> 
     <version>${org.slf4j.version}</version> 
     <scope>runtime</scope> 
    </dependency> 
    <dependency> 
     <groupId>org.slf4j</groupId> 
     <artifactId>slf4j-log4j12</artifactId> 
     <version>${org.slf4j.version}</version> 
     <scope>runtime</scope> 
    </dependency> 
</dependencies> 

另外，請確保您有它保存在你的classpath有效log4j.xml文件，並嘗試在xml中設置Spark的日誌級別而不是使用代碼。

例如：

<?xml version="1.0" encoding="UTF-8" ?> 
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> 

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/"> 
    <appender name="console" class="org.apache.log4j.ConsoleAppender"> 
    <param name="Target" value="System.out"/> 
    <layout class="org.apache.log4j.PatternLayout"> 
     <param name="ConversionPattern" value="%-5p %c{1} - %m%n"/> 
    </layout> 
    </appender> 

    <logger name="org.apache.spark"> 
    <level value="error"/> 
    </logger> 

    <logger name="org.spark-project"> 
    <level value="error"/> 
    </logger> 

    <logger name="org.spark-project"> 
    <level value="error"/> 
    </logger> 

    <root> 
    <priority value ="debug" /> 
    <appender-ref ref="console" /> 
    </root> 

</log4j:configuration>