引起︰java.lang.ClassNotFoundException︰org.jets3t.service.ServiceException

Question

我的代碼應該訪問一些存儲在S3上的文件（這個代碼在一臺機器上工作正常，而在另一臺機器上失敗;基本上它失敗了，當它被本地執行從IntelliJ IDEA的（而不是集羣））：

sc.hadoopConfiguration.set("fs.s3n.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem") 
sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "xxx") 
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "xxx") 

val sqlContext = new SQLContext(sc) 

var df = sqlContext.read.json("s3n://myPath/*.json") 

我得到在該行var df = sqlContext.read.json("s3n://myPath/*.json")以下錯誤：

Caused by: java.lang.ClassNotFoundException: org.jets3t.service.ServiceException 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 

我閱讀了有關這個問題類似的線程，它被提及在使用Spark 1.6.2的情況下，解決方案是使用org.apache.hadoop hadoop-aws 2.6.0。在我的情況下，它並沒有解決問題。

我pom.xml（摘錄自）：

<properties> 
     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> 
     <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> 

     <java.version>1.8</java.version> 
     <scala.version>2.10.6</scala.version> 
     <spark.version>1.6.2</spark.version> 
     <jackson.version>2.8.3</jackson.version> 
    </properties> 

<dependencies> 
     <dependency> 
      <groupId>org.scala-lang</groupId> 
      <artifactId>scala-library</artifactId> 
      <version>${scala.version}</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-streaming_2.10</artifactId> 
      <!--<scope>provided</scope>--> 
      <version>${spark.version}</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-streaming-kafka_2.10</artifactId> 
      <version>${spark.version}</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-sql_2.10</artifactId> 
      <!--<scope>provided</scope>--> 
      <version>${spark.version}</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-mllib_2.10</artifactId> 
      <version>${spark.version}</version> 
     </dependency> 
     <dependency> 
      <groupId>com.fasterxml.jackson.module</groupId> 
      <artifactId>jackson-module-scala_2.10</artifactId> 
      <version>${jackson.version}</version> 
     </dependency> 
     <dependency> 
      <groupId>com.fasterxml.jackson.core</groupId> 
      <artifactId>jackson-databind</artifactId> 
      <version>${jackson.version}</version> 
     </dependency> 
     <dependency> 
      <groupId>com.fasterxml.jackson.core</groupId> 
      <artifactId>jackson-annotations</artifactId> 
      <version>${jackson.version}</version> 
     </dependency> 
     <dependency> 
      <groupId>com.fasterxml.jackson.core</groupId> 
      <artifactId>jackson-core</artifactId> 
      <version>${jackson.version}</version> 
     </dependency> 
     <dependency> 
      <groupId>com.lambdaworks</groupId> 
      <artifactId>jacks_2.10</artifactId> 
      <version>2.3.3</version> 
     </dependency> 
     <dependency> 
      <groupId>com.typesafe</groupId> 
      <artifactId>config</artifactId> 
      <version>1.3.1</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.hadoop</groupId> 
      <artifactId>hadoop-aws</artifactId> 
      <version>2.6.0</version> 
     </dependency> 
     <dependency> 
      <groupId>com.amazonaws</groupId> 
      <artifactId>aws-java-sdk-s3</artifactId> 
      <version>1.11.53</version> 
     </dependency> 
     <dependency> 
      <groupId>net.debasishg</groupId> 
      <artifactId>redisclient_2.10</artifactId> 
      <version>3.3</version> 
     </dependency> 
    </dependencies> 

Answer 1

添加在dependency以下應該解決這一問題

<dependency> 
    <groupId>org.apache.hadoop</groupId> 
    <artifactId>hadoop-client</artifactId> 
    <version>2.6.0</version> 
</dependency> 

我希望這有助於