Spark 1.5和datastax-ddc-3.2.1 Cassandra依賴罐？

Question

我正在使用Spark 1.5和Cassandra 3.2.1。任何人都可以指定構建路徑中需要的確切罐子以連接，查詢和插入數據到Cassandra。

現在我使用的follwing罐子 火花卡桑德拉 - connector_2.10-1.5.0-M3.jar Apache的卡桑德拉 - clientutil-3.2.1.jar 卡桑德拉驅動核心 - 3.0.0 -beta1-bb1bce4-快照shaded.jar 火花組裝1.5.1 hadoop2.0.0-MR1-cdh4.2.0.jar 番石榴18.0.jar 網狀-全4.0.23.Final.jar

有了上述罐子，我可以連接到cassandra。我可以截斷表格，並刪除表格。但是，即使是一個簡單的插入查詢，我也無法插入任何數據。

以下是代碼：

import org.apache.spark.SparkConf; 
import org.apache.spark.api.java.JavaSparkContext; 

import com.datastax.driver.core.Session; 
import com.datastax.spark.connector.cql.CassandraConnector; 

public class Test { 

public static void main(String[] args) { 


    JavaSparkContext ctx = new JavaSparkContext(new SparkConf().setMaster("spark://blr-lt-203:7077").set("spark.cassandra.connection.host", "blr-lt-203").setAppName("testinsert").set("spark.serializer" ,"org.apache.spark.serializer.KryoSerializer").set("spark.kryoserializer.buffer.max" , "1024mb")); 

    CassandraConnector connector = CassandraConnector.apply(ctx.getConf()); 

    Session session = connector.openSession(); 

    session.execute("insert into test.table1 (name) values ('abcd')") ; 
    session.close(); 
    ctx.stop(); 

} 

} 

以下是日誌：

16/03/28 21:24:52 INFO BlockManagerMaster: Trying to register BlockManager 
16/03/28 21:24:52 INFO BlockManagerMasterEndpoint: Registering block manager localhost:50238 with 944.7 MB RAM,BlockManagerId(driver, localhost, 50238) 
16/03/28 21:24:52 INFO BlockManagerMaster: Registered BlockManager 
16/03/28 21:24:53 INFO NettyUtil: Did not find Netty's native epoll transport in the classpath, defaulting to NIO. 
16/03/28 21:24:53 INFO Cluster: New Cassandra host localhost/127.0.0.1:9042 added 
16/03/28 21:24:53 INFO CassandraConnector: Connected to Cassandra cluster: Test Cluster 

它只是在這裏結束了一段時間，然後超時與FOLL例外：

Exception in thread "main" com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive) 

1. 我在做什麼錯？

2. 請讓我知道什麼是所需的罐子或是否存在一些版本兼容性問題。

3. 什麼是火花（1.5）和卡桑德拉（？）的最穩定的版本

由於提前

Answer 1

問題的發生是由於在谷歌的番石榴庫之間的衝突。

解決方案是陰影番石榴庫存在於spark-cassandra-connector依賴項中。你可以通過使用maven shade插件來實現。 這是我的pom.xml來遮蓋番石榴庫。

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
<modelVersion>4.0.0</modelVersion> 

<groupId>com.pc.test</groupId> 
<artifactId>casparktest</artifactId> 
<version>0.0.1-SNAPSHOT</version> 
<packaging>jar</packaging> 

<name>casparktest</name> 
<url>http://maven.apache.org</url> 

<properties> 
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> 
</properties> 

<dependencies> 
<dependency> 
<groupId>org.apache.spark</groupId> 
<artifactId>spark-core_2.10</artifactId> 
<version>1.5.0</version> 
</dependency> 
<dependency> 
    <groupId>junit</groupId> 
    <artifactId>junit</artifactId> 
    <version>3.8.1</version> 
    <scope>test</scope> 
</dependency> 
<dependency> 
    <groupId>com.datastax.spark</groupId> 
    <artifactId>spark-cassandra-connector_2.10</artifactId> 
    <version>1.5.0</version> 
</dependency> 
<dependency> 
<groupId>com.datastax.cassandra</groupId> 
<artifactId>cassandra-driver-core</artifactId> 
<version>3.0.0-beta1</version> 
</dependency> 

</dependencies> 
<build> 

<plugins> 
    <plugin> 
     <groupId>org.apache.maven.plugins</groupId> 
     <artifactId>maven-shade-plugin</artifactId> 
     <version>2.3</version> 
     <executions> 
      <execution> 
       <phase>package</phase> 
       <goals> 
        <goal>shade</goal> 
       </goals> 
       <configuration> 
       <filters> 
    <filter> 
     <artifact>*:*</artifact> 
     <excludes> 
      <exclude>META-INF/*.SF</exclude> 
      <exclude>META-INF/*.DSA</exclude> 
      <exclude>META-INF/*.RSA</exclude> 
     </excludes> 
    </filter> 
</filters> 
        <relocations> 
         <relocation> 
          <pattern>com.google</pattern> 
          <shadedPattern>com.pointcross.shaded.google</shadedPattern> 
         </relocation> 

        </relocations> 
        <minimizeJar>false</minimizeJar> 
        <shadedArtifactAttached>true</shadedArtifactAttached> 
       </configuration> 
      </execution> 
     </executions> 
    </plugin> 
</plugins> 
</build> 

在這之後您做一個Maven構建，這將產生一個罐子在pom.xml也提到使用，你可以提交火花的作業陰影關閉番石榴庫的所有依賴。