0
我們正在運行Spark流作業從Kafka讀取，轉換爲csv，然後寫入Hbase。我正在使用API CSVBulkLoad來運行批量加載作業。火花作業開始罰款，轉換爲CSV但csvBulkLoad.run（）開始一個新的MR作業失敗與此異常Apache Phoenix 4.7 csvBulkLoad.run（）拋出ClassNotFoundException
Error: java.lang.ClassNotFoundException: org.apache.commons.csv.CSVFormat
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.phoenix.mapreduce.CsvToKeyValueMapper$CsvLineParser.<init>(CsvToKeyValueMapper.java:96)
at org.apache.phoenix.mapreduce.CsvToKeyValueMapper.setup(CsvToKeyValueMapper.java:69)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
這是我的火花作業命令
spark-submit --jars $(echo lib/*.jar | tr ' ' ',') --class "com.lam.app.Driver" --name "MyBulkLoader" --master yarn --deploy-mode cluster --driver-class-path $(echo lib/*.jar | tr ' ' ',') --driver-memory 4g --executor-memory 1g --num-executors 2 --executor-cores 2 --files conf.properties PipeLine-1.0.jar
早些時候我曾經得到另一個關於丟失類（java.lang.NoClassDefFoundError：com/yammer/metrics/core/MetricsRegistry）的錯誤，但在包含「--driver-class-path $（echo lib/*。jar | tr' ''，'）「。現在我們得到了關於缺少CSVFormat類的錯誤。
我的lib目錄中包含所有需要的jar文件，包括commons-csv jar，但仍然出現此錯誤。我使用maven來構建項目。 下面是運行csvbulkload
CsvBulkLoadTool csvBulkLoadTool = new CsvBulkLoadTool();
final org.apache.hadoop.conf.Configuration conf = HBaseConfiguration.create();
System.out.println("ZKQuorum " + this.ZKQUORUM);
conf.set(HConstants.ZOOKEEPER_QUORUM, this.ZKQUORUM);
conf.set("hbase.zookeeper.property.clientPort", String.valueOf(2181));
conf.set("zookeeper.znode.parent", "/hbase-unsecure");
csvBulkLoadTool.setConf(conf);
int exitCode = csvBulkLoadTool.run(new String[] { "--input", "\"" + this.hdfsInputFile + "\"",
"--table", this.TABLENAME, "--zookeeper", this.ZKQUORUM + ":2181:/hbase-unsecure" });
System.out.println("Return code of WDL bulk load execution " + exitCode);
我退出碼的代碼段爲-1始終。請讓我知道如果我失去了我的課程路徑