錯誤與卡桑德拉

Question

早安批量加載，

我試圖使用大容量裝載（http://www.datastax.com/dev/blog/bulk-loading）爲指導，以實現海量數據的轉儲卡桑德拉例子。

在該示例解決與腳本（http://www.datastax.com/wp-content/uploads/2011/08/DataImport）的依賴關係，但我發現cassandra庫不包含在這裏列出的目錄中的依賴項，因爲我正在與cassandra 2.0一起使用dse。那麼試圖覆蓋這樣的依賴關係獲得以下腳本。

#!/bin/sh 

# paths to the cassandra source tree, cassandra jar and java 

CASSANDRA_HOME="/usr/share/dse/cassandra" 
# CASSANDRA_JAR="./apache-cassandra-2.0.10.jar" 
JAVA=`which java` 

# Java classpath. Must include: 
# - directory of DataImportExample 
# - directory with cassandra/log4j config files 
# - cassandra jar 
# - cassandra depencies jar 
CLASSPATH=".:/usr/share/dse/dse.jar:./slf4j-1.7.7/slf4-nop-1.7.7.jar:./slf4j-1.7.7/slf4j-simple-1.7.7.jar:/etc/dse/cassandra" 

for jar in $CASSANDRA_HOME/lib/*.jar; do 
    CLASSPATH=$CLASSPATH:$jar 
done 

$JAVA -ea -cp $CLASSPATH -Xmx256M \ 
     -Dlog4j.configuration=log4j-tools.properties \ 
     CassandraDataBulk "$@" 

• CASSANDRA_JAR被註釋掉，我用 「卡桑德拉 - 全2.0.8.39.jar」 所在的文件夾中的 「在/ usr /共享/ DSE /卡桑德拉/ lib目錄」，並已被包括在內。

• 我解決了在1.7.7版本下載slf4j依賴關係。

由於cassandra版本的不同，我還不得不習慣使用SSTableSimpleUnsortedWriter構建器。

IPartitioner partitioner = new RandomPartitioner(); 

     SSTableSimpleUnsortedWriter sourcesWriter = new SSTableSimpleUnsortedWriter(
       directory, 
       partitioner, 
       keyspace, 
       table, 
       AsciiType.instance, 
       null, 
       64 
     ); 

看來現在的問題是仍然存在依賴關係。 下，我得到的跟蹤錯誤。

有一個依賴，但它似乎是「org.apache.commons.configuration.ConfigurationRuntimeException」真正的問題可能是另一個，可能有一個不好的配置「cassandra.yaml」？

謝謝， 問候！

[dmdb@vm-dmdb01 ~]$ ./init_env.sh export.csv 
[main] ERROR org.apache.cassandra.cql3.QueryProcessor - Unable to initialize MemoryMeter (jamm not specified as javaagent). This means Cassandra will be unable to measure object sizes accurately and may consequently OOM. 
[main] INFO org.apache.cassandra.config.YamlConfigurationLoader - Loading settings from file:/etc/dse/cassandra/cassandra.yaml 
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - Data files directories: [/data01, /data02] 
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - Commit log directory: /datatmp/commitlog 
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap 
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - disk_failure_policy is stop 
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - commit_failure_policy is stop 
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - Global memtable threshold is enabled at 61MB 
[main] INFO com.datastax.bdp.snitch.Workload - Setting my workload to Cassandra 
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationRuntimeException 
    at com.datastax.bdp.config.ConfigUtil.defaultValue(ConfigUtil.java:18) 
    at com.datastax.bdp.config.DseConfig.<clinit>(DseConfig.java:51) 
    at com.datastax.bdp.snitch.DseDelegateSnitch.<init>(DseDelegateSnitch.java:42) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 
    at java.lang.Class.newInstance(Class.java:374) 
    at org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:488) 
    at org.apache.cassandra.config.DatabaseDescriptor.createEndpointSnitch(DatabaseDescriptor.java:508) 
    at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:341) 
    at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:111) 
    at org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.<init>(AbstractSSTableSimpleWriter.java:50) 
    at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter.<init>(SSTableSimpleUnsortedWriter.java:96) 
    at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter.<init>(SSTableSimpleUnsortedWriter.java:80) 
    at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter.<init>(SSTableSimpleUnsortedWriter.java:91) 
    at CassandraDataBulk.main(CassandraDataBulk.java:35) 
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.ConfigurationRuntimeException 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358) 
    ... 17 more 

Answer 1

您在java調用中缺少「javaagent」參數。添加以下內容：

-javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar 

你的最後調用應該是這樣的：

$JAVA -ea -cp $CLASSPATH -Xmx256M \ 
     -Dlog4j.configuration=log4j-tools.properties \ 
     -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar 
     CassandraDataBulk "$@" 

注：調整路徑所必需的jamm.jar

Reference

至於運行時配置錯誤，請下載apache commons'lang'庫並將其包含到您的類路徑中。

Download here

如果收到執行修復後新的異常，下載谷歌-common.jar和番石榴16.0.1.jar，包括他們以及到類路徑中。這些都是我自己的批量加載器到目前爲止所需的所有JAR。