從現有快照文件創建配置單元表

Question

我是Hadoop和Hive的新手。我正在與hadoop 2.7.0和hive 1.1.1。

我有一些快速格式的文件。我已將它們複製到我的hdfs中。我想知道： 1.如何使用hdfs中的這些文件創建配置單元表。 2.如何配置hadoop使用snappy文件。

EDIT：我嘗試以下：

1.modified core-site.xml：

<property> 
    <name>io.compression.codecs</name> 
    <value>org.apache.hadoop.io.compress.GzipCodec, 
      org.apache.hadoop.io.compress.DefaultCodec, 
      org.apache.hadoop.io.compress.SnappyCodec 
    </value> 
</property> 

mapred-site.xml 2.Modified：

<property> 
    <name>mapreduce.map.output.compress</name> 
    <value>true</value> 
</property> 

<property> 
    <name>mapred.map.output.compress.codec</name> 
    <value>org.apache.hadoop.io.compress.SnappyCodec</value> 
</property> 


<property> 
    <name>mapreduce.admin.user.env</name> 
    <value>LD_LIBRARY_PATH=/Users/8417/repos/hadoop_ecosystem/hadoop-2.7.0/lib/native</value> 
</property> 

安裝時髦brew install snappy。

cp /usr/local/Cellar/snappy/<version>/lib/snappy.* to $HADOOP_HOME/lib/snappy

cp /usr/local/Cellar/snappy/<version>/lib/snappy.* /Library/Java/Home/jre/lib。

6.Added以下環境變量：

export JAVA_LIBRARY_PATH="$HADOOP_HOME/lib/native" 
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native 
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib" 

但我不能使它發揮作用。我收到以下錯誤：

Machine:native user$ hadoop fs -text hive/part-00000.snappy 
15/09/24 19:25:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z 
    at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) 
    at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) 
    at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:192) 
    at org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:178) 
    at org.apache.hadoop.io.compress.CompressionCodec$Util.createInputStreamWithCodecPool(CompressionCodec.java:157) 
    at org.apache.hadoop.io.compress.SnappyCodec.createInputStream(SnappyCodec.java:162) 
    at org.apache.hadoop.fs.shell.Display$Text.getInputStream(Display.java:163) 
    at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:102) 
    at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) 
    at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) 
    at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) 
    at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) 
    at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) 
    at org.apache.hadoop.fs.shell.Command.run(Command.java:165) 
    at org.apache.hadoop.fs.FsShell.run(FsShell.java:287) 
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) 
    at org.apache.hadoop.fs.FsShell.main(FsShell.java:340) 


Machine:native user$ hadoop checknative 
15/09/24 19:41:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
Native library checking: 
hadoop: false 
zlib: false 
snappy: false 
lz4:  false 
bzip2: false 
openssl: false 
15/09/24 19:41:16 INFO util.ExitUtil: Exiting with status 1 

Answer 1

確保您已經安裝了瞬間，你的Hadoop是能夠「看到」它：

hadoop checknative 
... 
snappy: true /usr/lib64/libsnappy.so.1 
... 

另外，還要確保你的文件的擴展名是「.snappy」 。這是非常多的。

啓動蜂巢或直線和創建表：

create external table t1(f1 int, ...) location '/path/to/dir/with/snappy/files';