我想將json/csv加載到hbase

Question

我想將JSON或CSV加載到HBASE中，而不使用任何mapreduce程序以及HIVEQL/pig支持，是否有可能以及哪一種更高效hive-hbase或mapreduce- HBase的。

Answer 1

也許你可以參考批量加載。鏈接在這裏。 bulk loading

Answer 2

我使用Perl腳本來做到這一點;

這是我的（perl的生成）JSON文件

{"c3":"c","c4":"d","c5":"tim","c2":"b","c6":"andrew","c1":"a"},"CURRENTLY20140131":{"c2":"tim2","c1":"bill2"},"THERE20140131"::{"c3":"c","c4":"d","c9":"bill2","c10":"tim2","c2":"b","c6":"andrew","c7":"bill","c5":"tim","c1":"a","c8":"tom"},"TODAY20140131":{"c2":"bill","c1":"tom"}} 

我分片上的繩子，用多列取決於誰/什麼引用的關鍵對象。

use strict; 
use warnings; 
use Data::Dumper; 

use JSON::XS qw(encode_json decode_json); 
use File::Slurp qw(read_file write_file); 

my %words =(); 
my $debug = 0; 

sub ReadHash { 
    my ($filename) = @_; 
    my $json = read_file($filename, { binmode => ':raw' }); 
    %words = %{ decode_json $json }; 
} 


# Main Starts here 
ReadHash("Save.json"); 
foreach my $key (keys %words) 
    { 
     printf("put 'test', '$key',"); 
     my $cnt=0; 
     foreach my $key2 (keys %{ $words{$key} }) { 
      my $val = $words{$key}{$key2}; 
     print "," if $cnt>0; 
      printf("'cf:$key2', '$val'"); 
     ++$cnt; 
    } 
     print "\n"; 
} 

生成Hbase命令，然後執行它們。

另外 - 我會看看happybase（Python），它也很快加載大型數據集。

希望這有助於

這應該產生這樣的輸出.....

put 'test', 'WHERE20140131','cf:c2', 'bill2','cf:c1', 'tim2' 
put 'test', 'OMAN20140131','cf:c3', 'c','cf:c4', 'd','cf:c5', 'tim','cf:c2', 'b','cf:c1', 'a','cf:c6', 'andrew' 
put 'test', 'CURRENTLY20140131','cf:c2', 'tim2','cf:c1', 'bill2'