如何設置Hbase表列家族的列族大小？

我想從CSV文件導入數據到HBase表中。但是我在導入時遇到了下面顯示的異常。如何設置Hbase表列家族的列族大小？

Error: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit. 
     at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110) 
     at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755) 
     at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701) 
     at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue.<init>(ClientProtos.java:8599) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue.<init>(ClientProtos.java:8563) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue$1.parsePartialFrom(ClientProtos.java:8672) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue$1.parsePartialFrom(ClientProtos.java:8667) 
     at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:8462) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:8404) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:8498) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:8493) 
     at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:7959) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:7890) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:8045) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:8040) 
     at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200) 
     at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) 
     at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) 
     at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) 
     at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) 
     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto.parseDelimitedFrom(ClientProtos.java:10468) 
     at org.apache.hadoop.hbase.mapreduce.MutationSerialization$MutationDeserializer.deserialize(MutationSerialization.java:60) 
     at org.apache.hadoop.hbase.mapreduce.MutationSerialization$MutationDeserializer.deserialize(MutationSerialization.java:50) 
     at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:146) 
     at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:121) 
     at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:302) 
     at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:170) 
     at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1651) 
     at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611) 
     at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1462) 
     at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700) 
     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770) 
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) 
     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:415) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) 
     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

我認爲這與數據長度高於默認大小有關。如何通過終端增加HBase列的列族大小？任何幫助將非常感激。

來源

2017-01-27 mayooran

要改變列族塊大小

alter 'my_table', {NAME => 'my_cf', BLOCKSIZE => '1048756'}

然後調用describe 'my_table'看錶的元信息並驗證它的工作。

來源

2017-01-27 23:04:23 AdamSkywalker

謝謝你，我可以通過這種方式設置塊大小。但是我仍然看到了上面所示的例外情況。還有其他尺寸需要改變嗎？有任何想法嗎？ :( – mayooran

@mayooran似乎是默認的64Mb限制。這裏是關於同一主題的問題http://stackoverflow.com/questions/31158900/change-hbase-size-limit – AdamSkywalker

它說，版本1.02修復了這個問題。我使用的是0.98。猜猜必須更新。謝謝！:) – mayooran

如何設置Hbase表列家族的列族大小？

回答

相關問題