使用內存通道增加文件大小

Question

下面是我的flume配置文件。即使在更改rollInterval和rollSize之後，也只有10個事件正在寫入，控制檯顯示rollCount = 10和events = 10。我也試着將rollCount增加到1000，但輸出沒有變化。任何人都可以建議增加正在寫入hdfs的文件大小。下面的conf文件有什麼問題？

#naming components 

NetAgent.sources = NetCat_1 NetCat_2 
NetAgent.sinks = HDFS 
NetAgent.channels = MemChannel 


NetAgent.sources.NetCat_1.type = netcat 
NetAgent.sources.NetCat_1.bind = localhost 
NetAgent.sources.NetCat_1.port = 8671 

NetAgent.sources.NetCat_2.type = netcat 
NetAgent.sources.NetCat_2.bind = localhost 
NetAgent.sources.NetCat_2.port = 8672 


NetAgent.sinks.HDFS.type = hdfs 
NetAgent.sinks.HDFS.hdfs.path = file path here 
NetAgent.sinks.HDFS.hdfs.filePrefix = test 
NetAgent.sinks.HDFS.hdfs.rollSize = 67108864 
NetAgent.sinks.HDFS.hdfs.rollInterval = 3600 
NetAgent.sinks.HDFS.rollCount = 0 
NetAgent.sinks.HDFS.hdfs.batchSize = 10000 
NetAgent.sinks.HDFS.hdfs.writeFormat = Text 
NetAgent.sinks.HDFS.hdfs.fileType = DataStream 


NetAgent.channels.MemChannel.type = memory 
NetAgent.channels.MemChannel.capacity = 20000 
NetAgent.channels.MemChannel.transactionCapacity = 20000 


NetAgent.sources.NetCat_1.channels = MemChannel 
NetAgent.sources.NetCat_2.channels = MemChannel 
NetAgent.sinks.HDFS.channel = MemChannel 

控制檯日誌爲

(SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUg-org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java)] 
rolling: rollCount: 10, events: 10 

the image shows the files written in HDFS

Answer 1

你忘了HDFS添加到您的rollCount配置。它使用默認值10，因爲它沒有看到您的配置。請注意，您的HDFS的配置是：

NetAgent.sinks.HDFS.type = hdfs 
NetAgent.sinks.HDFS.hdfs.rollSize = 67108864 
NetAgent.sinks.HDFS.hdfs.rollInterval = 3600 
NetAgent.sinks.HDFS.rollCount = 0 
NetAgent.sinks.HDFS.hdfs.batchSize = 10000 
NetAgent.sinks.HDFS.hdfs.writeFormat = Text 
NetAgent.sinks.HDFS.hdfs.fileType = DataStream 

在rollCount線，它需要：

NetAgent.sinks.HDFS.hdfs.rollCount = 0 

這將覆蓋默認rollCount和您的水槽代理人的行爲，你如何想它。