flume hdfs rollSize not working in multi channels and multi sinks

[Gyeong Deok Kwon <kk...@gmail.com>]

First, sorry about not good at English. I am trying to use Flume-ng to grab
128MB of log information and put it into a file in HDFS. But HDFS rolling
options not working. Flume-ng send log file per seconds. How can I fix
flume.conf file?

my flume.conf looks below:

agent01.sources = avroGenSrc
agent01.channels = memoryChannel hdfsChannel
agent01.sinks = fileSink hadoopSink

# For each one of the sources, the type is defined
agent01.sources.avroGenSrc.type = avro
agent01.sources.avroGenSrc.bind = dev-hadoop03.ncl
agent01.sources.avroGenSrc.port = 3333

# The channel can be defined as follows.
agent01.sources.avroGenSrc.channels = memoryChannel hdfsChannel

# Each sink's type must be defined
agent01.sinks.fileSink.type = file_roll
agent01.sinks.fileSink.sink.directory = /home1/irteam/flume/data
agent01.sinks.fileSink.sink.rollInterval = 3600
agent01.sinks.fileSink.sink.batchSize = 100

#Specify the channel the sink should use
agent01.sinks.fileSink.channel = memoryChannel



agent01.sinks.hadoopSink.type = hdfs
agent01.sinks.hadoopSink.hdfs.useLocalTimeStamp = true
agent01.sinks.hadoopSink.hdfs.path =
hdfs://dev-hadoop04.ncl:9000/user/hive/warehouse/raw_logs/year=%Y/month=%m/day=%d
agent01.sinks.hadoopSink.hdfs.filePrefix = AccessLog.%Y-%m-%d.%Hh
agent01.sinks.hadoopSink.hdfs.fileType = DataStream
agent01.sinks.hadoopSink.hdfs.writeFormat = Text
agent01.sinks.hadoopSink.hdfs.rollInterval = 0
agent01.sinks.hadoopSink.hdfs.rollSize = 134217728
agent01.sinks.hadoopSink.hdfs.rollCount = 0

#Specify the channel the sink should use
agent01.sinks.hadoopSink.channel = hdfsChannel


# Each channel's type is defined.
agent01.channels.memoryChannel.type = memory
agent01.channels.hdfsChannel.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent01.channels.memoryChannel.capacity = 100000
agent01.channels.memoryChannel.transactionCapacity = 10000

agent01.channels.hdfsChannel.capacity = 100000
agent01.channels.hdfsChannel.transactionCapacity = 10000