[jira] [Updated] (FLUME-3107) When batchSize of sink greater than transactionCapacity of File Channel, Flume can produce endless data

["Ferenc Szabo (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ferenc Szabo updated FLUME-3107:
--------------------------------
    Fix Version/s:     (was: 1.8.0)
                   1.9.0

> When batchSize of sink greater than transactionCapacity of File Channel, Flume can produce endless data
> -------------------------------------------------------------------------------------------------------
>
>                 Key: FLUME-3107
>                 URL: https://issues.apache.org/jira/browse/FLUME-3107
>             Project: Flume
>          Issue Type: Bug
>          Components: File Channel
>    Affects Versions: 1.7.0
>            Reporter: Yongxi Zhang
>             Fix For: 1.9.0
>
>         Attachments: FLUME-3107-0.patch
>
>
> This problem is the similar as it in FLUME-3106.Flume can produce endless data When batchSize of sink greater than transactionCapacity of File Channel, you can try it with the following config:
> {code:xml}
> agent.sources = src1
> agent.sinks = sink1
> agent.channels = ch2
> agent.sources.src1.type = spooldir
> agent.sources.src1.channels = ch2
> agent.sources.src1.spoolDir = /home/kafka/flumeSpooldir
> agent.sources.src1.fileHeader = false
> agent.sources.src1.batchSize = 5
> agent.channels.ch2.type=file
> agent.channels.ch2.capacity=100
> agent.channels.ch2.checkpointDir=/home/kafka/flumefilechannel/checkpointDir
> agent.channels.ch2.dataDirs=/home/kafka/flumefilechannel/dataDirs
> agent.channels.ch2.transactionCapacity=5
> agent.sinks.sink1.type = hdfs
> agent.sinks.sink1.channel = ch2
> agent.sinks.sink1.hdfs.path = hdfs://kafka1:9000/flume/
> agent.sinks.sink1.hdfs.rollInterval=1
> agent.sinks.sink1.hdfs.fileType = DataStream
> agent.sinks.sink1.hdfs.writeFormat = Text
> agent.sinks.sink1.hdfs.batchSize = 10
> {code}
> Exceptions like this:
> {code:xml}
> 17/06/09 17:16:18 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
> org.apache.flume.EventDeliveryException: org.apache.flume.ChannelException: Take list for FileBackedTransaction, capacity 5 full, consider
> committing more frequently, increasing capacity, or increasing thread count. [channel=ch2]
>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flume.ChannelException: Take list for FileBackedTransaction, capacity 5 full, consider committing more frequently, in
> creasing capacity, or increasing thread count. [channel=ch2]
>         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:531)
>         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:362)
>         ... 3 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)