You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flume.apache.org by "Alan Liu (JIRA)" <ji...@apache.org> on 2018/07/09 06:44:00 UTC
[jira] [Updated] (FLUME-3254) taildir source works abnormally when the monitored file has too fast append speed．
[ https://issues.apache.org/jira/browse/FLUME-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Liu updated FLUME-3254:
----------------------------
Description:
Hello,
I am using flume 1.8.0 for ubuntu 16.04. I met an question when using the agent, taildir-memory-fileroll. I programed for appending data into the file monitored by taildir *every 2 second*．Then at this speed the agent worked abnormally. Although the json file generated by taildir always updated and recorded the newest position, the file generated by rollfile didn't have any more data until I killed the bin which is appending data into the file monitored by taildir. When I changed the file appending speed to be *3 second each time*. Everything worked normally. the following is my agent configuration file.
_#a1 taildir-----> fileroll_
_a1.sources = src1_
_a1.sinks = sink1_
_a1.channels = chan1_
_a1.sources.src1.type = TAILDIR_
_a1.sources.src1.positionFile = /home/EFM/Downloads/apache-flume-1.8.0-bin/TailDir/position.log_
_a1.sources.src1.filegroups =f1_
_a1.sources.src1.filegroups.f1 = /home/EFM/Downloads/apache-flume-1.8.0-bin/TailDir/log2.txt_
_a1.sources.src1.writePosInterval=500_
_a1.channels.chan1.type=memory_
_a1.channels.chan1.capacity=100000_
_a1.channels.chan1.transactionCapacity=10000_
_a1.sinks.sink1.channel = chan1_
_a1.sinks.sink1.type = file_roll_
_a1.sinks.sink1.sink.directory = /home/EFM/Downloads/apache-flume-1.8.0-bin/FileRoll_
_a1.sinks.sink1.sink.rollInterval=0_
_a1.sources.src1.channels = chan1_
_a1.sinks.sink1.channel = chan1_
Would you like give me some suggestions for that questions?
thanks.
was:
Hello,
I am using flume 1.8.0 for ubuntu 16.04. I met an question when using the agent, taildir-memory-fileroll. I programed for appending data ｉnto the file monitored by taildir *every 2 second*．Then at this speed the agent worked abnormally. Although the json file generated by taildir always updated and recorded the newest position, the file generated by rollfile didn't have any more data until I killed the bin which is appending data into the file monitored by taildir. When I changed the file appending speed to be *3 second each tim*e. Everything worked normally. the following is my agent configuration file.
_#a1 taildir-----> fileroll_
_a1.sources = src1_
_a1.sinks = sink1_
_a1.channels = chan1_
_a1.sources.src1.type = TAILDIR_
_a1.sources.src1.positionFile = /home/EFM/Downloads/apache-flume-1.8.0-bin/TailDir/position.log_
_a1.sources.src1.filegroups =f1_
_a1.sources.src1.filegroups.f1 = /home/EFM/Downloads/apache-flume-1.8.0-bin/TailDir/log2.txt_
_a1.sources.src1.writePosInterval=500_
_a1.channels.chan1.type=memory_
_a1.channels.chan1.capacity=100000_
_a1.channels.chan1.transactionCapacity=10000_
_a1.sinks.sink1.channel = chan1_
_a1.sinks.sink1.type = file_roll_
_a1.sinks.sink1.sink.directory = /home/EFM/Downloads/apache-flume-1.8.0-bin/FileRoll_
_a1.sinks.sink1.sink.rollInterval=0_
_a1.sources.src1.channels = chan1_
_a1.sinks.sink1.channel = chan1_
Would you like give me some suggestions for that questions?
thanks.
> taildir source works abnormally when the monitored file has too fast append speed．
> ----------------------------------------------------------------------------------
>
> Key: FLUME-3254
> URL: https://issues.apache.org/jira/browse/FLUME-3254
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.8.0
> Environment: flume 1.8.0 virtual-machine-ubuntu16.04 desktop
> Reporter: Alan Liu
> Priority: Major
> Fix For: notrack
>
>
> Hello,
> I am using flume 1.8.0 for ubuntu 16.04. I met an question when using the agent, taildir-memory-fileroll. I programed for appending data into the file monitored by taildir *every 2 second*．Then at this speed the agent worked abnormally. Although the json file generated by taildir always updated and recorded the newest position, the file generated by rollfile didn't have any more data until I killed the bin which is appending data into the file monitored by taildir. When I changed the file appending speed to be *3 second each time*. Everything worked normally. the following is my agent configuration file.
> _#a1 taildir-----> fileroll_
> _a1.sources = src1_
> _a1.sinks = sink1_
> _a1.channels = chan1_
> _a1.sources.src1.type = TAILDIR_
> _a1.sources.src1.positionFile = /home/EFM/Downloads/apache-flume-1.8.0-bin/TailDir/position.log_
> _a1.sources.src1.filegroups =f1_
> _a1.sources.src1.filegroups.f1 = /home/EFM/Downloads/apache-flume-1.8.0-bin/TailDir/log2.txt_
> _a1.sources.src1.writePosInterval=500_
> _a1.channels.chan1.type=memory_
> _a1.channels.chan1.capacity=100000_
> _a1.channels.chan1.transactionCapacity=10000_
> _a1.sinks.sink1.channel = chan1_
> _a1.sinks.sink1.type = file_roll_
> _a1.sinks.sink1.sink.directory = /home/EFM/Downloads/apache-flume-1.8.0-bin/FileRoll_
> _a1.sinks.sink1.sink.rollInterval=0_
> _a1.sources.src1.channels = chan1_
> _a1.sinks.sink1.channel = chan1_
> Would you like give me some suggestions for that questions?
> thanks.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org