[jira] [Commented] (FLUME-2566) BucketWriter tries to close file endlessly

["Juan F. Tavira (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383555#comment-14383555 ] 

Juan F. Tavira commented on FLUME-2566:
---------------------------------------

v1.5.2
I have the same issue, running for 17h in a row with the same file.
Rebooting the agent solves it.


> BucketWriter tries to close file endlessly
> ------------------------------------------
>
>                 Key: FLUME-2566
>                 URL: https://issues.apache.org/jira/browse/FLUME-2566
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.5.1
>            Reporter: Tycho Lamerigts
>            Assignee: Johny Rufus
>
> The following scenario causes BucketWriter to go in a endless loop trying to close a file:
> On the first call to close, there is a timeout (due to hdfs being temporarily overloaded)
> {noformat}
> 12:53:57.363 [hdfs-hdfs_sink_3-roll-timer-0] WARN   o.a.flume.sink.hdfs.BucketWriter - failed to close() HDFSWriter for file (/rawdata/medusa/data/p_nl_omm_goat_medusa01/20141202/node1/FOO/medusa.1417521000000.1417521129207.avro.tmp). Exception follows.
> java.io.IOException: Callable timed out after 10000 ms on file: /rawdata/medusa/data/p_nl_omm_goat_medusa01/20141202/node1/FOO/medusa.1417521000000.1417521129207.avro.tmp
> 	at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:736) ~[flume-hdfs-sink.jar:na]
> 	at org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:417) ~[flume-hdfs-sink.jar:na]
> 	at org.apache.flume.sink.hdfs.BucketWriter$5.call(BucketWriter.java:476) [flume-hdfs-sink.jar:na]
> 	at org.apache.flume.sink.hdfs.BucketWriter$5.call(BucketWriter.java:471) [flume-hdfs-sink.jar:na]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_20]
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_20]
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_20]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_20]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_20]
> {noformat}
> BucketWriter schedules an operation to retry the close. This retry, and every other retry after that, fails because, by now, the channel succeeded in closing and refuses to flush/close again. Instead it throws an exception that causes BucketWriter to schedule another close retry (which will fail again).
> {noformat}
> 14:32:58.793 [hdfs-hdfs_sink_3-roll-timer-0] WARN   o.a.flume.sink.hdfs.BucketWriter - Closing file: /rawdata/medusa/data/p_nl_omm_goat_medusa01/20141202/node1/FOO/medusa.1417521000000.1417521129207.avro.tmp failed. Will retry again in 180 seconds.
> java.nio.channels.ClosedChannelException: null
> 	at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1527) ~[hadoop-hdfs.jar:na]
> 	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1843) ~[hadoop-hdfs.jar:na]
> 	at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1803) ~[hadoop-hdfs.jar:na]
> 	at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1788) ~[hadoop-hdfs.jar:na]
> 	at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:120) ~[hadoop-common.jar:na]
> 	at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:139) ~[flume-hdfs-sink.jar:na]
> 	at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:341) ~[flume-hdfs-sink.jar:na]
> 	at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:335) ~[flume-hdfs-sink.jar:na]
> 	at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:722) ~[flume-hdfs-sink.jar:na]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)