You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "chenshangan (JIRA)" <ji...@apache.org> on 2015/09/22 17:16:04 UTC
[jira] [Resolved] (FLUME-2353) BucketWriter throw IOException
endlessly while failed to close file
[ https://issues.apache.org/jira/browse/FLUME-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chenshangan resolved FLUME-2353.
--------------------------------
Resolution: Duplicate
Resolved in another task. It will provide max retry times option.
> BucketWriter throw IOException endlessly while failed to close file
> -------------------------------------------------------------------
>
> Key: FLUME-2353
> URL: https://issues.apache.org/jira/browse/FLUME-2353
> Project: Flume
> Issue Type: Improvement
> Reporter: chenshangan
> Assignee: chenshangan
>
> sometimes .tmp file might lost block in hdfs, and HDFSWriter can not go on writing events or flush or close the file, so it will repeatedly try and catch the IOException.
> The error stack is as following:
> 06 Aug 2013 04:27:08,853 WARN [DataStreamer for file **************************.1375732802628.lzo.tmp block blk_709795560527813415_25801594] (org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError:3159) - Error Recovery for block blk_709795560527813415_25801594 failed because recovery from primary datanode *****:50010 failed 1 times. Pipeline was ******:50010,******:50010, ******:50010. Will retry...
> 06 Aug 2013 04:27:08,990 WARN [DataStreamer for file **************************.1375732802628.lzo.tmp block blk_709795560527813415_25801594] (org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError:3159) - Error Recovery for block blk_709795560527813415_25801594 failed because recovery from primary datanode ******:50010 failed 2 times. Pipeline was ******:50010,******:50010, ******:50010. Will retry...
> …
> 06 Aug 2013 04:27:50,694 WARN [DataStreamer for file **************************.1375732802628.lzo.tmp block blk_709795560527813415_25801594] (org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError:3139) - Error Recovery for block blk_709795560527813415_25801594 failed because recovery from primary datanode ******:50010 failed 6 times. Pipeline was ******:50010,******:50010, ******:50010. Marking primary datanode as bad.
> 06 Aug 2013 04:30:40,365 WARN [SinkRunner-PollingRunner-FailoverSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:418) - HDFS IO error
> java.io.IOException: Error Recovery for block blk_709795560527813415_25801594 failed because recovery from primary datanode ********:50010 failed 6 times. Pipeline was *****:50010. Aborting...
> DFSClient will try to recovery for missing block with a maximum times , if failed finally it will throw IOException. But HDFSWriter will rethrow the exception to HDFSEventSink, HDFSEventSink will rollback transaction and repeat the error stack, so it's a dead loop.
> My suggestion and solution is to add a graceClose() method, and if it failed too many times, just leave the .tmp file alone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)