You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "David Stendardi (JIRA)" <ji...@apache.org> on 2014/04/30 12:25:20 UTC

[jira] [Created] (FLUME-2375) HDFS sink's fail to recover from datanode unavailability

David Stendardi created FLUME-2375:
--------------------------------------

             Summary: HDFS sink's fail to recover from datanode unavailability
                 Key: FLUME-2375
                 URL: https://issues.apache.org/jira/browse/FLUME-2375
             Project: Flume
          Issue Type: Bug
    Affects Versions: v1.4.0
            Reporter: David Stendardi


Hello !

We are running flume-ng with version cdh-4.5-1.4. When a datanode used by flume-ng goes done, we get the following exceptions :  

{code}
30 Apr 2014 01:10:38,130 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated:96)  - Unexpected error while checking replication factor
java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.flume.sink.hdfs.AbstractHDFSWriter.getNumCurrentReplicas(AbstractHDFSWriter.java:162)
        at org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:82)
        at org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:452)
        at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:387)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:392)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)
{code}

These exceptions are logged but not rethrown, and the AbstractHdfsSink::isUnderReplicated still returns false so the writer continue to try writing on the node.




--
This message was sent by Atlassian JIRA
(v6.2#6252)