You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Cameron Gandevia <cg...@gmail.com> on 2011/09/02 20:41:37 UTC
Sink Closes on Driver Error
Hi
I am experiencing a weird issue with my Flume configuration. Occasionally
when I start my agents they start delivering messages to the collectors and
after awhile I receive the following exceptions and stop receiving messages.
I am also curious if anyone can comment how they monitor these kind of
errors in their flume deployments. Currently we are monitoring the flume
process which is still running. I am guessing the best option is to also
monitor the hdfs destination to ensure we are constantly receiving a stream
of events from flume?
I am currently using the Cloudera Flume 0.9.4+25.9 build in multi master
mode.
The collectors are configured as follows.
Source: collectorSource(36892)
Sink: {regex("^(?:\\n)?(\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d\\d)", 1,
"date") => {exDate("date", "yyyy-MM-dd HH:mm") =>
collectorSink("hdfs://hadoop-namenode1:8020/logs/%{dateyear}-%{datemonth}-%{dateday}/%{datehr}00/%{host}",
"log-")}}
The agents are configured as follows.
Source: tailDir("/data1/logs", delim="\\n\\d\\d\\d\\d", delimMode="next")
Sink: agentDFOChain("flume-collector1:36892", "flume-collector2:36892")
The following exceptions are from my collectors.
INFO com.cloudera.flume.handlers.debug.InsistentAppendDecorator - Failed
due to unexpected runtime exception during append attempt
java.lang.NullPointerException
at
com.cloudera.flume.handlers.hdfs.CustomDfsSink.close(CustomDfsSink.java:88)
at
com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.close(EscapedCustomDfsSink.java:132)
at
com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
at
com.cloudera.flume.collector.CollectorSink$RollDetectDeco.close(CollectorSink.java:171)
at
com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:331)
at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
at
com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
at
com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:96)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
at
com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
ERROR com.cloudera.flume.core.connector.DirectDriver - Closing down due to
exception during append calls
java.lang.NullPointerException
at
com.cloudera.flume.handlers.hdfs.CustomDfsSink.close(CustomDfsSink.java:88)
at
com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.close(EscapedCustomDfsSink.java:132)
at
com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
at
com.cloudera.flume.collector.CollectorSink$RollDetectDeco.close(CollectorSink.java:171)
at
com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:331)
at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
at
com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
at
com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:96)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
at
com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129)
at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at
com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
The following are from my agents
ERROR com.cloudera.flume.core.connector.DirectDriver - Expected IDLE but
timed out in state ACTIVE
INFO com.cloudera.flume.agent.diskfailover.DiskFailoverDeco - Closing disk
failover log, subsink still making progre
ERROR com.cloudera.flume.agent.LogicalNode - Forcing driver to exit
uncleanly
INFO com.cloudera.flume.agent.LogicalNode - Node config successfully set to
com.cloudera.flume.conf.FlumeConfigData@19a0feb
WARN com.cloudera.flume.handlers.debug.LazyOpenDecorator - Closing a lazy
sink that was not logically opened
INFO com.cloudera.flume.handlers.text.TailDirSource - added file
/data1/mutator/logs/one.log
INFO com.cloudera.flume.handlers.rolling.RollSink - Created RollSink:
trigger=[TimeTrigger: maxAge=10000 tagger=com.
cloudera.flume.handlers.rolling.ProcessTagger@6279d] checkPeriodMs = 250
spec='NaiveFileFailover'
INFO com.cloudera.flume.handlers.rolling.RollSink - opening RollSink
'NaiveFileFailover'
ERROR com.cloudera.flume.agent.diskfailover.DiskFailoverDeco - WAL drain
thread interrupted
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at
com.cloudera.flume.core.connector.DirectDriver.waitForAtLeastState(DirectDriver.java:308)
at
com.cloudera.flume.agent.diskfailover.DiskFailoverDeco.ensureClosedDrainDriver(DiskFailoverDeco.java:129)
at
com.cloudera.flume.agent.diskfailover.DiskFailoverDeco.close(DiskFailoverDeco.java:177)
at
com.cloudera.flume.core.BackOffFailOverSink.close(BackOffFailOverSink.java:165)
at
com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
at
com.cloudera.flume.agent.AgentFailChainSink.close(AgentFailChainSink.java:98)
at
com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:126)
Thanks
Re: Sink Closes on Driver Error
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Cameron
Thanks for fliling the jira.
https://issues.apache.org/jira/browse/FLUME-757
I've commented there. Basically, I think there was a race introduced in the
new version. I've described two potential solutions in the jira.
Jon.
On Fri, Sep 2, 2011 at 11:41 AM, Cameron Gandevia <cg...@gmail.com>wrote:
> Hi
>
> I am experiencing a weird issue with my Flume configuration. Occasionally
> when I start my agents they start delivering messages to the collectors and
> after awhile I receive the following exceptions and stop receiving
> messages.
>
> I am also curious if anyone can comment how they monitor these kind of
> errors in their flume deployments. Currently we are monitoring the flume
> process which is still running. I am guessing the best option is to also
> monitor the hdfs destination to ensure we are constantly receiving a stream
> of events from flume?
>
> I am currently using the Cloudera Flume 0.9.4+25.9 build in multi master
> mode.
>
> The collectors are configured as follows.
>
> Source: collectorSource(36892)
> Sink: {regex("^(?:\\n)?(\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d\\d)", 1,
> "date") => {exDate("date", "yyyy-MM-dd HH:mm") =>
>
> collectorSink("hdfs://hadoop-namenode1:8020/logs/%{dateyear}-%{datemonth}-%{dateday}/%{datehr}00/%{host}",
> "log-")}}
>
> The agents are configured as follows.
>
> Source: tailDir("/data1/logs", delim="\\n\\d\\d\\d\\d", delimMode="next")
> Sink: agentDFOChain("flume-collector1:36892", "flume-collector2:36892")
>
> The following exceptions are from my collectors.
>
> INFO com.cloudera.flume.handlers.debug.InsistentAppendDecorator - Failed
> due to unexpected runtime exception during append attempt
> java.lang.NullPointerException
> at
> com.cloudera.flume.handlers.hdfs.CustomDfsSink.close(CustomDfsSink.java:88)
> at
>
> com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.close(EscapedCustomDfsSink.java:132)
> at
> com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
> at
>
> com.cloudera.flume.collector.CollectorSink$RollDetectDeco.close(CollectorSink.java:171)
> at
> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:331)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
> at
>
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
> at
>
> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:96)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
> at
> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
> at
>
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
> ERROR com.cloudera.flume.core.connector.DirectDriver - Closing down due to
> exception during append calls
> java.lang.NullPointerException
> at
> com.cloudera.flume.handlers.hdfs.CustomDfsSink.close(CustomDfsSink.java:88)
> at
>
> com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.close(EscapedCustomDfsSink.java:132)
> at
> com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
> at
>
> com.cloudera.flume.collector.CollectorSink$RollDetectDeco.close(CollectorSink.java:171)
> at
> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:331)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
> at
>
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
> at
>
> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:96)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
> at
> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129)
> at
>
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
>
> com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
> at
>
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
>
> The following are from my agents
>
> ERROR com.cloudera.flume.core.connector.DirectDriver - Expected IDLE but
> timed out in state ACTIVE
> INFO com.cloudera.flume.agent.diskfailover.DiskFailoverDeco - Closing disk
> failover log, subsink still making progre
> ERROR com.cloudera.flume.agent.LogicalNode - Forcing driver to exit
> uncleanly
> INFO com.cloudera.flume.agent.LogicalNode - Node config successfully set
> to
> com.cloudera.flume.conf.FlumeConfigData@19a0feb
> WARN com.cloudera.flume.handlers.debug.LazyOpenDecorator - Closing a lazy
> sink that was not logically opened
> INFO com.cloudera.flume.handlers.text.TailDirSource - added file
> /data1/mutator/logs/one.log
> INFO com.cloudera.flume.handlers.rolling.RollSink - Created RollSink:
> trigger=[TimeTrigger: maxAge=10000 tagger=com.
> cloudera.flume.handlers.rolling.ProcessTagger@6279d] checkPeriodMs = 250
> spec='NaiveFileFailover'
> INFO com.cloudera.flume.handlers.rolling.RollSink - opening RollSink
> 'NaiveFileFailover'
> ERROR com.cloudera.flume.agent.diskfailover.DiskFailoverDeco - WAL drain
> thread interrupted
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at
>
> com.cloudera.flume.core.connector.DirectDriver.waitForAtLeastState(DirectDriver.java:308)
> at
>
> com.cloudera.flume.agent.diskfailover.DiskFailoverDeco.ensureClosedDrainDriver(DiskFailoverDeco.java:129)
> at
>
> com.cloudera.flume.agent.diskfailover.DiskFailoverDeco.close(DiskFailoverDeco.java:177)
> at
>
> com.cloudera.flume.core.BackOffFailOverSink.close(BackOffFailOverSink.java:165)
> at
> com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
> at
>
> com.cloudera.flume.agent.AgentFailChainSink.close(AgentFailChainSink.java:98)
> at
> com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
> at
>
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:126)
>
> Thanks
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com