You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Eric Hauser (JIRA)" <ji...@apache.org> on 2011/09/16 00:21:09 UTC
[jira] [Issue Comment Edited] (FLUME-734) escapedFormatDfs goes
into a file creation frenzy
[ https://issues.apache.org/jira/browse/FLUME-734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105411#comment-13105411 ]
Eric Hauser edited comment on FLUME-734 at 9/15/11 10:20 PM:
-------------------------------------------------------------
When trying this patch using avrodata, I got the following:
java.lang.NullPointerException
at com.cloudera.flume.conf.SinkBuilderUtil.resolveOutputFormat(SinkBuilderUtil.java:59)
at com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.openWriter(EscapedCustomDfsSink.java:109)
at com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.append(EscapedCustomDfsSink.java:131)
at com.cloudera.flume.core.CompositeSink.append(CompositeSink.java:61)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at com.cloudera.flume.handlers.rolling.RollSink.synchronousAppend(RollSink.java:234)
at com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:183)
at com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:181)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
The context needs to be assigned to the member variable in the constructor. After doing this, the patch worked for us.
was (Author: ewhauser):
When trying this patch using avrodata, I got the following:
java.lang.NullPointerException
at com.cloudera.flume.conf.SinkBuilderUtil.resolveOutputFormat(SinkBuilderUtil.java:59)
at com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.openWriter(EscapedCustomDfsSink.java:109)
at com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.append(EscapedCustomDfsSink.java:131)
at com.cloudera.flume.core.CompositeSink.append(CompositeSink.java:61)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
at com.cloudera.flume.handlers.rolling.RollSink.synchronousAppend(RollSink.java:234)
at com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:183)
at com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:181)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> escapedFormatDfs goes into a file creation frenzy
> -------------------------------------------------
>
> Key: FLUME-734
> URL: https://issues.apache.org/jira/browse/FLUME-734
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Environment: CentOS 5.6
> Reporter: Eran Kutner
> Assignee: Jonathan Hsieh
> Priority: Critical
> Attachments: FLUME-734-draft.patch, flume.log
>
>
> Using this configuration:
> collectorSource(54001) | collector(600000) { escapedFormatDfs("hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/", "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec")) }
> The expected behavior is to see a new file created every 10 minutes. However, once in a while the collector would go into a file creation frenzy, creating new files every second.
> The log indicates that writing has failed with error: "OutputFormat instance can only write to the same OutputStream" causing the file to be closed a new one to be opened just to be closed again.
> Looking at the code I'm not even sure how the output stream could change but the behavior I'm seeing feels like some sort of a race condition. It is happening much more under heavy load than under low load.
> See attached log excerpt.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira