You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Arpit Gupta (JIRA)" <ji...@apache.org> on 2015/10/05 22:56:26 UTC
[jira] [Commented] (FLUME-2808) TailDirSource fails to process the
sources if one of the directories configured for file groups does not exist
[ https://issues.apache.org/jira/browse/FLUME-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944017#comment-14944017 ]
Arpit Gupta commented on FLUME-2808:
------------------------------------
My goal was to keep all agent configs the same. IE configure all the possible directories for all nodes assuming that directories that did not exist would be ignored. To me that feature would be helpful as one would not have to maintain configs per agent for the same source.
> TailDirSource fails to process the sources if one of the directories configured for file groups does not exist
> --------------------------------------------------------------------------------------------------------------
>
> Key: FLUME-2808
> URL: https://issues.apache.org/jira/browse/FLUME-2808
> Project: Flume
> Issue Type: Bug
> Affects Versions: v1.7.0
> Reporter: Arpit Gupta
>
> Configured TailDirSource with multiple file groups. Some of the file group directories do not exist on all nodes on the cluster.
> One the nodes where the file group directory did not exist TailDirSource failed to run with exception
> {code}
> 05 Oct 2015 20:39:56,433 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: /dir/flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
> java.lang.IllegalStateException: Directory does not exist: /dir/log/hive
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:86)
> at org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:55)
> at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:343)
> at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:91)
> at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:74)
> at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I built flume from sha: bd80c5e67deda4ec146f19fc92ff0c3bff61a982
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)