[jira] [Commented] (FLUME-2525) flume should handle a zero byte .flumespool-main.meta file for the spooldir source

["Patrick Dvorak (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187712#comment-14187712 ] 

Patrick Dvorak commented on FLUME-2525:
---------------------------------------

When a partition containing the trackerDir (or spoolDir in the event trackerDir is not specified) fills up, a new meta file can be created, but it will have a zero byte size.  This prevents flume from properly processing any files once the partition full issue has been resolved, even across restarts.  The error thrown does not really indicate the root issue.  Ideally flume should remove any zero byte spooldir meta files when it starts up, as these are invalid.


> flume should handle a zero byte .flumespool-main.meta file for the spooldir source
> ----------------------------------------------------------------------------------
>
>                 Key: FLUME-2525
>                 URL: https://issues.apache.org/jira/browse/FLUME-2525
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.5.0.1
>            Reporter: Patrick Dvorak
>
> When a zero byte .flumespool-main.meta file exists in the trackerDir (usually do to the partition filling up), flume will throw the following ambiguous error message when trying to read in new spool files:
> 2014-10-19 18:28:31,333 ERROR org.apache.flume.client.avro.ReliableSpoolingFileEventReader: Exception opening file: /home/spooldir/input.log
> java.io.IOException: Not a data file. 
> at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:102) 
> at org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97) 
> at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:160) 
> at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:149) 
> at org.apache.flume.serialization.DurablePositionTracker.<init>(DurablePositionTracker.java:141) 
> at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:76) 
> at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:420) 
> at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:215) 
> at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:182) 
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) 
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) 
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) 
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) 
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) 
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) 
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 
> at java.lang.Thread.run(Thread.java:662) 
> Restarts of the flume agent do not resolve the issue.  Only when the zero byte file is removed, will flume properly start processing files from the spooldir again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)