You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Hari Shreedharan (JIRA)" <ji...@apache.org> on 2014/11/03 23:56:35 UTC

[jira] [Commented] (FLUME-2525) flume should handle a zero byte .flumespool-main.meta file for the spooldir source

    [ https://issues.apache.org/jira/browse/FLUME-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195289#comment-14195289 ] 

Hari Shreedharan commented on FLUME-2525:
-----------------------------------------

Couple of comments:
- Do you really need to wait for the IOException before deleting the zero-byte file? Why not check beforehand, and delete the file?
- In the unit tests, please use junit Assert.* methods, not the standard java asserts.


> flume should handle a zero byte .flumespool-main.meta file for the spooldir source
> ----------------------------------------------------------------------------------
>
>                 Key: FLUME-2525
>                 URL: https://issues.apache.org/jira/browse/FLUME-2525
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.5.0.1
>            Reporter: Patrick Dvorak
>             Fix For: v1.6.0
>
>         Attachments: FLUME-2525.patch
>
>
> When a zero byte .flumespool-main.meta file exists in the trackerDir (usually do to the partition filling up), flume will throw the following ambiguous error message when trying to read in new spool files:
> 2014-10-19 18:28:31,333 ERROR org.apache.flume.client.avro.ReliableSpoolingFileEventReader: Exception opening file: /home/spooldir/input.log
> java.io.IOException: Not a data file. 
> at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:102) 
> at org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97) 
> at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:160) 
> at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:149) 
> at org.apache.flume.serialization.DurablePositionTracker.<init>(DurablePositionTracker.java:141) 
> at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:76) 
> at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:420) 
> at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:215) 
> at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:182) 
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) 
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) 
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) 
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) 
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) 
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) 
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 
> at java.lang.Thread.run(Thread.java:662) 
> Restarts of the flume agent do not resolve the issue.  Only when the zero byte file is removed, will flume properly start processing files from the spooldir again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)