You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Rudy Krol (JIRA)" <ji...@apache.org> on 2013/01/31 16:21:15 UTC
[jira] [Created] (FLUME-1887) SpoolingDirectorySource throwing
IllegalStateException while trying to rename/close stream
Rudy Krol created FLUME-1887:
--------------------------------
Summary: SpoolingDirectorySource throwing IllegalStateException while trying to rename/close stream
Key: FLUME-1887
URL: https://issues.apache.org/jira/browse/FLUME-1887
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.3.1
Reporter: Rudy Krol
I am using SpoolDirectorySource to retrieve a 700MB file.
Data is sent to Flume but at the end, when trying to rename file (.COMPLETED) and close stream, I get this exception :
01 Feb 2013 01:18:40,579 INFO [pool-7-thread-1] (org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229) - Preparing to move file /Probes/1/MyFile.txt to /Probes/1/MyFile.txt.COMPLETED
01 Feb 2013 01:18:40,580 ERROR [pool-7-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) - Uncaught exception in Runnable
java.lang.IllegalStateException: File has been modified since being read: /Probes/1/MyFile.txt
at org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineReader.java:237)
at org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:185)
at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
01 Feb 2013 01:18:41,081 ERROR [pool-7-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) - Uncaught exception in Runnable
java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:114)
at java.io.BufferedReader.readLine(BufferedReader.java:309)
at java.io.BufferedReader.readLine(BufferedReader.java:379)
at org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:180)
at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
01 Feb 2013 01:18:41,582 ERROR [pool-7-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) - Uncaught exception in Runnable
And this exception is thrown indefinitely, the file is never renamed.
The same configuration is OK with smaller files (2MB):
agent.channels = channel1
agent.channels.channel1.type = memory
agent.channels.channel1.capacity = 100000
agent.channels.channel1.transactionCapacity = 1000
agent.sources = source1
agent.sources.source1.channels = channel1
agent.sources.source1.type = spooldir
agent.sources.source1.spoolDir = /Probes/1/
agent.sources.source1.fileHeader = true
agent.sources.source1.fileHeaderKey = fileName
agent.sources.source1.batchSize = 100
agent.sinks = sink1
agent.sinks.sink1.channel = channel1
agent.sinks.sink1.type = avro
agent.sinks.sink1.hostname = 33.33.33.33
agent.sinks.sink1.port = 5141
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira