You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Brock Noland (JIRA)" <ji...@apache.org> on 2012/09/14 01:42:08 UTC
[jira] [Created] (FLUME-1580) FileChannel some sets of log files
cannot be replayed
Brock Noland created FLUME-1580:
-----------------------------------
Summary: FileChannel some sets of log files cannot be replayed
Key: FLUME-1580
URL: https://issues.apache.org/jira/browse/FLUME-1580
Project: Flume
Issue Type: Improvement
Components: Channel
Affects Versions: v1.3.0
Reporter: Brock Noland
Assignee: Brock Noland
When log files not longer have put's referenced in the queue we delete them. Deleting these logs files is necessary to free up space. However, this can cause the error below due to this scenario:
Imagine a queue with capacity 2 and the following activity:
put a in log 1
put b in log 1
take a from log 2
take b from log 2
put c in log 1
put d in log 1
roll logs 1 & 2
checkpoint and delete log 2 since no puts in the queue reference it
for whatever reason the checkpoint is deleted
On replay we will see:
put a in log 1
put b in log 1
put c in log 1 <- this will exceed the queue capacity and throw the error below
put d in log 1
{noformat}
2012-09-13 17:45:14,095 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.Log.replay(Log.java:354)] Failed to initialize Log on [channel=channel1]
java.lang.IllegalStateException: Unable to add FlumeEventPointer [fileID=15, offset=2104422]. Queue depth = 5000, Capacity = 5000
at org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:394)
at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:329)
at org.apache.flume.channel.file.Log.replay(Log.java:339)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:272)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
The solution at present is to delete the checkpoint, increase the capacity of the channel, and restart. There will be duplicate events in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1580) FileChannel some sets of log files
cannot be replayed
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-1580:
--------------------------------
Description:
When log files not longer have put's referenced in the queue we delete them. Deleting these logs files is necessary to free up space. However, this can cause the error below due to this scenario:
Imagine a queue with capacity 2 and the following activity:
put a in log 1
put b in log 1
take a from log 2
take b from log 2
put c in log 1
put d in log 1
roll logs 1 & 2
checkpoint and delete log 2 since no puts in the queue reference it
for whatever reason the checkpoint is deleted
On replay we will see:
put a in log 1
put b in log 1
put c in log 1 <- this will exceed the queue capacity and throw the error below
put d in log 1
Example error message:
{noformat}
2012-09-13 17:45:14,095 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.Log.replay(Log.java:354)] Failed to initialize Log on [channel=channel1]
java.lang.IllegalStateException: Unable to add FlumeEventPointer [fileID=15, offset=2104422]. Queue depth = 5000, Capacity = 5000
at org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:394)
at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:329)
at org.apache.flume.channel.file.Log.replay(Log.java:339)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:272)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
The solution at present is to delete the checkpoint, increase the capacity of the channel, and restart. There will be duplicate events in this case.
was:
When log files not longer have put's referenced in the queue we delete them. Deleting these logs files is necessary to free up space. However, this can cause the error below due to this scenario:
Imagine a queue with capacity 2 and the following activity:
put a in log 1
put b in log 1
take a from log 2
take b from log 2
put c in log 1
put d in log 1
roll logs 1 & 2
checkpoint and delete log 2 since no puts in the queue reference it
for whatever reason the checkpoint is deleted
On replay we will see:
put a in log 1
put b in log 1
put c in log 1 <- this will exceed the queue capacity and throw the error below
put d in log 1
{noformat}
2012-09-13 17:45:14,095 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.Log.replay(Log.java:354)] Failed to initialize Log on [channel=channel1]
java.lang.IllegalStateException: Unable to add FlumeEventPointer [fileID=15, offset=2104422]. Queue depth = 5000, Capacity = 5000
at org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:394)
at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:329)
at org.apache.flume.channel.file.Log.replay(Log.java:339)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:272)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
The solution at present is to delete the checkpoint, increase the capacity of the channel, and restart. There will be duplicate events in this case.
> FileChannel some sets of log files cannot be replayed
> -----------------------------------------------------
>
> Key: FLUME-1580
> URL: https://issues.apache.org/jira/browse/FLUME-1580
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.3.0
> Reporter: Brock Noland
> Assignee: Brock Noland
>
> When log files not longer have put's referenced in the queue we delete them. Deleting these logs files is necessary to free up space. However, this can cause the error below due to this scenario:
> Imagine a queue with capacity 2 and the following activity:
> put a in log 1
> put b in log 1
> take a from log 2
> take b from log 2
> put c in log 1
> put d in log 1
> roll logs 1 & 2
> checkpoint and delete log 2 since no puts in the queue reference it
> for whatever reason the checkpoint is deleted
> On replay we will see:
> put a in log 1
> put b in log 1
> put c in log 1 <- this will exceed the queue capacity and throw the error below
> put d in log 1
> Example error message:
> {noformat}
> 2012-09-13 17:45:14,095 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.Log.replay(Log.java:354)] Failed to initialize Log on [channel=channel1]
> java.lang.IllegalStateException: Unable to add FlumeEventPointer [fileID=15, offset=2104422]. Queue depth = 5000, Capacity = 5000
> at org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:394)
> at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:329)
> at org.apache.flume.channel.file.Log.replay(Log.java:339)
> at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:272)
> at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The solution at present is to delete the checkpoint, increase the capacity of the channel, and restart. There will be duplicate events in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1580) FileChannel some sets of log files
cannot be replayed
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455447#comment-13455447 ]
Brock Noland commented on FLUME-1580:
-------------------------------------
One solution to this would be to instead of deleting the log files when they are no longer active, archive them. Then during replay we can read them in a special way. We know that their puts are no longer useful, but the takes might be. Eventually archived logs will need to be deletes as well.
> FileChannel some sets of log files cannot be replayed
> -----------------------------------------------------
>
> Key: FLUME-1580
> URL: https://issues.apache.org/jira/browse/FLUME-1580
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.3.0
> Reporter: Brock Noland
> Assignee: Brock Noland
>
> When log files not longer have put's referenced in the queue we delete them. Deleting these logs files is necessary to free up space. However, this can cause the error below due to this scenario:
> Imagine a queue with capacity 2 and the following activity:
> put a in log 1
> put b in log 1
> take a from log 2
> take b from log 2
> put c in log 1
> put d in log 1
> roll logs 1 & 2
> checkpoint and delete log 2 since no puts in the queue reference it
> for whatever reason the checkpoint is deleted
> On replay we will see:
> put a in log 1
> put b in log 1
> put c in log 1 <- this will exceed the queue capacity and throw the error below
> put d in log 1
> {noformat}
> 2012-09-13 17:45:14,095 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.Log.replay(Log.java:354)] Failed to initialize Log on [channel=channel1]
> java.lang.IllegalStateException: Unable to add FlumeEventPointer [fileID=15, offset=2104422]. Queue depth = 5000, Capacity = 5000
> at org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:394)
> at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:329)
> at org.apache.flume.channel.file.Log.replay(Log.java:339)
> at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:272)
> at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The solution at present is to delete the checkpoint, increase the capacity of the channel, and restart. There will be duplicate events in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1580) FileChannel some sets of log files
cannot be replayed
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456002#comment-13456002 ]
Brock Noland commented on FLUME-1580:
-------------------------------------
"lowest numbered active put"
This is going to be the interesting part because we don't track the write order id of puts and doing so would be troublesome. However, I think we can cheat here and do a peek on the head of the queue and assume that is the lowest write order. Since the queue will be nearly ordered that is not a terrible assumption and in the case we are wrong will only mean a few events are delivered twice in the case of full replay. We can also keep a couple extra logs around as we do now and that scenario is even less likely.
> FileChannel some sets of log files cannot be replayed
> -----------------------------------------------------
>
> Key: FLUME-1580
> URL: https://issues.apache.org/jira/browse/FLUME-1580
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.3.0
> Reporter: Brock Noland
> Assignee: Brock Noland
>
> When log files not longer have put's referenced in the queue we delete them. Deleting these logs files is necessary to free up space. However, this can cause the error below due to this scenario:
> Imagine a queue with capacity 2 and the following activity:
> put a in log 1
> put b in log 1
> take a from log 2
> take b from log 2
> put c in log 1
> put d in log 1
> roll logs 1 & 2
> checkpoint and delete log 2 since no puts in the queue reference it
> for whatever reason the checkpoint is deleted
> On replay we will see:
> put a in log 1
> put b in log 1
> put c in log 1 <- this will exceed the queue capacity and throw the error below
> put d in log 1
> Example error message:
> {noformat}
> 2012-09-13 17:45:14,095 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.Log.replay(Log.java:354)] Failed to initialize Log on [channel=channel1]
> java.lang.IllegalStateException: Unable to add FlumeEventPointer [fileID=15, offset=2104422]. Queue depth = 5000, Capacity = 5000
> at org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:394)
> at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:329)
> at org.apache.flume.channel.file.Log.replay(Log.java:339)
> at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:272)
> at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The solution at present is to delete the checkpoint, increase the capacity of the channel, and restart. There will be duplicate events in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1580) FileChannel some sets of log files
cannot be replayed
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456042#comment-13456042 ]
Brock Noland commented on FLUME-1580:
-------------------------------------
In the case of rollbacks the ordering of the queue gets screwed up and the peek method might now work.
> FileChannel some sets of log files cannot be replayed
> -----------------------------------------------------
>
> Key: FLUME-1580
> URL: https://issues.apache.org/jira/browse/FLUME-1580
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.3.0
> Reporter: Brock Noland
> Assignee: Brock Noland
>
> When log files not longer have put's referenced in the queue we delete them. Deleting these logs files is necessary to free up space. However, this can cause the error below due to this scenario:
> Imagine a queue with capacity 2 and the following activity:
> put a in log 1
> put b in log 1
> take a from log 2
> take b from log 2
> put c in log 1
> put d in log 1
> roll logs 1 & 2
> checkpoint and delete log 2 since no puts in the queue reference it
> for whatever reason the checkpoint is deleted
> On replay we will see:
> put a in log 1
> put b in log 1
> put c in log 1 <- this will exceed the queue capacity and throw the error below
> put d in log 1
> Example error message:
> {noformat}
> 2012-09-13 17:45:14,095 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.Log.replay(Log.java:354)] Failed to initialize Log on [channel=channel1]
> java.lang.IllegalStateException: Unable to add FlumeEventPointer [fileID=15, offset=2104422]. Queue depth = 5000, Capacity = 5000
> at org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:394)
> at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:329)
> at org.apache.flume.channel.file.Log.replay(Log.java:339)
> at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:272)
> at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The solution at present is to delete the checkpoint, increase the capacity of the channel, and restart. There will be duplicate events in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1580) FileChannel some sets of log files
cannot be replayed
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455991#comment-13455991 ]
Brock Noland commented on FLUME-1580:
-------------------------------------
Yes that makes sense. We still have to skip Put's on replay for non-active (call them archived) files otherwise you are in a recursive scenario requiring all log files ever.
> FileChannel some sets of log files cannot be replayed
> -----------------------------------------------------
>
> Key: FLUME-1580
> URL: https://issues.apache.org/jira/browse/FLUME-1580
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.3.0
> Reporter: Brock Noland
> Assignee: Brock Noland
>
> When log files not longer have put's referenced in the queue we delete them. Deleting these logs files is necessary to free up space. However, this can cause the error below due to this scenario:
> Imagine a queue with capacity 2 and the following activity:
> put a in log 1
> put b in log 1
> take a from log 2
> take b from log 2
> put c in log 1
> put d in log 1
> roll logs 1 & 2
> checkpoint and delete log 2 since no puts in the queue reference it
> for whatever reason the checkpoint is deleted
> On replay we will see:
> put a in log 1
> put b in log 1
> put c in log 1 <- this will exceed the queue capacity and throw the error below
> put d in log 1
> Example error message:
> {noformat}
> 2012-09-13 17:45:14,095 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.Log.replay(Log.java:354)] Failed to initialize Log on [channel=channel1]
> java.lang.IllegalStateException: Unable to add FlumeEventPointer [fileID=15, offset=2104422]. Queue depth = 5000, Capacity = 5000
> at org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:394)
> at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:329)
> at org.apache.flume.channel.file.Log.replay(Log.java:339)
> at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:272)
> at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The solution at present is to delete the checkpoint, increase the capacity of the channel, and restart. There will be duplicate events in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1580) FileChannel some sets of log files
cannot be replayed
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455975#comment-13455975 ]
Mike Percy commented on FLUME-1580:
-----------------------------------
I think the invariant that we want here is do not delete log files with takes with a totally-ordered sequence number higher than the totally-ordered sequence number of any other active put. I think we could get away with keeping track of the highest numbered take in each log file and the lowest numbered active put in each log file, in addition to the information we store now about the active puts, and that should be sufficient to implement this logic.
> FileChannel some sets of log files cannot be replayed
> -----------------------------------------------------
>
> Key: FLUME-1580
> URL: https://issues.apache.org/jira/browse/FLUME-1580
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.3.0
> Reporter: Brock Noland
> Assignee: Brock Noland
>
> When log files not longer have put's referenced in the queue we delete them. Deleting these logs files is necessary to free up space. However, this can cause the error below due to this scenario:
> Imagine a queue with capacity 2 and the following activity:
> put a in log 1
> put b in log 1
> take a from log 2
> take b from log 2
> put c in log 1
> put d in log 1
> roll logs 1 & 2
> checkpoint and delete log 2 since no puts in the queue reference it
> for whatever reason the checkpoint is deleted
> On replay we will see:
> put a in log 1
> put b in log 1
> put c in log 1 <- this will exceed the queue capacity and throw the error below
> put d in log 1
> Example error message:
> {noformat}
> 2012-09-13 17:45:14,095 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.Log.replay(Log.java:354)] Failed to initialize Log on [channel=channel1]
> java.lang.IllegalStateException: Unable to add FlumeEventPointer [fileID=15, offset=2104422]. Queue depth = 5000, Capacity = 5000
> at org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:394)
> at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:329)
> at org.apache.flume.channel.file.Log.replay(Log.java:339)
> at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:272)
> at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The solution at present is to delete the checkpoint, increase the capacity of the channel, and restart. There will be duplicate events in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira