You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Johny Rufus (JIRA)" <ji...@apache.org> on 2015/03/01 14:34:04 UTC
[jira] [Assigned] (FLUME-2507) File Channel cannot support massive
capacities
[ https://issues.apache.org/jira/browse/FLUME-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johny Rufus reassigned FLUME-2507:
----------------------------------
Assignee: Johny Rufus
> File Channel cannot support massive capacities
> ----------------------------------------------
>
> Key: FLUME-2507
> URL: https://issues.apache.org/jira/browse/FLUME-2507
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Johny Rufus
>
> In the File Channel, we map the checkpoint file. Java cannot seem to be able to map a single file of more than 2GB, since MappedByteBuffer is indexed on int. So even if we get a LongBuffer using asLongBuffer (since that is only a view on the real buffer), the checkpointing will fail when the data is being put, with a nasty exception that looks like:
> {code}
> 2014-10-16 13:48:54,684 (Log-BackgroundWorker-FileChannel1) [ERROR - org.apache.flume.channel.file.Log$BackgroundWorker.run(Log.java:1188)] General error in checkpoint worker
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:512)
> at java.nio.DirectLongBufferS.put(DirectLongBufferS.java:270)
> at org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint(EventQueueBackingStoreFile.java:269)
> at org.apache.flume.channel.file.FlumeEventQueue.checkpoint(FlumeEventQueue.java:145)
> at org.apache.flume.channel.file.Log.writeCheckpoint(Log.java:991)
> at org.apache.flume.channel.file.Log.writeCheckpoint(Log.java:968)
> at org.apache.flume.channel.file.Log.access$200(Log.java:75)
> at org.apache.flume.channel.file.Log$BackgroundWorker.run(Log.java:1183)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)