[jira] [Commented] (FLUME-2450) Improve replay index insertion speed.

["Gary Malouf (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117697#comment-14117697 ] 

Gary Malouf commented on FLUME-2450:
------------------------------------

The frustration right now for us is that our flume nodes are basically 'down' until this recovery completes.  Could it make sense to do one or both of the following:

1) Make a new config option to run the version that requires extending the amount of JVM memory.  This would allow for faster recovery than the current solution, but only be used if it is explicitly set.

2) Open the ports/start consuming while recovery is taking place.  

> Improve replay index insertion speed.
> -------------------------------------
>
>                 Key: FLUME-2450
>                 URL: https://issues.apache.org/jira/browse/FLUME-2450
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Hari Shreedharan
>            Assignee: Hari Shreedharan
>             Fix For: v1.6.0
>
>         Attachments: FLUME-2450.patch
>
>
> Insertion into the replay index can take long sometimes because we use a file based index and tree set. We should switch this out for a memory mapped db and a hash set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)