[jira] [Commented] (FALCON-1368) Improve Falcon server restart time

["sandeep samudrala (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FALCON-1368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658741#comment-14658741 ] 

sandeep samudrala commented on FALCON-1368:
-------------------------------------------

I have made changes to make configuration store loading of entities parallel for each entity type in the respective order ( CLUSTER, FEED, PROCESS)

This is a sample of how it looks and there is a huge dip in the restart time of falcon server. 
{noformat}
2015-08-06 00:59:04,687 INFO  - [pool-5-thread-95:] ~ Restored configuration FEED/ipfeed34 (ConfigurationStore:420)
2015-08-06 00:59:04,690 INFO  - [pool-5-thread-94:] ~ Restored configuration FEED/ipfeed18 (ConfigurationStore:420)
2015-08-06 00:59:04,694 INFO  - [pool-5-thread-93:] ~ Restored configuration FEED/ipfeed10 (ConfigurationStore:420)
{noformat}



> Improve Falcon server restart  time
> -----------------------------------
>
>                 Key: FALCON-1368
>                 URL: https://issues.apache.org/jira/browse/FALCON-1368
>             Project: Falcon
>          Issue Type: Improvement
>            Reporter: Ajay Yadava
>            Assignee: sandeep samudrala
>         Attachments: FALCON-1368.patch
>
>
> Currently on restart, Falcon server loads all the entities from HDFS one by one. In a large set up like the one at Inmobi, where we have several thousands of feeds and processes, this takes several minutes to increase the start up time.
> Since this is an IO intensive task(reading a file from HDFS into memory), having multiple threads to load entities in parallel will improve the start up time of the server. 
> Two points need to be taken care of:
> 1. Only a single type of entities should be load in parallel to preserve the order of load of entities.
> 2. Currently Falcon server fails to start if there is an error in loading any entity, it will be slightly tricky but we should preserve the same behaviour in case of parallel uploads via threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)