[jira] [Commented] (FALCON-25) Resource Manager

["Nikhil Mulley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FALCON-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675875#comment-13675875 ] 

Nikhil Mulley commented on FALCON-25:
-------------------------------------

Hi Suhas,

Nice initiative. 

Just a couple of questions:
a) is the amount of bandwidth as the only resource that will be controlled via tokens or even the number of maps in execution?
b) when we say effective use of bandwidth available, are the tokens issued only after the completion of the existing jobs or also while they are in progress and perhaps relinquishing some of the bandwidth already in the midst(or at end) of their transferrs.
 lets say, after acquiring the token, a data transferr (distcp) job has run for a while and only couple of maps are in execution meaning to say only small amount of data transferr needs to be complete, which means job may no longer requires the total bandwidth that has been allocated to it during its initial run, would be left unused while the other transferr jobs are still waiting for the token to be issued.
c) how will you know the bandwidth totally available for the transferrs between different cluster(s) or data center(s), will this be a static property or dynamically detected from the network management resource(s)?
d) while distcp (now) has got options for throttling bandwidth at MB/KB level after recent changes from Srikanth, would this be inline with resource(bandwidth) management?


thanks

Nikhil
                
> Resource Manager
> ----------------
>
>                 Key: FALCON-25
>                 URL: https://issues.apache.org/jira/browse/FALCON-25
>             Project: Falcon
>          Issue Type: Improvement
>          Components: replication
>            Reporter: Suhas Vasu
>            Priority: Minor
>
> We need a way where we can control replication so that we can make effective use of the bandwidth available. We can introduce a resource manager like zoo-keeper and have a counter/semaphore through which we can allocate resource to a replication job. The job acquires maps only if it is allocated resource from the resource manager. 
> This way we can ensure good transfer rates as well as avoid jobs starving for bandwidth & efficient use of mappers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira