[jira] [Updated] (KAFKA-1677) Governor on concurrent replica reassignments

["Neha Narkhede (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede updated KAFKA-1677:
---------------------------------
    Labels: newbie++  (was: )

> Governor on concurrent replica reassignments
> --------------------------------------------
>
>                 Key: KAFKA-1677
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1677
>             Project: Kafka
>          Issue Type: New Feature
>          Components: controller
>            Reporter: Ryan Berdeen
>            Assignee: Neha Narkhede
>              Labels: newbie++
>
> We have seen a cluster be killed via too many concurrent partition transfers. An ideal solution is a configuration setting to limit the number of concurrent transfers per host (dynamically tunable). (eg: transfer_limit defined in http://docs.basho.com/riak/latest/ops/advanced/configs/configuration-files/#Ring).
> To work around this, we generate our assignments, then use a tool to feed the reassignments in small batches.
> The size of the batch is based on either
> * *the number partitions*, e.g., reassign all replicas for the first 2 partitions that have any moves
> * *the number of individual replica moves*, e.g. when reassigning \[1,2,3,4] to \[5,6,7,8], first reassign to \[5,6,3,4] then reassign to \[5,6,7,8]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)