You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Neha Narkhede <ne...@gmail.com> on 2013/12/11 07:30:56 UTC

Review Request 16175: Patch for KAFKA-1178

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16175/
-----------------------------------------------------------

Review request for kafka.


Bugs: KAFKA-1178
    https://issues.apache.org/jira/browse/KAFKA-1178


Repository: kafka


Description
-------

1. Fixed broker to only follow become-follower/leader request if the broker exists in the assigned replica list 2. Changed the under replicated check to compare in sync replica set size with assigned replica set size instead of passed in replication factor 3. Fixed the fetcher id computation to not create more fetcher threads than necessary due to the negative (overflown) hashcode value. 4. Fixed some more state change logging 5. Tested partition reassignment several times on a cluster with several 100s of topics, verified reassignment completion followed by no under replicated partitions (through jmx and through zookeeper)


Diffs
-----

  core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 65c04edc11a1eb2771bfd741d59886b5fef9ccbc 
  core/src/main/scala/kafka/cluster/Partition.scala 13f48baa1bc44533c33153fde6f1c03a067d83bc 
  core/src/main/scala/kafka/server/AbstractFetcherManager.scala 394e981c858fa53a28d3c7fcef0afbd4faf07858 
  core/src/main/scala/kafka/server/ReplicaManager.scala b0a39622ab91e209f9657231917a5ae30fd835f0 

Diff: https://reviews.apache.org/r/16175/diff/


Testing
-------


Thanks,

Neha Narkhede


Re: Review Request 16175: Patch for KAFKA-1178

Posted by Neha Narkhede <ne...@gmail.com>.

> On Dec. 11, 2013, 5:48 p.m., Jun Rao wrote:
> > Thanks for the patch.
> > 
> > 1. In Controller.updateLeaderEpochAndSendRequest(), the following comment seems not accurate. Could we fix it?
> >         // send the shrunk assigned replica list to all the replicas, including the leader, so that it no longer
> >         // allows old replicas to enter ISR
> > 
> > 2. Just a question, in Controller.onPartitionReassignment(), after calling updateAssignedReplicasForPartition(), we need to send LeaderAndIsr requests to propagate the new assigned replica list to the brokers, right? Where does this happen? Couldn't seem to find it in the code.
> > 
> > 3. In Controller.onPartitionReassignment(), under "case false", it seems that we can update the assigned replicas and leader/isr path in ZK unnecessarily since this method will be called every time ISR changes.

1. fixed
2. It is sent in updateLeaderEpochAndSendRequest()
3. Not really. It is only invoked if all reassigned replicas have caught up


- Neha


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16175/#review30204
-----------------------------------------------------------


On Dec. 11, 2013, 6:30 a.m., Neha Narkhede wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16175/
> -----------------------------------------------------------
> 
> (Updated Dec. 11, 2013, 6:30 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1178
>     https://issues.apache.org/jira/browse/KAFKA-1178
> 
> 
> Repository: kafka
> 
> 
> Description
> -------
> 
> 1. Fixed broker to only follow become-follower/leader request if the broker exists in the assigned replica list 2. Changed the under replicated check to compare in sync replica set size with assigned replica set size instead of passed in replication factor 3. Fixed the fetcher id computation to not create more fetcher threads than necessary due to the negative (overflown) hashcode value. 4. Fixed some more state change logging 5. Tested partition reassignment several times on a cluster with several 100s of topics, verified reassignment completion followed by no under replicated partitions (through jmx and through zookeeper)
> 
> 
> Diffs
> -----
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 65c04edc11a1eb2771bfd741d59886b5fef9ccbc 
>   core/src/main/scala/kafka/cluster/Partition.scala 13f48baa1bc44533c33153fde6f1c03a067d83bc 
>   core/src/main/scala/kafka/server/AbstractFetcherManager.scala 394e981c858fa53a28d3c7fcef0afbd4faf07858 
>   core/src/main/scala/kafka/server/ReplicaManager.scala b0a39622ab91e209f9657231917a5ae30fd835f0 
> 
> Diff: https://reviews.apache.org/r/16175/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>


Re: Review Request 16175: Patch for KAFKA-1178

Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16175/#review30204
-----------------------------------------------------------


Thanks for the patch.

1. In Controller.updateLeaderEpochAndSendRequest(), the following comment seems not accurate. Could we fix it?
        // send the shrunk assigned replica list to all the replicas, including the leader, so that it no longer
        // allows old replicas to enter ISR

2. Just a question, in Controller.onPartitionReassignment(), after calling updateAssignedReplicasForPartition(), we need to send LeaderAndIsr requests to propagate the new assigned replica list to the brokers, right? Where does this happen? Couldn't seem to find it in the code.

3. In Controller.onPartitionReassignment(), under "case false", it seems that we can update the assigned replicas and leader/isr path in ZK unnecessarily since this method will be called every time ISR changes.

- Jun Rao


On Dec. 11, 2013, 6:30 a.m., Neha Narkhede wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16175/
> -----------------------------------------------------------
> 
> (Updated Dec. 11, 2013, 6:30 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1178
>     https://issues.apache.org/jira/browse/KAFKA-1178
> 
> 
> Repository: kafka
> 
> 
> Description
> -------
> 
> 1. Fixed broker to only follow become-follower/leader request if the broker exists in the assigned replica list 2. Changed the under replicated check to compare in sync replica set size with assigned replica set size instead of passed in replication factor 3. Fixed the fetcher id computation to not create more fetcher threads than necessary due to the negative (overflown) hashcode value. 4. Fixed some more state change logging 5. Tested partition reassignment several times on a cluster with several 100s of topics, verified reassignment completion followed by no under replicated partitions (through jmx and through zookeeper)
> 
> 
> Diffs
> -----
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 65c04edc11a1eb2771bfd741d59886b5fef9ccbc 
>   core/src/main/scala/kafka/cluster/Partition.scala 13f48baa1bc44533c33153fde6f1c03a067d83bc 
>   core/src/main/scala/kafka/server/AbstractFetcherManager.scala 394e981c858fa53a28d3c7fcef0afbd4faf07858 
>   core/src/main/scala/kafka/server/ReplicaManager.scala b0a39622ab91e209f9657231917a5ae30fd835f0 
> 
> Diff: https://reviews.apache.org/r/16175/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>


Re: Review Request 16175: Patch for KAFKA-1178

Posted by Neha Narkhede <ne...@gmail.com>.

> On Dec. 11, 2013, 5:08 p.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/server/ReplicaManager.scala, line 237
> > <https://reviews.apache.org/r/16175/diff/1/?file=396497#file396497line237>
> >
> >     Under which condition the broker will not be in the assigned replica list? Since the controller have full control of sending LeaderAndIsr request to brokers, this case sounds more like a bug to me.

During partition reassignment, when new set of replicas are communicated, the request is sent to all leaders and followers. But it can happen that during the shrinking of the replication factor, a follower is not in the assigned replica list (for a small period of time). This protects against that case


- Neha


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16175/#review30202
-----------------------------------------------------------


On Dec. 11, 2013, 6:30 a.m., Neha Narkhede wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16175/
> -----------------------------------------------------------
> 
> (Updated Dec. 11, 2013, 6:30 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1178
>     https://issues.apache.org/jira/browse/KAFKA-1178
> 
> 
> Repository: kafka
> 
> 
> Description
> -------
> 
> 1. Fixed broker to only follow become-follower/leader request if the broker exists in the assigned replica list 2. Changed the under replicated check to compare in sync replica set size with assigned replica set size instead of passed in replication factor 3. Fixed the fetcher id computation to not create more fetcher threads than necessary due to the negative (overflown) hashcode value. 4. Fixed some more state change logging 5. Tested partition reassignment several times on a cluster with several 100s of topics, verified reassignment completion followed by no under replicated partitions (through jmx and through zookeeper)
> 
> 
> Diffs
> -----
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 65c04edc11a1eb2771bfd741d59886b5fef9ccbc 
>   core/src/main/scala/kafka/cluster/Partition.scala 13f48baa1bc44533c33153fde6f1c03a067d83bc 
>   core/src/main/scala/kafka/server/AbstractFetcherManager.scala 394e981c858fa53a28d3c7fcef0afbd4faf07858 
>   core/src/main/scala/kafka/server/ReplicaManager.scala b0a39622ab91e209f9657231917a5ae30fd835f0 
> 
> Diff: https://reviews.apache.org/r/16175/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>


Re: Review Request 16175: Patch for KAFKA-1178

Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16175/#review30202
-----------------------------------------------------------



core/src/main/scala/kafka/server/ReplicaManager.scala
<https://reviews.apache.org/r/16175/#comment57813>

    Under which condition the broker will not be in the assigned replica list? Since the controller have full control of sending LeaderAndIsr request to brokers, this case sounds more like a bug to me.


- Guozhang Wang


On Dec. 11, 2013, 6:30 a.m., Neha Narkhede wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16175/
> -----------------------------------------------------------
> 
> (Updated Dec. 11, 2013, 6:30 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1178
>     https://issues.apache.org/jira/browse/KAFKA-1178
> 
> 
> Repository: kafka
> 
> 
> Description
> -------
> 
> 1. Fixed broker to only follow become-follower/leader request if the broker exists in the assigned replica list 2. Changed the under replicated check to compare in sync replica set size with assigned replica set size instead of passed in replication factor 3. Fixed the fetcher id computation to not create more fetcher threads than necessary due to the negative (overflown) hashcode value. 4. Fixed some more state change logging 5. Tested partition reassignment several times on a cluster with several 100s of topics, verified reassignment completion followed by no under replicated partitions (through jmx and through zookeeper)
> 
> 
> Diffs
> -----
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 65c04edc11a1eb2771bfd741d59886b5fef9ccbc 
>   core/src/main/scala/kafka/cluster/Partition.scala 13f48baa1bc44533c33153fde6f1c03a067d83bc 
>   core/src/main/scala/kafka/server/AbstractFetcherManager.scala 394e981c858fa53a28d3c7fcef0afbd4faf07858 
>   core/src/main/scala/kafka/server/ReplicaManager.scala b0a39622ab91e209f9657231917a5ae30fd835f0 
> 
> Diff: https://reviews.apache.org/r/16175/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>


Re: Review Request 16175: Patch for KAFKA-1178

Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16175/#review30214
-----------------------------------------------------------

Ship it!


Looks good. I would be good to expand the comments before KafkaController.onPartitionReassignment() with a bit more details such as whether LeaderAndIsr requests are sent, to which brokers, and the assigned replica set in the requests.

- Jun Rao


On Dec. 11, 2013, 6:30 p.m., Neha Narkhede wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16175/
> -----------------------------------------------------------
> 
> (Updated Dec. 11, 2013, 6:30 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1178
>     https://issues.apache.org/jira/browse/KAFKA-1178
> 
> 
> Repository: kafka
> 
> 
> Description
> -------
> 
> Addressed review comments regarding comments, improved unit tests to check for under replicated partition count post reassignment
> 
> 
> 1. Fixed broker to only follow become-follower/leader request if the broker exists in the assigned replica list 2. Changed the under replicated check to compare in sync replica set size with assigned replica set size instead of passed in replication factor 3. Fixed the fetcher id computation to not create more fetcher threads than necessary due to the negative (overflown) hashcode value. 4. Fixed some more state change logging 5. Tested partition reassignment several times on a cluster with several 100s of topics, verified reassignment completion followed by no under replicated partitions (through jmx and through zookeeper)
> 
> 
> Diffs
> -----
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 65c04edc11a1eb2771bfd741d59886b5fef9ccbc 
>   core/src/main/scala/kafka/cluster/Partition.scala 13f48baa1bc44533c33153fde6f1c03a067d83bc 
>   core/src/main/scala/kafka/controller/KafkaController.scala fd92c65f1d067d1bf7096df41beb652edaceec87 
>   core/src/main/scala/kafka/server/AbstractFetcherManager.scala 394e981c858fa53a28d3c7fcef0afbd4faf07858 
>   core/src/main/scala/kafka/server/ReplicaManager.scala b0a39622ab91e209f9657231917a5ae30fd835f0 
>   core/src/test/scala/unit/kafka/admin/AdminTest.scala 86f88f5f06ecc2cb1c8d5436db93b36ccea9e938 
> 
> Diff: https://reviews.apache.org/r/16175/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>


Re: Review Request 16175: Patch for KAFKA-1178

Posted by Neha Narkhede <ne...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16175/
-----------------------------------------------------------

(Updated Dec. 11, 2013, 7:09 p.m.)


Review request for kafka.


Bugs: KAFKA-1178
    https://issues.apache.org/jira/browse/KAFKA-1178


Repository: kafka


Description (updated)
-------

Improved the documentation of onPartitionReassignment() API to list the different steps in partition reassignment in detail


Addressed review comments regarding comments, improved unit tests to check for under replicated partition count post reassignment


1. Fixed broker to only follow become-follower/leader request if the broker exists in the assigned replica list 2. Changed the under replicated check to compare in sync replica set size with assigned replica set size instead of passed in replication factor 3. Fixed the fetcher id computation to not create more fetcher threads than necessary due to the negative (overflown) hashcode value. 4. Fixed some more state change logging 5. Tested partition reassignment several times on a cluster with several 100s of topics, verified reassignment completion followed by no under replicated partitions (through jmx and through zookeeper)


Diffs (updated)
-----

  core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 65c04edc11a1eb2771bfd741d59886b5fef9ccbc 
  core/src/main/scala/kafka/cluster/Partition.scala 13f48baa1bc44533c33153fde6f1c03a067d83bc 
  core/src/main/scala/kafka/controller/KafkaController.scala fd92c65f1d067d1bf7096df41beb652edaceec87 
  core/src/main/scala/kafka/server/AbstractFetcherManager.scala 394e981c858fa53a28d3c7fcef0afbd4faf07858 
  core/src/main/scala/kafka/server/ReplicaManager.scala b0a39622ab91e209f9657231917a5ae30fd835f0 
  core/src/test/scala/unit/kafka/admin/AdminTest.scala 86f88f5f06ecc2cb1c8d5436db93b36ccea9e938 

Diff: https://reviews.apache.org/r/16175/diff/


Testing
-------


Thanks,

Neha Narkhede


Re: Review Request 16175: Patch for KAFKA-1178

Posted by Neha Narkhede <ne...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16175/
-----------------------------------------------------------

(Updated Dec. 11, 2013, 6:30 p.m.)


Review request for kafka.


Bugs: KAFKA-1178
    https://issues.apache.org/jira/browse/KAFKA-1178


Repository: kafka


Description (updated)
-------

Addressed review comments regarding comments, improved unit tests to check for under replicated partition count post reassignment


1. Fixed broker to only follow become-follower/leader request if the broker exists in the assigned replica list 2. Changed the under replicated check to compare in sync replica set size with assigned replica set size instead of passed in replication factor 3. Fixed the fetcher id computation to not create more fetcher threads than necessary due to the negative (overflown) hashcode value. 4. Fixed some more state change logging 5. Tested partition reassignment several times on a cluster with several 100s of topics, verified reassignment completion followed by no under replicated partitions (through jmx and through zookeeper)


Diffs (updated)
-----

  core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 65c04edc11a1eb2771bfd741d59886b5fef9ccbc 
  core/src/main/scala/kafka/cluster/Partition.scala 13f48baa1bc44533c33153fde6f1c03a067d83bc 
  core/src/main/scala/kafka/controller/KafkaController.scala fd92c65f1d067d1bf7096df41beb652edaceec87 
  core/src/main/scala/kafka/server/AbstractFetcherManager.scala 394e981c858fa53a28d3c7fcef0afbd4faf07858 
  core/src/main/scala/kafka/server/ReplicaManager.scala b0a39622ab91e209f9657231917a5ae30fd835f0 
  core/src/test/scala/unit/kafka/admin/AdminTest.scala 86f88f5f06ecc2cb1c8d5436db93b36ccea9e938 

Diff: https://reviews.apache.org/r/16175/diff/


Testing
-------


Thanks,

Neha Narkhede