You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2012/07/26 19:13:34 UTC
[jira] [Created] (KAFKA-420) maintain HW correctly with only 1
replica in ISR
Jun Rao created KAFKA-420:
-----------------------------
Summary: maintain HW correctly with only 1 replica in ISR
Key: KAFKA-420
URL: https://issues.apache.org/jira/browse/KAFKA-420
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8
Reporter: Jun Rao
Currently, the HW maintenance logic is only triggered when handling fetch requests from the follower. As a result, if the ISR has only 1 replica, the HW won't be incremented since there is no request from the follower to trigger the maintenance logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-420) maintain HW correctly with only 1
replica in ISR
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-420:
--------------------------
Attachment: kafka-420_v1.patch
Attach patch v1.
This is a simpler fix. Instead of waiting for data flushed to disk, this patch just advances the HW after the data is in memory. If a user cares about durability, he can always increase the replication factor.
Patched an existing unit test to expose the problem. Patched 3 places where we may need to increment the HW: (1) new produce requests coming to the leader; (2) ISR shrinks; (3) a replica becomes the leader.
> maintain HW correctly with only 1 replica in ISR
> ------------------------------------------------
>
> Key: KAFKA-420
> URL: https://issues.apache.org/jira/browse/KAFKA-420
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: bugs
> Attachments: kafka-420_v1.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Currently, the HW maintenance logic is only triggered when handling fetch requests from the follower. As a result, if the ISR has only 1 replica, the HW won't be incremented since there is no request from the follower to trigger the maintenance logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-420) maintain HW correctly with only 1
replica in ISR
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao reassigned KAFKA-420:
-----------------------------
Assignee: Jun Rao
> maintain HW correctly with only 1 replica in ISR
> ------------------------------------------------
>
> Key: KAFKA-420
> URL: https://issues.apache.org/jira/browse/KAFKA-420
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Labels: bugs
> Attachments: kafka-420_v1.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Currently, the HW maintenance logic is only triggered when handling fetch requests from the follower. As a result, if the ISR has only 1 replica, the HW won't be incremented since there is no request from the follower to trigger the maintenance logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-420) maintain HW correctly with only 1
replica in ISR
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440878#comment-13440878 ]
Neha Narkhede commented on KAFKA-420:
-------------------------------------
Looks good. +1
> maintain HW correctly with only 1 replica in ISR
> ------------------------------------------------
>
> Key: KAFKA-420
> URL: https://issues.apache.org/jira/browse/KAFKA-420
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Labels: bugs
> Attachments: kafka-420_v1.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Currently, the HW maintenance logic is only triggered when handling fetch requests from the follower. As a result, if the ISR has only 1 replica, the HW won't be incremented since there is no request from the follower to trigger the maintenance logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-420) maintain HW correctly with only 1
replica in ISR
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423248#comment-13423248 ]
Jun Rao commented on KAFKA-420:
-------------------------------
First of all, we need to think through what message commits mean when there is only 1 replica in ISR. Does it mean message is in memory in the only replica or do we have to wait for the message to be flushed to disk. It seems to me that the latter is probably more meaningful and is compatible with what we have in 0.7. If the latter is what we want to implement, we will need to add a call to ReplicaManager.maybeIncrementLeaderHW from the log flushing logic.
> maintain HW correctly with only 1 replica in ISR
> ------------------------------------------------
>
> Key: KAFKA-420
> URL: https://issues.apache.org/jira/browse/KAFKA-420
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Currently, the HW maintenance logic is only triggered when handling fetch requests from the follower. As a result, if the ISR has only 1 replica, the HW won't be incremented since there is no request from the follower to trigger the maintenance logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-420) maintain HW correctly with only 1
replica in ISR
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-420:
-----------------------------
Labels: bugs (was: )
> maintain HW correctly with only 1 replica in ISR
> ------------------------------------------------
>
> Key: KAFKA-420
> URL: https://issues.apache.org/jira/browse/KAFKA-420
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: bugs
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Currently, the HW maintenance logic is only triggered when handling fetch requests from the follower. As a result, if the ISR has only 1 replica, the HW won't be incremented since there is no request from the follower to trigger the maintenance logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira