[jira] [Resolved] (KAFKA-1998) Partitions Missing From MetadataResponse

["Guozhang Wang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Guozhang Wang resolved KAFKA-1998.
----------------------------------
    Resolution: Not a Problem

> Partitions Missing From MetadataResponse
> ----------------------------------------
>
>                 Key: KAFKA-1998
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1998
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.8.2.0
>            Reporter: Evan Huus
>            Assignee: Mayuresh Gharat
>
> It is known behaviour that when a partition is entirely offline (it has no leader because all of its replicas are down) then that partition will not be included in the metadata returned by other brokers. For example, if topic "foo" has 3 partitions, but all replicas of partition 3 are offline, then requesting metadata for "foo" will only return information about partitions 1 and 2.
> This means that there is no way to reliably determine the number of partitions for a topic via kafka's metadata API; if I receive information on partitions 1 and 2, I don't know if partition 3 is offline or if it is simply that there are only two partitions total. (You can presumably still ask zookeeper directly, but that is a work-around).
> This ambiguity, in turn, can lead to a consistency problem with the default partitioner, since that effectively implements `hash(key) mod #partitions`. If a partition goes offline and is removed from the metadata response, then the number of partitions the producer knows about will change (on its next metadata refresh) and the mapping from keys to partitions will also change. Instead of distributing messages among (for example) 3 partitions, and failing to produce to the offline partition, it will distribute *all* messages among the two online partitions. This results in messages being sent to the wrong partition.
> Since kafka already returns partitions with error messages in many cases (e.g. `LeaderNotAvailable`) I think it makes much more sense and fixes the above partition problem if it would simply return offline partitions as well with the appropriate error (whether that is `LeaderNotAvailable` or it would be better to add an additional error is up to you).
> CC [~guozhang]
> (This issue was originally described/discussed on the kafka-users mailing list, in the thread involving https://mail-archives.apache.org/mod_mbox/kafka-users/201503.mbox/%3CCAA4pprAZvp2XhdNmy0%2BqVZ1UVdVxmUfz3DDArhGbwP-iiH%2BGyg%40mail.gmail.com%3E)
> If there are any questions I am happy to clarify, I realize the scenario is somewhat complex.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)