You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Matt (Created) (JIRA)" <ji...@apache.org> on 2012/02/17 23:58:57 UTC
[jira] [Created] (KAFKA-278) Topic is never distributed to nodes
added to an existing cluster
Topic is never distributed to nodes added to an existing cluster
----------------------------------------------------------------
Key: KAFKA-278
URL: https://issues.apache.org/jira/browse/KAFKA-278
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.7
Reporter: Matt
Priority: Minor
If you add one or more brokers to an existing cluster, existing topics will never be seen by the new brokers.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send messages for a topic to the cluster.
3) Add a new broker to the cluster.
4) New broker will never see the existing topic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Updated] (KAFKA-278) Issues partitioning a new topic
Posted by Prashanth Menon <pr...@gmail.com>.
I can take a look at it this weekend :)
- Prashanth
On Fri, Apr 20, 2012 at 10:17 AM, Jun Rao <ju...@gmail.com> wrote:
> This jira may not be hard to fix since the required changes are likely
> isolated in ZKBrokerPartitionInfo.getBrokerPartitionInfo(). Basically, we
> just need to bootstrap a broker if it doesn't have a topic already. Anyone
> interested in taking a shot at it?
>
> Thanks,
>
> Jun
>
> On Thu, Apr 19, 2012 at 7:17 PM, Bao Thai Ngo <ba...@gmail.com>
> wrote:
>
> > Hi there,
> >
> > Is there any news on this issue?
> >
> > Thanks,
> > ~Thai
> >
> > On Fri, Mar 16, 2012 at 8:04 AM, Matt (Updated) (JIRA) <jira@apache.org
> > >wrote:
> >
> > >
> > > [
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> > ]
> > >
> > > Matt updated KAFKA-278:
> > > -----------------------
> > >
> > > Description:
> > > There are two cases where correct partitioning fails for a new topic.
> > >
> > > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> > > the cluster. The new broker will never host partitions for the existing
> > > topic.
> > >
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages for a topic to the cluster.
> > > 3) Add a new broker to the cluster.
> > > 4) New broker will never see the existing topic.
> > >
> > > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > > messages to a new topic that did not previously exist in the cluster.
> If,
> > > during the producer session, one or more partitions are not created on
> a
> > > broker, the broker will never host those partitions.
> > >
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages to a new topic.
> > > 3) Shut down the producer before the topic is created on at least one
> > > broker.
> > > 4) The broker that did not allocate the topic will never host the
> topic.
> > >
> > > My guess(!) here is that when a new producer is created, it gets a list
> > of
> > > topics and partitions based on the current state of the brokers in the
> > > cluster. Since some brokers are missing the topic, the producer will
> > never
> > > send messages to that broker and partitions will never be created.
> > >
> > >
> > > Work around:
> > > Manually create the topic/partition directories in the kafka logs
> > > directory and reboot kafka. It will register the topic/partitions in
> > > ZooKeeper.
> > >
> > > was:
> > > There are two cases where correct partitioning fails for a new topic.
> > >
> > > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> > > the cluster. The new broker will never host partitions for the existing
> > > topic.
> > >
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages for a topic to the cluster.
> > > 3) Add a new broker to the cluster.
> > > 4) New broker will never see the existing topic.
> > >
> > > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > > messages to a new topic that did not previously exist in the cluster.
> If,
> > > during the producer session, one or more partitions are not created on
> a
> > > broker, the broker will never host those partitions.
> > >
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages to a new topic.
> > > 3) Shut down the producer before the topic is created on at least one
> > > broker.
> > > 4) The broker that did not allocate the topic will never host the
> topic.
> > >
> > > My guess(!) here is that when a new producer is created, it gets a list
> > of
> > > topics and partitions based on the current state of the brokers in the
> > > cluster. Since some brokers are missing the topic, the producer will
> > never
> > > send messages to that broker and partitions will never be created.
> > >
> > >
> > > Added a work around.
> > >
> > > > Issues partitioning a new topic
> > > > -------------------------------
> > > >
> > > > Key: KAFKA-278
> > > > URL: https://issues.apache.org/jira/browse/KAFKA-278
> > > > Project: Kafka
> > > > Issue Type: Bug
> > > > Components: core
> > > > Affects Versions: 0.7
> > > > Reporter: Matt
> > > > Priority: Minor
> > > >
> > > > There are two cases where correct partitioning fails for a new topic.
> > > > Case 1: Topic exists on current Kafka cluster. A new broker is added
> to
> > > the cluster. The new broker will never host partitions for the existing
> > > topic.
> > > > To reproduce:
> > > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > > 2) Send messages for a topic to the cluster.
> > > > 3) Add a new broker to the cluster.
> > > > 4) New broker will never see the existing topic.
> > > > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > > messages to a new topic that did not previously exist in the cluster.
> If,
> > > during the producer session, one or more partitions are not created on
> a
> > > broker, the broker will never host those partitions.
> > > > To reproduce:
> > > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > > 2) Send messages to a new topic.
> > > > 3) Shut down the producer before the topic is created on at least one
> > > broker.
> > > > 4) The broker that did not allocate the topic will never host the
> > topic.
> > > > My guess(!) here is that when a new producer is created, it gets a
> list
> > > of topics and partitions based on the current state of the brokers in
> the
> > > cluster. Since some brokers are missing the topic, the producer will
> > never
> > > send messages to that broker and partitions will never be created.
> > > > Work around:
> > > > Manually create the topic/partition directories in the kafka logs
> > > directory and reboot kafka. It will register the topic/partitions in
> > > ZooKeeper.
> > >
> > > --
> > > This message is automatically generated by JIRA.
> > > If you think it was sent incorrectly, please contact your JIRA
> > > administrators:
> > >
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> > > For more information on JIRA, see:
> > http://www.atlassian.com/software/jira
> > >
> > >
> > >
> >
>
Re: [jira] [Updated] (KAFKA-278) Issues partitioning a new topic
Posted by Jun Rao <ju...@gmail.com>.
This jira may not be hard to fix since the required changes are likely
isolated in ZKBrokerPartitionInfo.getBrokerPartitionInfo(). Basically, we
just need to bootstrap a broker if it doesn't have a topic already. Anyone
interested in taking a shot at it?
Thanks,
Jun
On Thu, Apr 19, 2012 at 7:17 PM, Bao Thai Ngo <ba...@gmail.com> wrote:
> Hi there,
>
> Is there any news on this issue?
>
> Thanks,
> ~Thai
>
> On Fri, Mar 16, 2012 at 8:04 AM, Matt (Updated) (JIRA) <jira@apache.org
> >wrote:
>
> >
> > [
> >
> https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
> >
> > Matt updated KAFKA-278:
> > -----------------------
> >
> > Description:
> > There are two cases where correct partitioning fails for a new topic.
> >
> > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> > the cluster. The new broker will never host partitions for the existing
> > topic.
> >
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages for a topic to the cluster.
> > 3) Add a new broker to the cluster.
> > 4) New broker will never see the existing topic.
> >
> > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > messages to a new topic that did not previously exist in the cluster. If,
> > during the producer session, one or more partitions are not created on a
> > broker, the broker will never host those partitions.
> >
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages to a new topic.
> > 3) Shut down the producer before the topic is created on at least one
> > broker.
> > 4) The broker that did not allocate the topic will never host the topic.
> >
> > My guess(!) here is that when a new producer is created, it gets a list
> of
> > topics and partitions based on the current state of the brokers in the
> > cluster. Since some brokers are missing the topic, the producer will
> never
> > send messages to that broker and partitions will never be created.
> >
> >
> > Work around:
> > Manually create the topic/partition directories in the kafka logs
> > directory and reboot kafka. It will register the topic/partitions in
> > ZooKeeper.
> >
> > was:
> > There are two cases where correct partitioning fails for a new topic.
> >
> > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> > the cluster. The new broker will never host partitions for the existing
> > topic.
> >
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages for a topic to the cluster.
> > 3) Add a new broker to the cluster.
> > 4) New broker will never see the existing topic.
> >
> > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > messages to a new topic that did not previously exist in the cluster. If,
> > during the producer session, one or more partitions are not created on a
> > broker, the broker will never host those partitions.
> >
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages to a new topic.
> > 3) Shut down the producer before the topic is created on at least one
> > broker.
> > 4) The broker that did not allocate the topic will never host the topic.
> >
> > My guess(!) here is that when a new producer is created, it gets a list
> of
> > topics and partitions based on the current state of the brokers in the
> > cluster. Since some brokers are missing the topic, the producer will
> never
> > send messages to that broker and partitions will never be created.
> >
> >
> > Added a work around.
> >
> > > Issues partitioning a new topic
> > > -------------------------------
> > >
> > > Key: KAFKA-278
> > > URL: https://issues.apache.org/jira/browse/KAFKA-278
> > > Project: Kafka
> > > Issue Type: Bug
> > > Components: core
> > > Affects Versions: 0.7
> > > Reporter: Matt
> > > Priority: Minor
> > >
> > > There are two cases where correct partitioning fails for a new topic.
> > > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> > the cluster. The new broker will never host partitions for the existing
> > topic.
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages for a topic to the cluster.
> > > 3) Add a new broker to the cluster.
> > > 4) New broker will never see the existing topic.
> > > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > messages to a new topic that did not previously exist in the cluster. If,
> > during the producer session, one or more partitions are not created on a
> > broker, the broker will never host those partitions.
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages to a new topic.
> > > 3) Shut down the producer before the topic is created on at least one
> > broker.
> > > 4) The broker that did not allocate the topic will never host the
> topic.
> > > My guess(!) here is that when a new producer is created, it gets a list
> > of topics and partitions based on the current state of the brokers in the
> > cluster. Since some brokers are missing the topic, the producer will
> never
> > send messages to that broker and partitions will never be created.
> > > Work around:
> > > Manually create the topic/partition directories in the kafka logs
> > directory and reboot kafka. It will register the topic/partitions in
> > ZooKeeper.
> >
> > --
> > This message is automatically generated by JIRA.
> > If you think it was sent incorrectly, please contact your JIRA
> > administrators:
> > https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
> >
> >
>
Re: [jira] [Updated] (KAFKA-278) Issues partitioning a new topic
Posted by Bao Thai Ngo <ba...@gmail.com>.
Hi there,
Is there any news on this issue?
Thanks,
~Thai
On Fri, Mar 16, 2012 at 8:04 AM, Matt (Updated) (JIRA) <ji...@apache.org>wrote:
>
> [
> https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Matt updated KAFKA-278:
> -----------------------
>
> Description:
> There are two cases where correct partitioning fails for a new topic.
>
> Case 1: Topic exists on current Kafka cluster. A new broker is added to
> the cluster. The new broker will never host partitions for the existing
> topic.
>
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages for a topic to the cluster.
> 3) Add a new broker to the cluster.
> 4) New broker will never see the existing topic.
>
> Case 2: Topic does not exist on current Kafka cluster. Producer sends
> messages to a new topic that did not previously exist in the cluster. If,
> during the producer session, one or more partitions are not created on a
> broker, the broker will never host those partitions.
>
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages to a new topic.
> 3) Shut down the producer before the topic is created on at least one
> broker.
> 4) The broker that did not allocate the topic will never host the topic.
>
> My guess(!) here is that when a new producer is created, it gets a list of
> topics and partitions based on the current state of the brokers in the
> cluster. Since some brokers are missing the topic, the producer will never
> send messages to that broker and partitions will never be created.
>
>
> Work around:
> Manually create the topic/partition directories in the kafka logs
> directory and reboot kafka. It will register the topic/partitions in
> ZooKeeper.
>
> was:
> There are two cases where correct partitioning fails for a new topic.
>
> Case 1: Topic exists on current Kafka cluster. A new broker is added to
> the cluster. The new broker will never host partitions for the existing
> topic.
>
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages for a topic to the cluster.
> 3) Add a new broker to the cluster.
> 4) New broker will never see the existing topic.
>
> Case 2: Topic does not exist on current Kafka cluster. Producer sends
> messages to a new topic that did not previously exist in the cluster. If,
> during the producer session, one or more partitions are not created on a
> broker, the broker will never host those partitions.
>
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages to a new topic.
> 3) Shut down the producer before the topic is created on at least one
> broker.
> 4) The broker that did not allocate the topic will never host the topic.
>
> My guess(!) here is that when a new producer is created, it gets a list of
> topics and partitions based on the current state of the brokers in the
> cluster. Since some brokers are missing the topic, the producer will never
> send messages to that broker and partitions will never be created.
>
>
> Added a work around.
>
> > Issues partitioning a new topic
> > -------------------------------
> >
> > Key: KAFKA-278
> > URL: https://issues.apache.org/jira/browse/KAFKA-278
> > Project: Kafka
> > Issue Type: Bug
> > Components: core
> > Affects Versions: 0.7
> > Reporter: Matt
> > Priority: Minor
> >
> > There are two cases where correct partitioning fails for a new topic.
> > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> the cluster. The new broker will never host partitions for the existing
> topic.
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages for a topic to the cluster.
> > 3) Add a new broker to the cluster.
> > 4) New broker will never see the existing topic.
> > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> messages to a new topic that did not previously exist in the cluster. If,
> during the producer session, one or more partitions are not created on a
> broker, the broker will never host those partitions.
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages to a new topic.
> > 3) Shut down the producer before the topic is created on at least one
> broker.
> > 4) The broker that did not allocate the topic will never host the topic.
> > My guess(!) here is that when a new producer is created, it gets a list
> of topics and partitions based on the current state of the brokers in the
> cluster. Since some brokers are missing the topic, the producer will never
> send messages to that broker and partitions will never be created.
> > Work around:
> > Manually create the topic/partition directories in the kafka logs
> directory and reboot kafka. It will register the topic/partitions in
> ZooKeeper.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
[jira] [Updated] (KAFKA-278) Issues partitioning a new topic
Posted by "Matt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt updated KAFKA-278:
-----------------------
Description:
There are two cases where correct partitioning fails for a new topic.
Case 1: Topic exists on current Kafka cluster. A new broker is added to the cluster. The new broker will never host partitions for the existing topic.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send messages for a topic to the cluster.
3) Add a new broker to the cluster.
4) New broker will never see the existing topic.
Case 2: Topic does not exist on current Kafka cluster. Producer sends messages to a new topic that did not previously exist in the cluster. If, during the producer session, one or more partitions are not created on a broker, the broker will never host those partitions.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send messages to a new topic.
3) Shut down the producer before the topic is created on at least one broker.
4) The broker that did not allocate the topic will never host the topic.
My guess(!) here is that when a new producer is created, it gets a list of topics and partitions based on the current state of the brokers in the cluster. Since some brokers are missing the topic, the producer will never send messages to that broker and partitions will never be created.
was:
There are two cases where correct partitioning fails for a new topic.
Case 1: Topic exists on current Kafka cluster. A newly added broker to the cluster will never be notified of the topic and will never host partitions.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send messages for a topic to the cluster.
3) Add a new broker to the cluster.
4) New broker will never see the existing topic.
Case 2: Topic does not exist on current Kafka cluster. Session in which producer sends first set of messages for new topic does not last long enough for bootstrapping. Partitions will not be allocated on all brokers.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send a "small" number of messages to a new topic, then shut down producer.
3) Partitions will be allocated on a random(?) set of brokers. Not all brokers will host partitions.
4) Brokers that did not allocate partitions for topic will never see the new topic.
For case 2, I'm not sure about the exact length of time, or the number of messages required before all brokers have partitions for the new topic.
Refined the steps to reproduce.
> Issues partitioning a new topic
> -------------------------------
>
> Key: KAFKA-278
> URL: https://issues.apache.org/jira/browse/KAFKA-278
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7
> Reporter: Matt
> Priority: Minor
>
> There are two cases where correct partitioning fails for a new topic.
> Case 1: Topic exists on current Kafka cluster. A new broker is added to the cluster. The new broker will never host partitions for the existing topic.
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages for a topic to the cluster.
> 3) Add a new broker to the cluster.
> 4) New broker will never see the existing topic.
> Case 2: Topic does not exist on current Kafka cluster. Producer sends messages to a new topic that did not previously exist in the cluster. If, during the producer session, one or more partitions are not created on a broker, the broker will never host those partitions.
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages to a new topic.
> 3) Shut down the producer before the topic is created on at least one broker.
> 4) The broker that did not allocate the topic will never host the topic.
> My guess(!) here is that when a new producer is created, it gets a list of topics and partitions based on the current state of the brokers in the cluster. Since some brokers are missing the topic, the producer will never send messages to that broker and partitions will never be created.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-278) Issues partitioning a new topic
Posted by "Nick Howard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Howard updated KAFKA-278:
------------------------------
Attachment: bootstrap_new_brokers.patch
Here's a patch we applied to our fork to deal with this issue.
What it does is bootstrap new brokers in the same way existing brokers are bootstrapped for new topics.
It includes a test.
> Issues partitioning a new topic
> -------------------------------
>
> Key: KAFKA-278
> URL: https://issues.apache.org/jira/browse/KAFKA-278
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7
> Reporter: Matt
> Priority: Minor
> Attachments: bootstrap_new_brokers.patch
>
>
> There are two cases where correct partitioning fails for a new topic.
> Case 1: Topic exists on current Kafka cluster. A new broker is added to the cluster. The new broker will never host partitions for the existing topic.
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages for a topic to the cluster.
> 3) Add a new broker to the cluster.
> 4) New broker will never see the existing topic.
> Case 2: Topic does not exist on current Kafka cluster. Producer sends messages to a new topic that did not previously exist in the cluster. If, during the producer session, one or more partitions are not created on a broker, the broker will never host those partitions.
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages to a new topic.
> 3) Shut down the producer before the topic is created on at least one broker.
> 4) The broker that did not allocate the topic will never host the topic.
> My guess(!) here is that when a new producer is created, it gets a list of topics and partitions based on the current state of the brokers in the cluster. Since some brokers are missing the topic, the producer will never send messages to that broker and partitions will never be created.
> Work around:
> Manually create the topic/partition directories in the kafka logs directory and reboot kafka. It will register the topic/partitions in ZooKeeper.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-278) Issues partitioning a new topic
Posted by "Matt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt updated KAFKA-278:
-----------------------
Description:
There are two cases where correct partitioning fails for a new topic.
Case 1: Topic exists on current Kafka cluster. A new broker is added to the cluster. The new broker will never host partitions for the existing topic.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send messages for a topic to the cluster.
3) Add a new broker to the cluster.
4) New broker will never see the existing topic.
Case 2: Topic does not exist on current Kafka cluster. Producer sends messages to a new topic that did not previously exist in the cluster. If, during the producer session, one or more partitions are not created on a broker, the broker will never host those partitions.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send messages to a new topic.
3) Shut down the producer before the topic is created on at least one broker.
4) The broker that did not allocate the topic will never host the topic.
My guess(!) here is that when a new producer is created, it gets a list of topics and partitions based on the current state of the brokers in the cluster. Since some brokers are missing the topic, the producer will never send messages to that broker and partitions will never be created.
Work around:
Manually create the topic/partition directories in the kafka logs directory and reboot kafka. It will register the topic/partitions in ZooKeeper.
was:
There are two cases where correct partitioning fails for a new topic.
Case 1: Topic exists on current Kafka cluster. A new broker is added to the cluster. The new broker will never host partitions for the existing topic.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send messages for a topic to the cluster.
3) Add a new broker to the cluster.
4) New broker will never see the existing topic.
Case 2: Topic does not exist on current Kafka cluster. Producer sends messages to a new topic that did not previously exist in the cluster. If, during the producer session, one or more partitions are not created on a broker, the broker will never host those partitions.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send messages to a new topic.
3) Shut down the producer before the topic is created on at least one broker.
4) The broker that did not allocate the topic will never host the topic.
My guess(!) here is that when a new producer is created, it gets a list of topics and partitions based on the current state of the brokers in the cluster. Since some brokers are missing the topic, the producer will never send messages to that broker and partitions will never be created.
Added a work around.
> Issues partitioning a new topic
> -------------------------------
>
> Key: KAFKA-278
> URL: https://issues.apache.org/jira/browse/KAFKA-278
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7
> Reporter: Matt
> Priority: Minor
>
> There are two cases where correct partitioning fails for a new topic.
> Case 1: Topic exists on current Kafka cluster. A new broker is added to the cluster. The new broker will never host partitions for the existing topic.
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages for a topic to the cluster.
> 3) Add a new broker to the cluster.
> 4) New broker will never see the existing topic.
> Case 2: Topic does not exist on current Kafka cluster. Producer sends messages to a new topic that did not previously exist in the cluster. If, during the producer session, one or more partitions are not created on a broker, the broker will never host those partitions.
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages to a new topic.
> 3) Shut down the producer before the topic is created on at least one broker.
> 4) The broker that did not allocate the topic will never host the topic.
> My guess(!) here is that when a new producer is created, it gets a list of topics and partitions based on the current state of the brokers in the cluster. Since some brokers are missing the topic, the producer will never send messages to that broker and partitions will never be created.
> Work around:
> Manually create the topic/partition directories in the kafka logs directory and reboot kafka. It will register the topic/partitions in ZooKeeper.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-278) Issues partitioning a new topic
Posted by "Matt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt updated KAFKA-278:
-----------------------
Description:
There are two cases where correct partitioning fails for a new topic.
Case 1: Topic exists on current Kafka cluster. A newly added broker to the cluster will never be notified of the topic and will never host partitions.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send messages for a topic to the cluster.
3) Add a new broker to the cluster.
4) New broker will never see the existing topic.
Case 2: Topic does not exist on current Kafka cluster. Session in which producer sends first set of messages for new topic does not last long enough for bootstrapping. Partitions will not be allocated on all brokers.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send a "small" number of messages to a new topic, then shut down producer.
3) Partitions will be allocated on a random(?) set of brokers. Not all brokers will host partitions.
4) Brokers that did not allocate partitions for topic will never see the new topic.
For case 2, I'm not sure about the exact length of time, or the number of messages required before all brokers have partitions for the new topic.
was:
If you add one or more brokers to an existing cluster, existing topics will never be seen by the new brokers.
To reproduce:
1) Create a cluster of brokers along with a ZooKeeper ensemble.
2) Send messages for a topic to the cluster.
3) Add a new broker to the cluster.
4) New broker will never see the existing topic.
Summary: Issues partitioning a new topic (was: Topic is never distributed to nodes added to an existing cluster)
I changed the description and title to reflect that there are 2 ways that this condition can arise.
> Issues partitioning a new topic
> -------------------------------
>
> Key: KAFKA-278
> URL: https://issues.apache.org/jira/browse/KAFKA-278
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7
> Reporter: Matt
> Priority: Minor
>
> There are two cases where correct partitioning fails for a new topic.
> Case 1: Topic exists on current Kafka cluster. A newly added broker to the cluster will never be notified of the topic and will never host partitions.
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages for a topic to the cluster.
> 3) Add a new broker to the cluster.
> 4) New broker will never see the existing topic.
> Case 2: Topic does not exist on current Kafka cluster. Session in which producer sends first set of messages for new topic does not last long enough for bootstrapping. Partitions will not be allocated on all brokers.
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send a "small" number of messages to a new topic, then shut down producer.
> 3) Partitions will be allocated on a random(?) set of brokers. Not all brokers will host partitions.
> 4) Brokers that did not allocate partitions for topic will never see the new topic.
> For case 2, I'm not sure about the exact length of time, or the number of messages required before all brokers have partitions for the new topic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira