You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Bao Thai Ngo <ba...@gmail.com> on 2012/04/20 04:17:03 UTC
Re: [jira] [Updated] (KAFKA-278) Issues partitioning a new topic
Hi there,
Is there any news on this issue?
Thanks,
~Thai
On Fri, Mar 16, 2012 at 8:04 AM, Matt (Updated) (JIRA) <ji...@apache.org>wrote:
>
> [
> https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Matt updated KAFKA-278:
> -----------------------
>
> Description:
> There are two cases where correct partitioning fails for a new topic.
>
> Case 1: Topic exists on current Kafka cluster. A new broker is added to
> the cluster. The new broker will never host partitions for the existing
> topic.
>
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages for a topic to the cluster.
> 3) Add a new broker to the cluster.
> 4) New broker will never see the existing topic.
>
> Case 2: Topic does not exist on current Kafka cluster. Producer sends
> messages to a new topic that did not previously exist in the cluster. If,
> during the producer session, one or more partitions are not created on a
> broker, the broker will never host those partitions.
>
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages to a new topic.
> 3) Shut down the producer before the topic is created on at least one
> broker.
> 4) The broker that did not allocate the topic will never host the topic.
>
> My guess(!) here is that when a new producer is created, it gets a list of
> topics and partitions based on the current state of the brokers in the
> cluster. Since some brokers are missing the topic, the producer will never
> send messages to that broker and partitions will never be created.
>
>
> Work around:
> Manually create the topic/partition directories in the kafka logs
> directory and reboot kafka. It will register the topic/partitions in
> ZooKeeper.
>
> was:
> There are two cases where correct partitioning fails for a new topic.
>
> Case 1: Topic exists on current Kafka cluster. A new broker is added to
> the cluster. The new broker will never host partitions for the existing
> topic.
>
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages for a topic to the cluster.
> 3) Add a new broker to the cluster.
> 4) New broker will never see the existing topic.
>
> Case 2: Topic does not exist on current Kafka cluster. Producer sends
> messages to a new topic that did not previously exist in the cluster. If,
> during the producer session, one or more partitions are not created on a
> broker, the broker will never host those partitions.
>
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages to a new topic.
> 3) Shut down the producer before the topic is created on at least one
> broker.
> 4) The broker that did not allocate the topic will never host the topic.
>
> My guess(!) here is that when a new producer is created, it gets a list of
> topics and partitions based on the current state of the brokers in the
> cluster. Since some brokers are missing the topic, the producer will never
> send messages to that broker and partitions will never be created.
>
>
> Added a work around.
>
> > Issues partitioning a new topic
> > -------------------------------
> >
> > Key: KAFKA-278
> > URL: https://issues.apache.org/jira/browse/KAFKA-278
> > Project: Kafka
> > Issue Type: Bug
> > Components: core
> > Affects Versions: 0.7
> > Reporter: Matt
> > Priority: Minor
> >
> > There are two cases where correct partitioning fails for a new topic.
> > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> the cluster. The new broker will never host partitions for the existing
> topic.
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages for a topic to the cluster.
> > 3) Add a new broker to the cluster.
> > 4) New broker will never see the existing topic.
> > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> messages to a new topic that did not previously exist in the cluster. If,
> during the producer session, one or more partitions are not created on a
> broker, the broker will never host those partitions.
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages to a new topic.
> > 3) Shut down the producer before the topic is created on at least one
> broker.
> > 4) The broker that did not allocate the topic will never host the topic.
> > My guess(!) here is that when a new producer is created, it gets a list
> of topics and partitions based on the current state of the brokers in the
> cluster. Since some brokers are missing the topic, the producer will never
> send messages to that broker and partitions will never be created.
> > Work around:
> > Manually create the topic/partition directories in the kafka logs
> directory and reboot kafka. It will register the topic/partitions in
> ZooKeeper.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
Re: [jira] [Updated] (KAFKA-278) Issues partitioning a new topic
Posted by Prashanth Menon <pr...@gmail.com>.
I can take a look at it this weekend :)
- Prashanth
On Fri, Apr 20, 2012 at 10:17 AM, Jun Rao <ju...@gmail.com> wrote:
> This jira may not be hard to fix since the required changes are likely
> isolated in ZKBrokerPartitionInfo.getBrokerPartitionInfo(). Basically, we
> just need to bootstrap a broker if it doesn't have a topic already. Anyone
> interested in taking a shot at it?
>
> Thanks,
>
> Jun
>
> On Thu, Apr 19, 2012 at 7:17 PM, Bao Thai Ngo <ba...@gmail.com>
> wrote:
>
> > Hi there,
> >
> > Is there any news on this issue?
> >
> > Thanks,
> > ~Thai
> >
> > On Fri, Mar 16, 2012 at 8:04 AM, Matt (Updated) (JIRA) <jira@apache.org
> > >wrote:
> >
> > >
> > > [
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> > ]
> > >
> > > Matt updated KAFKA-278:
> > > -----------------------
> > >
> > > Description:
> > > There are two cases where correct partitioning fails for a new topic.
> > >
> > > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> > > the cluster. The new broker will never host partitions for the existing
> > > topic.
> > >
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages for a topic to the cluster.
> > > 3) Add a new broker to the cluster.
> > > 4) New broker will never see the existing topic.
> > >
> > > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > > messages to a new topic that did not previously exist in the cluster.
> If,
> > > during the producer session, one or more partitions are not created on
> a
> > > broker, the broker will never host those partitions.
> > >
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages to a new topic.
> > > 3) Shut down the producer before the topic is created on at least one
> > > broker.
> > > 4) The broker that did not allocate the topic will never host the
> topic.
> > >
> > > My guess(!) here is that when a new producer is created, it gets a list
> > of
> > > topics and partitions based on the current state of the brokers in the
> > > cluster. Since some brokers are missing the topic, the producer will
> > never
> > > send messages to that broker and partitions will never be created.
> > >
> > >
> > > Work around:
> > > Manually create the topic/partition directories in the kafka logs
> > > directory and reboot kafka. It will register the topic/partitions in
> > > ZooKeeper.
> > >
> > > was:
> > > There are two cases where correct partitioning fails for a new topic.
> > >
> > > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> > > the cluster. The new broker will never host partitions for the existing
> > > topic.
> > >
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages for a topic to the cluster.
> > > 3) Add a new broker to the cluster.
> > > 4) New broker will never see the existing topic.
> > >
> > > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > > messages to a new topic that did not previously exist in the cluster.
> If,
> > > during the producer session, one or more partitions are not created on
> a
> > > broker, the broker will never host those partitions.
> > >
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages to a new topic.
> > > 3) Shut down the producer before the topic is created on at least one
> > > broker.
> > > 4) The broker that did not allocate the topic will never host the
> topic.
> > >
> > > My guess(!) here is that when a new producer is created, it gets a list
> > of
> > > topics and partitions based on the current state of the brokers in the
> > > cluster. Since some brokers are missing the topic, the producer will
> > never
> > > send messages to that broker and partitions will never be created.
> > >
> > >
> > > Added a work around.
> > >
> > > > Issues partitioning a new topic
> > > > -------------------------------
> > > >
> > > > Key: KAFKA-278
> > > > URL: https://issues.apache.org/jira/browse/KAFKA-278
> > > > Project: Kafka
> > > > Issue Type: Bug
> > > > Components: core
> > > > Affects Versions: 0.7
> > > > Reporter: Matt
> > > > Priority: Minor
> > > >
> > > > There are two cases where correct partitioning fails for a new topic.
> > > > Case 1: Topic exists on current Kafka cluster. A new broker is added
> to
> > > the cluster. The new broker will never host partitions for the existing
> > > topic.
> > > > To reproduce:
> > > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > > 2) Send messages for a topic to the cluster.
> > > > 3) Add a new broker to the cluster.
> > > > 4) New broker will never see the existing topic.
> > > > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > > messages to a new topic that did not previously exist in the cluster.
> If,
> > > during the producer session, one or more partitions are not created on
> a
> > > broker, the broker will never host those partitions.
> > > > To reproduce:
> > > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > > 2) Send messages to a new topic.
> > > > 3) Shut down the producer before the topic is created on at least one
> > > broker.
> > > > 4) The broker that did not allocate the topic will never host the
> > topic.
> > > > My guess(!) here is that when a new producer is created, it gets a
> list
> > > of topics and partitions based on the current state of the brokers in
> the
> > > cluster. Since some brokers are missing the topic, the producer will
> > never
> > > send messages to that broker and partitions will never be created.
> > > > Work around:
> > > > Manually create the topic/partition directories in the kafka logs
> > > directory and reboot kafka. It will register the topic/partitions in
> > > ZooKeeper.
> > >
> > > --
> > > This message is automatically generated by JIRA.
> > > If you think it was sent incorrectly, please contact your JIRA
> > > administrators:
> > >
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> > > For more information on JIRA, see:
> > http://www.atlassian.com/software/jira
> > >
> > >
> > >
> >
>
Re: [jira] [Updated] (KAFKA-278) Issues partitioning a new topic
Posted by Jun Rao <ju...@gmail.com>.
This jira may not be hard to fix since the required changes are likely
isolated in ZKBrokerPartitionInfo.getBrokerPartitionInfo(). Basically, we
just need to bootstrap a broker if it doesn't have a topic already. Anyone
interested in taking a shot at it?
Thanks,
Jun
On Thu, Apr 19, 2012 at 7:17 PM, Bao Thai Ngo <ba...@gmail.com> wrote:
> Hi there,
>
> Is there any news on this issue?
>
> Thanks,
> ~Thai
>
> On Fri, Mar 16, 2012 at 8:04 AM, Matt (Updated) (JIRA) <jira@apache.org
> >wrote:
>
> >
> > [
> >
> https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
> >
> > Matt updated KAFKA-278:
> > -----------------------
> >
> > Description:
> > There are two cases where correct partitioning fails for a new topic.
> >
> > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> > the cluster. The new broker will never host partitions for the existing
> > topic.
> >
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages for a topic to the cluster.
> > 3) Add a new broker to the cluster.
> > 4) New broker will never see the existing topic.
> >
> > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > messages to a new topic that did not previously exist in the cluster. If,
> > during the producer session, one or more partitions are not created on a
> > broker, the broker will never host those partitions.
> >
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages to a new topic.
> > 3) Shut down the producer before the topic is created on at least one
> > broker.
> > 4) The broker that did not allocate the topic will never host the topic.
> >
> > My guess(!) here is that when a new producer is created, it gets a list
> of
> > topics and partitions based on the current state of the brokers in the
> > cluster. Since some brokers are missing the topic, the producer will
> never
> > send messages to that broker and partitions will never be created.
> >
> >
> > Work around:
> > Manually create the topic/partition directories in the kafka logs
> > directory and reboot kafka. It will register the topic/partitions in
> > ZooKeeper.
> >
> > was:
> > There are two cases where correct partitioning fails for a new topic.
> >
> > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> > the cluster. The new broker will never host partitions for the existing
> > topic.
> >
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages for a topic to the cluster.
> > 3) Add a new broker to the cluster.
> > 4) New broker will never see the existing topic.
> >
> > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > messages to a new topic that did not previously exist in the cluster. If,
> > during the producer session, one or more partitions are not created on a
> > broker, the broker will never host those partitions.
> >
> > To reproduce:
> > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > 2) Send messages to a new topic.
> > 3) Shut down the producer before the topic is created on at least one
> > broker.
> > 4) The broker that did not allocate the topic will never host the topic.
> >
> > My guess(!) here is that when a new producer is created, it gets a list
> of
> > topics and partitions based on the current state of the brokers in the
> > cluster. Since some brokers are missing the topic, the producer will
> never
> > send messages to that broker and partitions will never be created.
> >
> >
> > Added a work around.
> >
> > > Issues partitioning a new topic
> > > -------------------------------
> > >
> > > Key: KAFKA-278
> > > URL: https://issues.apache.org/jira/browse/KAFKA-278
> > > Project: Kafka
> > > Issue Type: Bug
> > > Components: core
> > > Affects Versions: 0.7
> > > Reporter: Matt
> > > Priority: Minor
> > >
> > > There are two cases where correct partitioning fails for a new topic.
> > > Case 1: Topic exists on current Kafka cluster. A new broker is added to
> > the cluster. The new broker will never host partitions for the existing
> > topic.
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages for a topic to the cluster.
> > > 3) Add a new broker to the cluster.
> > > 4) New broker will never see the existing topic.
> > > Case 2: Topic does not exist on current Kafka cluster. Producer sends
> > messages to a new topic that did not previously exist in the cluster. If,
> > during the producer session, one or more partitions are not created on a
> > broker, the broker will never host those partitions.
> > > To reproduce:
> > > 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> > > 2) Send messages to a new topic.
> > > 3) Shut down the producer before the topic is created on at least one
> > broker.
> > > 4) The broker that did not allocate the topic will never host the
> topic.
> > > My guess(!) here is that when a new producer is created, it gets a list
> > of topics and partitions based on the current state of the brokers in the
> > cluster. Since some brokers are missing the topic, the producer will
> never
> > send messages to that broker and partitions will never be created.
> > > Work around:
> > > Manually create the topic/partition directories in the kafka logs
> > directory and reboot kafka. It will register the topic/partitions in
> > ZooKeeper.
> >
> > --
> > This message is automatically generated by JIRA.
> > If you think it was sent incorrectly, please contact your JIRA
> > administrators:
> > https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
> >
> >
>