You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mayuresh Gharat <gh...@gmail.com> on 2016/01/07 22:49:23 UTC
TMR should nopt create topic if not existing.
Hi,
There has been discussion on the ticket :
https://issues.apache.org/jira/browse/KAFKA-2887, that we are going to
deprecate auto-creation of topic or at least turn it off by default once we
have the CreateTopics API. It also says the patch is available for this.
The only other ticket that I came across on the ticket is
https://issues.apache.org/jira/browse/KAFKA-1507.
I wanted to confirm that https://issues.apache.org/jira/browse/KAFKA-1507
is the ticket that has the CreateTopics API patch.
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: TMR should nopt create topic if not existing.
Posted by Grant Henke <gh...@cloudera.com>.
Hi Mayuresh,
Your understand aligns with my understanding.
Thanks,
Grant
On Fri, Jan 8, 2016 at 11:38 AM, Mayuresh Gharat <gharatmayuresh15@gmail.com
> wrote:
> Hi Grant,
>
> Thanks a lot for the explanation.This is great.
> It was little confusing with KIP-4 and KAFKA-1507 both talking about the
> somewhat same thing and KAFKA-1507 has been lying around since last August
> and not checked in or iterated on.
>
> Here is what I understand is :
> 1) KAFKA-2945 will implement the CreateTopic Request that will block on the
> server until the topic is successfully created. This will return success or
> should return failure on timeout.
> 2) Once this is done, whether we want to have a config in Kafka Producer
> and Consumer OR we want to let the users create the topic explicitly using
> the createTopicRequest API is still under discussion.
>
> Am I right on both the above points?
>
> The reason I want to confirm this is because it is very important for us
> when we want to delete topics in a pipeline and there are several clients
> producing, consuming and mirror-makers and monitoring/auditing systems
> running at the same time.
>
>
> Thanks,
>
> Mayuresh
>
> On Fri, Jan 8, 2016 at 8:41 AM, Grant Henke <gh...@cloudera.com> wrote:
>
> > Hi Mayuresh,
> >
> > I am working on KIP-4 and the create topic wire protocol is a part of
> that.
> > I have a patch but it needs to be changed to be synchronous on the server
> > side yet. You can see the discussion on the pull request.
> > That work can be tracked here:
> > https://issues.apache.org/jira/browse/KAFKA-2945
> >
> > Once the create topic work is done, clients can then create missing
> topics
> > themselves and we wont depend on the server auto creating them. the
> client,
> > auto creation could be baked into the client, or users could use the
> error
> > codes to make the call themselves, that work and details has not been
> > vetted yet.
> > There is a jira to track that here:
> > https://issues.apache.org/jira/browse/KAFKA-2410
> >
> > Hope that helps.
> >
> > Thanks,
> > Grant
> >
> > On Thu, Jan 7, 2016 at 6:27 PM, Mayuresh Gharat <
> > gharatmayuresh15@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I might have missed out the discussions on this :(
> > >
> > > I had some more questions :
> > >
> > > 1) We need a config in KafkaProducer for this. So when the
> KafkaProducer
> > > issues a TMR for a topic and receives a response that the topic does
> not
> > > exist, depending on the value of this config it should use the
> > CreateTopic
> > > request to create the topic or exit. Is there a ticket for this?
> > >
> > > 2) We need a config in KafkaConsumer for this. So when the
> > > KafkaConsumer(old and new) issues a TMR for a topic and receives a
> > response
> > > that the topic does not exist, depending on the value of this config it
> > > should use the CreateTopic request to create the topic or exit. Is
> there
> > a
> > > ticket for this?
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Thu, Jan 7, 2016 at 3:27 PM, Mayuresh Gharat <
> > > gharatmayuresh15@gmail.com>
> > > wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Thanks. I had found this ticket before but the KAFKA-1507 also has
> some
> > > > context about this and I was confused basically exactly which patch
> is
> > > > going to go in.
> > > > Thanks a lot for confirming.
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Thu, Jan 7, 2016 at 3:08 PM, Jason Gustafson <ja...@confluent.io>
> > > > wrote:
> > > >
> > > >> Hey Mayuresh,
> > > >>
> > > >> The ticket that Grant Henke has been working on is here:
> > > >> https://issues.apache.org/jira/browse/KAFKA-2945. Is that what you
> > were
> > > >> looking for?
> > > >>
> > > >> -Jason
> > > >>
> > > >> On Thu, Jan 7, 2016 at 1:50 PM, Mayuresh Gharat <
> > > >> gharatmayuresh15@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > + dev
> > > >> >
> > > >> > Hi
> > > >> >
> > > >> > There has been discussion on the ticket :
> > > >> > https://issues.apache.org/jira/browse/KAFKA-2887, that we are
> going
> > > to
> > > >> > deprecate auto-creation of topic or at least turn it off by
> default
> > > >> once we
> > > >> > have the CreateTopics API. It also says the patch is available for
> > > this.
> > > >> >
> > > >> > The only other ticket that I came across on the ticket is
> > > >> > https://issues.apache.org/jira/browse/KAFKA-1507.
> > > >> >
> > > >> > I wanted to confirm that
> > > >> https://issues.apache.org/jira/browse/KAFKA-1507
> > > >> > is the ticket that has the CreateTopics API patch.
> > > >> > <%28862%29%20250-7125>
> > > >> >
> > > >> >
> > > >> > --
> > > >> > -Regards,
> > > >> > Mayuresh R. Gharat
> > > >> > (862) 250-7125
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > -Regards,
> > > > Mayuresh R. Gharat
> > > > (862) 250-7125
> > > >
> > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
--
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: TMR should nopt create topic if not existing.
Posted by Mayuresh Gharat <gh...@gmail.com>.
Hi Grant,
Thanks a lot for the explanation.This is great.
It was little confusing with KIP-4 and KAFKA-1507 both talking about the
somewhat same thing and KAFKA-1507 has been lying around since last August
and not checked in or iterated on.
Here is what I understand is :
1) KAFKA-2945 will implement the CreateTopic Request that will block on the
server until the topic is successfully created. This will return success or
should return failure on timeout.
2) Once this is done, whether we want to have a config in Kafka Producer
and Consumer OR we want to let the users create the topic explicitly using
the createTopicRequest API is still under discussion.
Am I right on both the above points?
The reason I want to confirm this is because it is very important for us
when we want to delete topics in a pipeline and there are several clients
producing, consuming and mirror-makers and monitoring/auditing systems
running at the same time.
Thanks,
Mayuresh
On Fri, Jan 8, 2016 at 8:41 AM, Grant Henke <gh...@cloudera.com> wrote:
> Hi Mayuresh,
>
> I am working on KIP-4 and the create topic wire protocol is a part of that.
> I have a patch but it needs to be changed to be synchronous on the server
> side yet. You can see the discussion on the pull request.
> That work can be tracked here:
> https://issues.apache.org/jira/browse/KAFKA-2945
>
> Once the create topic work is done, clients can then create missing topics
> themselves and we wont depend on the server auto creating them. the client,
> auto creation could be baked into the client, or users could use the error
> codes to make the call themselves, that work and details has not been
> vetted yet.
> There is a jira to track that here:
> https://issues.apache.org/jira/browse/KAFKA-2410
>
> Hope that helps.
>
> Thanks,
> Grant
>
> On Thu, Jan 7, 2016 at 6:27 PM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com>
> wrote:
>
> > Hi,
> >
> > I might have missed out the discussions on this :(
> >
> > I had some more questions :
> >
> > 1) We need a config in KafkaProducer for this. So when the KafkaProducer
> > issues a TMR for a topic and receives a response that the topic does not
> > exist, depending on the value of this config it should use the
> CreateTopic
> > request to create the topic or exit. Is there a ticket for this?
> >
> > 2) We need a config in KafkaConsumer for this. So when the
> > KafkaConsumer(old and new) issues a TMR for a topic and receives a
> response
> > that the topic does not exist, depending on the value of this config it
> > should use the CreateTopic request to create the topic or exit. Is there
> a
> > ticket for this?
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Thu, Jan 7, 2016 at 3:27 PM, Mayuresh Gharat <
> > gharatmayuresh15@gmail.com>
> > wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks. I had found this ticket before but the KAFKA-1507 also has some
> > > context about this and I was confused basically exactly which patch is
> > > going to go in.
> > > Thanks a lot for confirming.
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Thu, Jan 7, 2016 at 3:08 PM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > >> Hey Mayuresh,
> > >>
> > >> The ticket that Grant Henke has been working on is here:
> > >> https://issues.apache.org/jira/browse/KAFKA-2945. Is that what you
> were
> > >> looking for?
> > >>
> > >> -Jason
> > >>
> > >> On Thu, Jan 7, 2016 at 1:50 PM, Mayuresh Gharat <
> > >> gharatmayuresh15@gmail.com>
> > >> wrote:
> > >>
> > >> > + dev
> > >> >
> > >> > Hi
> > >> >
> > >> > There has been discussion on the ticket :
> > >> > https://issues.apache.org/jira/browse/KAFKA-2887, that we are going
> > to
> > >> > deprecate auto-creation of topic or at least turn it off by default
> > >> once we
> > >> > have the CreateTopics API. It also says the patch is available for
> > this.
> > >> >
> > >> > The only other ticket that I came across on the ticket is
> > >> > https://issues.apache.org/jira/browse/KAFKA-1507.
> > >> >
> > >> > I wanted to confirm that
> > >> https://issues.apache.org/jira/browse/KAFKA-1507
> > >> > is the ticket that has the CreateTopics API patch.
> > >> > <%28862%29%20250-7125>
> > >> >
> > >> >
> > >> > --
> > >> > -Regards,
> > >> > Mayuresh R. Gharat
> > >> > (862) 250-7125
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: TMR should nopt create topic if not existing.
Posted by Grant Henke <gh...@cloudera.com>.
Hi Mayuresh,
I am working on KIP-4 and the create topic wire protocol is a part of that.
I have a patch but it needs to be changed to be synchronous on the server
side yet. You can see the discussion on the pull request.
That work can be tracked here:
https://issues.apache.org/jira/browse/KAFKA-2945
Once the create topic work is done, clients can then create missing topics
themselves and we wont depend on the server auto creating them. the client,
auto creation could be baked into the client, or users could use the error
codes to make the call themselves, that work and details has not been
vetted yet.
There is a jira to track that here:
https://issues.apache.org/jira/browse/KAFKA-2410
Hope that helps.
Thanks,
Grant
On Thu, Jan 7, 2016 at 6:27 PM, Mayuresh Gharat <gh...@gmail.com>
wrote:
> Hi,
>
> I might have missed out the discussions on this :(
>
> I had some more questions :
>
> 1) We need a config in KafkaProducer for this. So when the KafkaProducer
> issues a TMR for a topic and receives a response that the topic does not
> exist, depending on the value of this config it should use the CreateTopic
> request to create the topic or exit. Is there a ticket for this?
>
> 2) We need a config in KafkaConsumer for this. So when the
> KafkaConsumer(old and new) issues a TMR for a topic and receives a response
> that the topic does not exist, depending on the value of this config it
> should use the CreateTopic request to create the topic or exit. Is there a
> ticket for this?
>
> Thanks,
>
> Mayuresh
>
> On Thu, Jan 7, 2016 at 3:27 PM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com>
> wrote:
>
> > Hi Jason,
> >
> > Thanks. I had found this ticket before but the KAFKA-1507 also has some
> > context about this and I was confused basically exactly which patch is
> > going to go in.
> > Thanks a lot for confirming.
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Thu, Jan 7, 2016 at 3:08 PM, Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> >> Hey Mayuresh,
> >>
> >> The ticket that Grant Henke has been working on is here:
> >> https://issues.apache.org/jira/browse/KAFKA-2945. Is that what you were
> >> looking for?
> >>
> >> -Jason
> >>
> >> On Thu, Jan 7, 2016 at 1:50 PM, Mayuresh Gharat <
> >> gharatmayuresh15@gmail.com>
> >> wrote:
> >>
> >> > + dev
> >> >
> >> > Hi
> >> >
> >> > There has been discussion on the ticket :
> >> > https://issues.apache.org/jira/browse/KAFKA-2887, that we are going
> to
> >> > deprecate auto-creation of topic or at least turn it off by default
> >> once we
> >> > have the CreateTopics API. It also says the patch is available for
> this.
> >> >
> >> > The only other ticket that I came across on the ticket is
> >> > https://issues.apache.org/jira/browse/KAFKA-1507.
> >> >
> >> > I wanted to confirm that
> >> https://issues.apache.org/jira/browse/KAFKA-1507
> >> > is the ticket that has the CreateTopics API patch.
> >> > <%28862%29%20250-7125>
> >> >
> >> >
> >> > --
> >> > -Regards,
> >> > Mayuresh R. Gharat
> >> > (862) 250-7125
> >> >
> >>
> >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
--
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: TMR should nopt create topic if not existing.
Posted by Mayuresh Gharat <gh...@gmail.com>.
Hi,
I might have missed out the discussions on this :(
I had some more questions :
1) We need a config in KafkaProducer for this. So when the KafkaProducer
issues a TMR for a topic and receives a response that the topic does not
exist, depending on the value of this config it should use the CreateTopic
request to create the topic or exit. Is there a ticket for this?
2) We need a config in KafkaConsumer for this. So when the
KafkaConsumer(old and new) issues a TMR for a topic and receives a response
that the topic does not exist, depending on the value of this config it
should use the CreateTopic request to create the topic or exit. Is there a
ticket for this?
Thanks,
Mayuresh
On Thu, Jan 7, 2016 at 3:27 PM, Mayuresh Gharat <gh...@gmail.com>
wrote:
> Hi Jason,
>
> Thanks. I had found this ticket before but the KAFKA-1507 also has some
> context about this and I was confused basically exactly which patch is
> going to go in.
> Thanks a lot for confirming.
>
> Thanks,
>
> Mayuresh
>
> On Thu, Jan 7, 2016 at 3:08 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
>> Hey Mayuresh,
>>
>> The ticket that Grant Henke has been working on is here:
>> https://issues.apache.org/jira/browse/KAFKA-2945. Is that what you were
>> looking for?
>>
>> -Jason
>>
>> On Thu, Jan 7, 2016 at 1:50 PM, Mayuresh Gharat <
>> gharatmayuresh15@gmail.com>
>> wrote:
>>
>> > + dev
>> >
>> > Hi
>> >
>> > There has been discussion on the ticket :
>> > https://issues.apache.org/jira/browse/KAFKA-2887, that we are going to
>> > deprecate auto-creation of topic or at least turn it off by default
>> once we
>> > have the CreateTopics API. It also says the patch is available for this.
>> >
>> > The only other ticket that I came across on the ticket is
>> > https://issues.apache.org/jira/browse/KAFKA-1507.
>> >
>> > I wanted to confirm that
>> https://issues.apache.org/jira/browse/KAFKA-1507
>> > is the ticket that has the CreateTopics API patch.
>> > <%28862%29%20250-7125>
>> >
>> >
>> > --
>> > -Regards,
>> > Mayuresh R. Gharat
>> > (862) 250-7125
>> >
>>
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: TMR should nopt create topic if not existing.
Posted by Mayuresh Gharat <gh...@gmail.com>.
Hi Jason,
Thanks. I had found this ticket before but the KAFKA-1507 also has some
context about this and I was confused basically exactly which patch is
going to go in.
Thanks a lot for confirming.
Thanks,
Mayuresh
On Thu, Jan 7, 2016 at 3:08 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Hey Mayuresh,
>
> The ticket that Grant Henke has been working on is here:
> https://issues.apache.org/jira/browse/KAFKA-2945. Is that what you were
> looking for?
>
> -Jason
>
> On Thu, Jan 7, 2016 at 1:50 PM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com>
> wrote:
>
> > + dev
> >
> > Hi
> >
> > There has been discussion on the ticket :
> > https://issues.apache.org/jira/browse/KAFKA-2887, that we are going to
> > deprecate auto-creation of topic or at least turn it off by default once
> we
> > have the CreateTopics API. It also says the patch is available for this.
> >
> > The only other ticket that I came across on the ticket is
> > https://issues.apache.org/jira/browse/KAFKA-1507.
> >
> > I wanted to confirm that
> https://issues.apache.org/jira/browse/KAFKA-1507
> > is the ticket that has the CreateTopics API patch.
> > <%28862%29%20250-7125>
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: TMR should nopt create topic if not existing.
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Mayuresh,
The ticket that Grant Henke has been working on is here:
https://issues.apache.org/jira/browse/KAFKA-2945. Is that what you were
looking for?
-Jason
On Thu, Jan 7, 2016 at 1:50 PM, Mayuresh Gharat <gh...@gmail.com>
wrote:
> + dev
>
> Hi
>
> There has been discussion on the ticket :
> https://issues.apache.org/jira/browse/KAFKA-2887, that we are going to
> deprecate auto-creation of topic or at least turn it off by default once we
> have the CreateTopics API. It also says the patch is available for this.
>
> The only other ticket that I came across on the ticket is
> https://issues.apache.org/jira/browse/KAFKA-1507.
>
> I wanted to confirm that https://issues.apache.org/jira/browse/KAFKA-1507
> is the ticket that has the CreateTopics API patch.
> <%28862%29%20250-7125>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: TMR should nopt create topic if not existing.
Posted by Mayuresh Gharat <gh...@gmail.com>.
+ dev
Hi
There has been discussion on the ticket :
https://issues.apache.org/jira/browse/KAFKA-2887, that we are going to
deprecate auto-creation of topic or at least turn it off by default once we
have the CreateTopics API. It also says the patch is available for this.
The only other ticket that I came across on the ticket is
https://issues.apache.org/jira/browse/KAFKA-1507.
I wanted to confirm that https://issues.apache.org/jira/browse/KAFKA-1507
is the ticket that has the CreateTopics API patch.
<%28862%29%20250-7125>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: TMR should nopt create topic if not existing.
Posted by Mayuresh Gharat <gh...@gmail.com>.
+ dev
Hi
There has been discussion on the ticket :
https://issues.apache.org/jira/browse/KAFKA-2887, that we are going to
deprecate auto-creation of topic or at least turn it off by default once we
have the CreateTopics API. It also says the patch is available for this.
The only other ticket that I came across on the ticket is
https://issues.apache.org/jira/browse/KAFKA-1507.
I wanted to confirm that https://issues.apache.org/jira/browse/KAFKA-1507
is the ticket that has the CreateTopics API patch.
<%28862%29%20250-7125>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125