You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Stevo Slavić <ss...@gmail.com> on 2014/10/02 23:44:31 UTC
auto topic creation not working for attempts to consume non-existing topic
Hello Apache Kafka community,
auto.create.topics.enable configuration option docs state:
"Enable auto creation of topic on the server. If this is set to true then
attempts to produce, consume, or fetch metadata for a non-existent topic
will automatically create it with the default replication factor and number
of partitions."
I read this that topic should be created on any attempt to consume
non-existing topic.
With auto.create.topics.enable left at default or explicitly set to true,
attempts to consume non existing topic, using blocking consumer, or a
non-blocking consumer with positive consumer.timeout.ms configured, will
not result in topic creation (I cannot see one registered in ZooKeeper).
Additionally, for non-blocking consumer with timeout, no offset will be
recorded. This further means, if such consumer had auto.offset.reset set to
largest, that it will miss at least one message (initial one that when
published creates the topic), even though consumer attempted to read before
first message was published.
I'm using Kafka 0.8.1.1 but I see same issue exists in current trunk.
Is this a known issue? Or are my expectations/assumptions wrong and this is
expected behavior?
Kind regards,
Stevo Slavic.
Re: auto topic creation not working for attempts to consume
non-existing topic
Posted by Jun Rao <ju...@gmail.com>.
Thanks for creating the jiras. The new java producer is mostly done and
will be released as beta in 0.8.2.
Thanks,
Jun
On Mon, Oct 6, 2014 at 5:58 AM, Stevo Slavić <ss...@gmail.com> wrote:
> Created https://issues.apache.org/jira/browse/KAFKA-1674 for likely docs
> issue.
>
> Is this umbrella/epic for new producer api
> https://issues.apache.org/jira/browse/KAFKA-1239 ?
> Is it planned to be complete in 0.9.0 release? If so, then consider
> switching in KAFKA-1239 0.9.0 from "affects version" to "fix version".
>
> Thanks for fast feedback!
>
> Kind regards,
> Stevo Slavic
>
> On Mon, Oct 6, 2014 at 5:17 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Yes, the docs can be improved. Could you file a jira?
> >
> > For the 2nd issue, the new java producer handles this better.
> >
> > Thanks,
> >
> > jun
> >
> > On Fri, Oct 3, 2014 at 1:31 AM, Stevo Slavić <ss...@gmail.com> wrote:
> >
> > > OK, thanks,
> > >
> > > Do you agree then that the docs for auto topic creation configuration
> > > parameter are misleading and should be changed?
> > >
> > > Another issue is that when the topic auto creation is disabled,
> attempts
> > to
> > > publish a message on a non-existing topic using high-level api will
> > throw a
> > > generic FailedToSendMessageException (even when
> message.send.max.retries
> > is
> > > 0) without having UnknownTopicOrPartitionException at least as cause.
> Is
> > > this feature or a bug, and more importantly could it be improved?
> > >
> > > Kind regards,
> > > Stevo Slavic.
> > > On Oct 3, 2014 6:30 AM, "Jun Rao" <ju...@gmail.com> wrote:
> > >
> > > > In general, only writers should trigger auto topic creation, but not
> > the
> > > > readers. So, a topic can be auto created by the producer, but not the
> > > > consumer.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Oct 2, 2014 at 2:44 PM, Stevo Slavić <ss...@gmail.com>
> > wrote:
> > > >
> > > > > Hello Apache Kafka community,
> > > > >
> > > > > auto.create.topics.enable configuration option docs state:
> > > > > "Enable auto creation of topic on the server. If this is set to
> true
> > > then
> > > > > attempts to produce, consume, or fetch metadata for a non-existent
> > > topic
> > > > > will automatically create it with the default replication factor
> and
> > > > number
> > > > > of partitions."
> > > > >
> > > > > I read this that topic should be created on any attempt to consume
> > > > > non-existing topic.
> > > > >
> > > > > With auto.create.topics.enable left at default or explicitly set to
> > > true,
> > > > > attempts to consume non existing topic, using blocking consumer,
> or a
> > > > > non-blocking consumer with positive consumer.timeout.ms
> configured,
> > > will
> > > > > not result in topic creation (I cannot see one registered in
> > > ZooKeeper).
> > > > >
> > > > > Additionally, for non-blocking consumer with timeout, no offset
> will
> > be
> > > > > recorded. This further means, if such consumer had
> auto.offset.reset
> > > set
> > > > to
> > > > > largest, that it will miss at least one message (initial one that
> > when
> > > > > published creates the topic), even though consumer attempted to
> read
> > > > before
> > > > > first message was published.
> > > > >
> > > > > I'm using Kafka 0.8.1.1 but I see same issue exists in current
> trunk.
> > > > >
> > > > > Is this a known issue? Or are my expectations/assumptions wrong and
> > > this
> > > > is
> > > > > expected behavior?
> > > > >
> > > > > Kind regards,
> > > > > Stevo Slavic.
> > > > >
> > > >
> > >
> >
>
Re: auto topic creation not working for attempts to consume
non-existing topic
Posted by Stevo Slavić <ss...@gmail.com>.
Created https://issues.apache.org/jira/browse/KAFKA-1674 for likely docs
issue.
Is this umbrella/epic for new producer api
https://issues.apache.org/jira/browse/KAFKA-1239 ?
Is it planned to be complete in 0.9.0 release? If so, then consider
switching in KAFKA-1239 0.9.0 from "affects version" to "fix version".
Thanks for fast feedback!
Kind regards,
Stevo Slavic
On Mon, Oct 6, 2014 at 5:17 AM, Jun Rao <ju...@gmail.com> wrote:
> Yes, the docs can be improved. Could you file a jira?
>
> For the 2nd issue, the new java producer handles this better.
>
> Thanks,
>
> jun
>
> On Fri, Oct 3, 2014 at 1:31 AM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > OK, thanks,
> >
> > Do you agree then that the docs for auto topic creation configuration
> > parameter are misleading and should be changed?
> >
> > Another issue is that when the topic auto creation is disabled, attempts
> to
> > publish a message on a non-existing topic using high-level api will
> throw a
> > generic FailedToSendMessageException (even when message.send.max.retries
> is
> > 0) without having UnknownTopicOrPartitionException at least as cause. Is
> > this feature or a bug, and more importantly could it be improved?
> >
> > Kind regards,
> > Stevo Slavic.
> > On Oct 3, 2014 6:30 AM, "Jun Rao" <ju...@gmail.com> wrote:
> >
> > > In general, only writers should trigger auto topic creation, but not
> the
> > > readers. So, a topic can be auto created by the producer, but not the
> > > consumer.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Oct 2, 2014 at 2:44 PM, Stevo Slavić <ss...@gmail.com>
> wrote:
> > >
> > > > Hello Apache Kafka community,
> > > >
> > > > auto.create.topics.enable configuration option docs state:
> > > > "Enable auto creation of topic on the server. If this is set to true
> > then
> > > > attempts to produce, consume, or fetch metadata for a non-existent
> > topic
> > > > will automatically create it with the default replication factor and
> > > number
> > > > of partitions."
> > > >
> > > > I read this that topic should be created on any attempt to consume
> > > > non-existing topic.
> > > >
> > > > With auto.create.topics.enable left at default or explicitly set to
> > true,
> > > > attempts to consume non existing topic, using blocking consumer, or a
> > > > non-blocking consumer with positive consumer.timeout.ms configured,
> > will
> > > > not result in topic creation (I cannot see one registered in
> > ZooKeeper).
> > > >
> > > > Additionally, for non-blocking consumer with timeout, no offset will
> be
> > > > recorded. This further means, if such consumer had auto.offset.reset
> > set
> > > to
> > > > largest, that it will miss at least one message (initial one that
> when
> > > > published creates the topic), even though consumer attempted to read
> > > before
> > > > first message was published.
> > > >
> > > > I'm using Kafka 0.8.1.1 but I see same issue exists in current trunk.
> > > >
> > > > Is this a known issue? Or are my expectations/assumptions wrong and
> > this
> > > is
> > > > expected behavior?
> > > >
> > > > Kind regards,
> > > > Stevo Slavic.
> > > >
> > >
> >
>
Re: auto topic creation not working for attempts to consume
non-existing topic
Posted by Jun Rao <ju...@gmail.com>.
Yes, the docs can be improved. Could you file a jira?
For the 2nd issue, the new java producer handles this better.
Thanks,
jun
On Fri, Oct 3, 2014 at 1:31 AM, Stevo Slavić <ss...@gmail.com> wrote:
> OK, thanks,
>
> Do you agree then that the docs for auto topic creation configuration
> parameter are misleading and should be changed?
>
> Another issue is that when the topic auto creation is disabled, attempts to
> publish a message on a non-existing topic using high-level api will throw a
> generic FailedToSendMessageException (even when message.send.max.retries is
> 0) without having UnknownTopicOrPartitionException at least as cause. Is
> this feature or a bug, and more importantly could it be improved?
>
> Kind regards,
> Stevo Slavic.
> On Oct 3, 2014 6:30 AM, "Jun Rao" <ju...@gmail.com> wrote:
>
> > In general, only writers should trigger auto topic creation, but not the
> > readers. So, a topic can be auto created by the producer, but not the
> > consumer.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Oct 2, 2014 at 2:44 PM, Stevo Slavić <ss...@gmail.com> wrote:
> >
> > > Hello Apache Kafka community,
> > >
> > > auto.create.topics.enable configuration option docs state:
> > > "Enable auto creation of topic on the server. If this is set to true
> then
> > > attempts to produce, consume, or fetch metadata for a non-existent
> topic
> > > will automatically create it with the default replication factor and
> > number
> > > of partitions."
> > >
> > > I read this that topic should be created on any attempt to consume
> > > non-existing topic.
> > >
> > > With auto.create.topics.enable left at default or explicitly set to
> true,
> > > attempts to consume non existing topic, using blocking consumer, or a
> > > non-blocking consumer with positive consumer.timeout.ms configured,
> will
> > > not result in topic creation (I cannot see one registered in
> ZooKeeper).
> > >
> > > Additionally, for non-blocking consumer with timeout, no offset will be
> > > recorded. This further means, if such consumer had auto.offset.reset
> set
> > to
> > > largest, that it will miss at least one message (initial one that when
> > > published creates the topic), even though consumer attempted to read
> > before
> > > first message was published.
> > >
> > > I'm using Kafka 0.8.1.1 but I see same issue exists in current trunk.
> > >
> > > Is this a known issue? Or are my expectations/assumptions wrong and
> this
> > is
> > > expected behavior?
> > >
> > > Kind regards,
> > > Stevo Slavic.
> > >
> >
>
Re: auto topic creation not working for attempts to consume
non-existing topic
Posted by Stevo Slavić <ss...@gmail.com>.
OK, thanks,
Do you agree then that the docs for auto topic creation configuration
parameter are misleading and should be changed?
Another issue is that when the topic auto creation is disabled, attempts to
publish a message on a non-existing topic using high-level api will throw a
generic FailedToSendMessageException (even when message.send.max.retries is
0) without having UnknownTopicOrPartitionException at least as cause. Is
this feature or a bug, and more importantly could it be improved?
Kind regards,
Stevo Slavic.
On Oct 3, 2014 6:30 AM, "Jun Rao" <ju...@gmail.com> wrote:
> In general, only writers should trigger auto topic creation, but not the
> readers. So, a topic can be auto created by the producer, but not the
> consumer.
>
> Thanks,
>
> Jun
>
> On Thu, Oct 2, 2014 at 2:44 PM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > Hello Apache Kafka community,
> >
> > auto.create.topics.enable configuration option docs state:
> > "Enable auto creation of topic on the server. If this is set to true then
> > attempts to produce, consume, or fetch metadata for a non-existent topic
> > will automatically create it with the default replication factor and
> number
> > of partitions."
> >
> > I read this that topic should be created on any attempt to consume
> > non-existing topic.
> >
> > With auto.create.topics.enable left at default or explicitly set to true,
> > attempts to consume non existing topic, using blocking consumer, or a
> > non-blocking consumer with positive consumer.timeout.ms configured, will
> > not result in topic creation (I cannot see one registered in ZooKeeper).
> >
> > Additionally, for non-blocking consumer with timeout, no offset will be
> > recorded. This further means, if such consumer had auto.offset.reset set
> to
> > largest, that it will miss at least one message (initial one that when
> > published creates the topic), even though consumer attempted to read
> before
> > first message was published.
> >
> > I'm using Kafka 0.8.1.1 but I see same issue exists in current trunk.
> >
> > Is this a known issue? Or are my expectations/assumptions wrong and this
> is
> > expected behavior?
> >
> > Kind regards,
> > Stevo Slavic.
> >
>
Re: auto topic creation not working for attempts to consume
non-existing topic
Posted by Jun Rao <ju...@gmail.com>.
In general, only writers should trigger auto topic creation, but not the
readers. So, a topic can be auto created by the producer, but not the
consumer.
Thanks,
Jun
On Thu, Oct 2, 2014 at 2:44 PM, Stevo Slavić <ss...@gmail.com> wrote:
> Hello Apache Kafka community,
>
> auto.create.topics.enable configuration option docs state:
> "Enable auto creation of topic on the server. If this is set to true then
> attempts to produce, consume, or fetch metadata for a non-existent topic
> will automatically create it with the default replication factor and number
> of partitions."
>
> I read this that topic should be created on any attempt to consume
> non-existing topic.
>
> With auto.create.topics.enable left at default or explicitly set to true,
> attempts to consume non existing topic, using blocking consumer, or a
> non-blocking consumer with positive consumer.timeout.ms configured, will
> not result in topic creation (I cannot see one registered in ZooKeeper).
>
> Additionally, for non-blocking consumer with timeout, no offset will be
> recorded. This further means, if such consumer had auto.offset.reset set to
> largest, that it will miss at least one message (initial one that when
> published creates the topic), even though consumer attempted to read before
> first message was published.
>
> I'm using Kafka 0.8.1.1 but I see same issue exists in current trunk.
>
> Is this a known issue? Or are my expectations/assumptions wrong and this is
> expected behavior?
>
> Kind regards,
> Stevo Slavic.
>