You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Enrico Olivelli <eo...@gmail.com> on 2022/02/13 10:49:25 UTC
New Feature Proposal: flag to require a non-partitioned topic on Producer/Consumer Builder
Hello,
I have a use case in which my application MUST use a non-partitioned
topic to work properly, but the topic name is (of course) configurable
by the user.
If I am not using a non-partitioned topic, I do not have string
guarantees on the ordering of the message (because messages will be
spread across multiple partitions).
Currently there is no way to require that the topic IS a
NON-PARTITIONED topic. Especially when the topic does not exist yet,
you configured the namespace to create partitioned topics by default.
Using a PulsarAdmin preliminary call is not a good workaround because
it is expensive, and ideally I want to be able to create the Producer
(or the Consumer) and see that everything works automatically, failing
in case of a partitioned topic.
Thoughts ?
Enrico
Re: New Feature Proposal: flag to require a non-partitioned topic on Producer/Consumer Builder
Posted by Michael Marshall <mm...@apache.org>.
> Additionally it would be good to force triggering the creation of a non
> partitioned topic even if the namespace/broker is configured to
> automatically create partitioned topics
I think this additional feature request is a bit more controversial
than the fail fast logic discussed earlier. To trigger non partitioned
topic creation would essentially require that the non-admin client can
create topics outside of auto-topic creation. To date, that feature
has been limited to the admin client and is not available in the
Pulsar protocol. It'd be pretty easy to expand the pulsar protocol so
that the Producer command has a "partition_count" field and that field
is only applied when the topic is created (and perhaps fails if the
count is 0 for an already created partitioned topic). I could see this
as a valuable addition for use cases that are sensitive to message
ordering, but perhaps it is worth asking whether we should just make
topic creation a first class feature in the Pulsar protocol?
Thanks,
Michael
On Thu, Feb 17, 2022 at 5:49 AM Enrico Olivelli <eo...@gmail.com> wrote:
>
> Sijie,
>
> Il Gio 17 Feb 2022, 07:13 Michael Marshall <mm...@apache.org> ha
> scritto:
>
> > > Ideally, people shouldn't be aware of partitioned vs non-partitioned.
> >
>
> Sometimes it is a requirement to have a non-partioned topic and the
> application will misbehave if running on a partitioned topic with more than
> one partition.
> This is because if you have a partitioned topic Pulsar does not provide
> guarantees about ordering of all the messages sent by a producer, as it
> will depend on the key (partition).
>
> Unfortunately when you create a Producer you may trigger the creation of a
> topic and this topic maybe a partitioned topic.
>
> The problem is also on the Consumer side, with a partitioned topic you are
> not guaranteed to receive the messages in the same order as they were
> written, because the order is guaranteed per partition.
>
> Do you know a way to address my problem in spite of any broker
> side/namespace configuration option ? Because the user may not have control
> over such configurations or even over the Admin API
>
>
>
>
>
> > I'm not sure I agree that people shouldn't be aware of these details
> > considering the two types of topics provide different guarantees
> > regarding ordering. Enrico's request is essentially to have a way to
> > fail fast when a topic is partitioned.
> >
>
> Yes, this is correct.
> Additionally it would be good to force triggering the creation of a non
> partitioned topic even if the namespace/broker is configured to
> automatically create partitioned topics
>
> Enrico
>
>
>
>
> > Note that the logic for whether or not a topic is partitioned is
> > already available to the client: that is how it knows whether to
> > create a partitioned producer or a non-partitioned producer.
> >
> > The simplest solution is to add a method to the client named
> > "createNonPartitionedProducer". The method would then fail if the
> > topic lookup results in a topic with more than 0 partitions.
> >
> > Alternatively, you could create a custom `MessageRouter` that always
> > picks the 0 partition when `choosePartition` is called. However, I
> > view this as a bit of a hack and think my first solution is a better
> > fit for Enrico's request.
> >
> > Thanks,
> > Michael
> >
> > On Wed, Feb 16, 2022 at 3:27 PM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > I don't think we should expose this feature flag on the client-side.
> > > Ideally, people shouldn't be aware of partitioned vs non-partitioned.
> > >
> > > - Sijie
> > >
> > > On Wed, Feb 16, 2022 at 1:37 AM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> > >
> > > > Ping
> > > >
> > > > Il Dom 13 Feb 2022, 11:49 Enrico Olivelli <eo...@gmail.com> ha
> > > > scritto:
> > > >
> > > > > Hello,
> > > > > I have a use case in which my application MUST use a non-partitioned
> > > > > topic to work properly, but the topic name is (of course)
> > configurable
> > > > > by the user.
> > > > > If I am not using a non-partitioned topic, I do not have string
> > > > > guarantees on the ordering of the message (because messages will be
> > > > > spread across multiple partitions).
> > > > >
> > > > > Currently there is no way to require that the topic IS a
> > > > > NON-PARTITIONED topic. Especially when the topic does not exist yet,
> > > > > you configured the namespace to create partitioned topics by default.
> > > > >
> > > > > Using a PulsarAdmin preliminary call is not a good workaround because
> > > > > it is expensive, and ideally I want to be able to create the Producer
> > > > > (or the Consumer) and see that everything works automatically,
> > failing
> > > > > in case of a partitioned topic.
> > > > >
> > > > > Thoughts ?
> > > > >
> > > > >
> > > > > Enrico
> > > > >
> > > >
> >
Re: New Feature Proposal: flag to require a non-partitioned topic on Producer/Consumer Builder
Posted by Enrico Olivelli <eo...@gmail.com>.
Sijie,
Il Gio 17 Feb 2022, 07:13 Michael Marshall <mm...@apache.org> ha
scritto:
> > Ideally, people shouldn't be aware of partitioned vs non-partitioned.
>
Sometimes it is a requirement to have a non-partioned topic and the
application will misbehave if running on a partitioned topic with more than
one partition.
This is because if you have a partitioned topic Pulsar does not provide
guarantees about ordering of all the messages sent by a producer, as it
will depend on the key (partition).
Unfortunately when you create a Producer you may trigger the creation of a
topic and this topic maybe a partitioned topic.
The problem is also on the Consumer side, with a partitioned topic you are
not guaranteed to receive the messages in the same order as they were
written, because the order is guaranteed per partition.
Do you know a way to address my problem in spite of any broker
side/namespace configuration option ? Because the user may not have control
over such configurations or even over the Admin API
> I'm not sure I agree that people shouldn't be aware of these details
> considering the two types of topics provide different guarantees
> regarding ordering. Enrico's request is essentially to have a way to
> fail fast when a topic is partitioned.
>
Yes, this is correct.
Additionally it would be good to force triggering the creation of a non
partitioned topic even if the namespace/broker is configured to
automatically create partitioned topics
Enrico
> Note that the logic for whether or not a topic is partitioned is
> already available to the client: that is how it knows whether to
> create a partitioned producer or a non-partitioned producer.
>
> The simplest solution is to add a method to the client named
> "createNonPartitionedProducer". The method would then fail if the
> topic lookup results in a topic with more than 0 partitions.
>
> Alternatively, you could create a custom `MessageRouter` that always
> picks the 0 partition when `choosePartition` is called. However, I
> view this as a bit of a hack and think my first solution is a better
> fit for Enrico's request.
>
> Thanks,
> Michael
>
> On Wed, Feb 16, 2022 at 3:27 PM Sijie Guo <gu...@gmail.com> wrote:
> >
> > I don't think we should expose this feature flag on the client-side.
> > Ideally, people shouldn't be aware of partitioned vs non-partitioned.
> >
> > - Sijie
> >
> > On Wed, Feb 16, 2022 at 1:37 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
> >
> > > Ping
> > >
> > > Il Dom 13 Feb 2022, 11:49 Enrico Olivelli <eo...@gmail.com> ha
> > > scritto:
> > >
> > > > Hello,
> > > > I have a use case in which my application MUST use a non-partitioned
> > > > topic to work properly, but the topic name is (of course)
> configurable
> > > > by the user.
> > > > If I am not using a non-partitioned topic, I do not have string
> > > > guarantees on the ordering of the message (because messages will be
> > > > spread across multiple partitions).
> > > >
> > > > Currently there is no way to require that the topic IS a
> > > > NON-PARTITIONED topic. Especially when the topic does not exist yet,
> > > > you configured the namespace to create partitioned topics by default.
> > > >
> > > > Using a PulsarAdmin preliminary call is not a good workaround because
> > > > it is expensive, and ideally I want to be able to create the Producer
> > > > (or the Consumer) and see that everything works automatically,
> failing
> > > > in case of a partitioned topic.
> > > >
> > > > Thoughts ?
> > > >
> > > >
> > > > Enrico
> > > >
> > >
>
Re: New Feature Proposal: flag to require a non-partitioned topic on Producer/Consumer Builder
Posted by Michael Marshall <mm...@apache.org>.
> Ideally, people shouldn't be aware of partitioned vs non-partitioned.
I'm not sure I agree that people shouldn't be aware of these details
considering the two types of topics provide different guarantees
regarding ordering. Enrico's request is essentially to have a way to
fail fast when a topic is partitioned.
Note that the logic for whether or not a topic is partitioned is
already available to the client: that is how it knows whether to
create a partitioned producer or a non-partitioned producer.
The simplest solution is to add a method to the client named
"createNonPartitionedProducer". The method would then fail if the
topic lookup results in a topic with more than 0 partitions.
Alternatively, you could create a custom `MessageRouter` that always
picks the 0 partition when `choosePartition` is called. However, I
view this as a bit of a hack and think my first solution is a better
fit for Enrico's request.
Thanks,
Michael
On Wed, Feb 16, 2022 at 3:27 PM Sijie Guo <gu...@gmail.com> wrote:
>
> I don't think we should expose this feature flag on the client-side.
> Ideally, people shouldn't be aware of partitioned vs non-partitioned.
>
> - Sijie
>
> On Wed, Feb 16, 2022 at 1:37 AM Enrico Olivelli <eo...@gmail.com> wrote:
>
> > Ping
> >
> > Il Dom 13 Feb 2022, 11:49 Enrico Olivelli <eo...@gmail.com> ha
> > scritto:
> >
> > > Hello,
> > > I have a use case in which my application MUST use a non-partitioned
> > > topic to work properly, but the topic name is (of course) configurable
> > > by the user.
> > > If I am not using a non-partitioned topic, I do not have string
> > > guarantees on the ordering of the message (because messages will be
> > > spread across multiple partitions).
> > >
> > > Currently there is no way to require that the topic IS a
> > > NON-PARTITIONED topic. Especially when the topic does not exist yet,
> > > you configured the namespace to create partitioned topics by default.
> > >
> > > Using a PulsarAdmin preliminary call is not a good workaround because
> > > it is expensive, and ideally I want to be able to create the Producer
> > > (or the Consumer) and see that everything works automatically, failing
> > > in case of a partitioned topic.
> > >
> > > Thoughts ?
> > >
> > >
> > > Enrico
> > >
> >
Re: New Feature Proposal: flag to require a non-partitioned topic on Producer/Consumer Builder
Posted by Sijie Guo <gu...@gmail.com>.
I don't think we should expose this feature flag on the client-side.
Ideally, people shouldn't be aware of partitioned vs non-partitioned.
- Sijie
On Wed, Feb 16, 2022 at 1:37 AM Enrico Olivelli <eo...@gmail.com> wrote:
> Ping
>
> Il Dom 13 Feb 2022, 11:49 Enrico Olivelli <eo...@gmail.com> ha
> scritto:
>
> > Hello,
> > I have a use case in which my application MUST use a non-partitioned
> > topic to work properly, but the topic name is (of course) configurable
> > by the user.
> > If I am not using a non-partitioned topic, I do not have string
> > guarantees on the ordering of the message (because messages will be
> > spread across multiple partitions).
> >
> > Currently there is no way to require that the topic IS a
> > NON-PARTITIONED topic. Especially when the topic does not exist yet,
> > you configured the namespace to create partitioned topics by default.
> >
> > Using a PulsarAdmin preliminary call is not a good workaround because
> > it is expensive, and ideally I want to be able to create the Producer
> > (or the Consumer) and see that everything works automatically, failing
> > in case of a partitioned topic.
> >
> > Thoughts ?
> >
> >
> > Enrico
> >
>
Re: New Feature Proposal: flag to require a non-partitioned topic on Producer/Consumer Builder
Posted by Enrico Olivelli <eo...@gmail.com>.
Ping
Il Dom 13 Feb 2022, 11:49 Enrico Olivelli <eo...@gmail.com> ha scritto:
> Hello,
> I have a use case in which my application MUST use a non-partitioned
> topic to work properly, but the topic name is (of course) configurable
> by the user.
> If I am not using a non-partitioned topic, I do not have string
> guarantees on the ordering of the message (because messages will be
> spread across multiple partitions).
>
> Currently there is no way to require that the topic IS a
> NON-PARTITIONED topic. Especially when the topic does not exist yet,
> you configured the namespace to create partitioned topics by default.
>
> Using a PulsarAdmin preliminary call is not a good workaround because
> it is expensive, and ideally I want to be able to create the Producer
> (or the Consumer) and see that everything works automatically, failing
> in case of a partitioned topic.
>
> Thoughts ?
>
>
> Enrico
>