You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Enrico Olivelli <eo...@gmail.com> on 2022/05/10 12:56:36 UTC
[DISCUSS] PIP-157 Exclusive Producer: new mode ExclusiveWithFencing
Hello,
I created a new PIP about a new AccessMode for the Producer.
https://github.com/apache/pulsar/issues/15528
This is the PR: https://github.com/apache/pulsar/pull/15488
Honestly I don't like the name "ExclusiveWithFencing", any suggestion
is really appreciated.
Enrico
Motivation
In PIP-68 we introduced two access modes for the Producer:
Exclusive: The producer is the only one who can publish to the topic.
Fail if there is another Exclusive Producer connected to the topic
while creating the new Producer.
WaitForExclusive: Like Exclusive, but instead of Failing we are going
to wait for the current Exclusive Producer to disconnect.
Those two modes are very powerful and allow you to perform some kind
of Locking on a topic.
We are missing a third more, in which the Producer always succeeds to
acquire the Exclusive lock on the topic by fencing out any other
Producer that is connected, even the current Exclusive Producer and
the other Producers waiting in WaitForExclusive mode.
Goal
The modes that are available with PIP-68 require a writer to acquire
the lock and release it as soon as possible in order to allow other
clients to write to the topic.
With the new mode it will be possible to implement locking in another
way: the Producer holds the lock until someone else steals it. This
way when you have very low contention you can achieve better latency
for writes because you don't have to acquire the lock every time you
want to write,.
API and Wire protocol Changes
Changes:
a new constant on the Wire Protocol for AccessMode
a new constant in the Java Client API AccessMode#ExclusiveWithFencing
Implementation
The new mode will behave mostly like AccessMode#Exclusive but instead
of failing in case of the presence of other Producers it will force
all of the current connected Producers to be removed and invalidated
(they will see ProducerFencedException).
Reject Alternatives
None
Re: [DISCUSS] PIP-157 Exclusive Producer: new mode ExclusiveWithFencing
Posted by Enrico Olivelli <eo...@gmail.com>.
Kezhu,
Il giorno mer 11 mag 2022 alle ore 04:04 Kezhu Wang <ke...@gmail.com>
ha scritto:
>
> Nice improvement.
>
> I was searching for similar functionalities to ban/fence old producers and
> found a similar improvement/suggestion from [PIP 68 Suggestions](
> https://github.com/apache/pulsar/issues/7584) in addition to PIP 68.
This (7584) is interesting.
My proposal is in fact to extend PIP-68.
Enrico
>
> Best,
> Kezhu Wang
>
> On May 10, 2022 at 21:59:43, Enrico Olivelli (eolivelli@gmail.com) wrote:
>
> I have renamed this to PIP-161 (there is some mess in the wiki page)
>
> Enrico
>
> Il giorno mar 10 mag 2022 alle ore 14:56 Enrico Olivelli
> <eo...@gmail.com> ha scritto:
> >
> > Hello,
> > I created a new PIP about a new AccessMode for the Producer.
> > https://github.com/apache/pulsar/issues/15528
> >
> > This is the PR: https://github.com/apache/pulsar/pull/15488
> >
> > Honestly I don't like the name "ExclusiveWithFencing", any suggestion
> > is really appreciated.
> >
> > Enrico
> >
> > Motivation
> >
> > In PIP-68 we introduced two access modes for the Producer:
> >
> > Exclusive: The producer is the only one who can publish to the topic.
> > Fail if there is another Exclusive Producer connected to the topic
> > while creating the new Producer.
> > WaitForExclusive: Like Exclusive, but instead of Failing we are going
> > to wait for the current Exclusive Producer to disconnect.
> >
> > Those two modes are very powerful and allow you to perform some kind
> > of Locking on a topic.
> >
> > We are missing a third more, in which the Producer always succeeds to
> > acquire the Exclusive lock on the topic by fencing out any other
> > Producer that is connected, even the current Exclusive Producer and
> > the other Producers waiting in WaitForExclusive mode.
> >
> > Goal
> >
> > The modes that are available with PIP-68 require a writer to acquire
> > the lock and release it as soon as possible in order to allow other
> > clients to write to the topic.
> >
> > With the new mode it will be possible to implement locking in another
> > way: the Producer holds the lock until someone else steals it. This
> > way when you have very low contention you can achieve better latency
> > for writes because you don't have to acquire the lock every time you
> > want to write,.
> >
> > API and Wire protocol Changes
> >
> > Changes:
> >
> > a new constant on the Wire Protocol for AccessMode
> > a new constant in the Java Client API AccessMode#ExclusiveWithFencing
> >
> > Implementation
> >
> > The new mode will behave mostly like AccessMode#Exclusive but instead
> > of failing in case of the presence of other Producers it will force
> > all of the current connected Producers to be removed and invalidated
> > (they will see ProducerFencedException).
> >
> > Reject Alternatives
> >
> > None
Re: [DISCUSS] PIP-157 Exclusive Producer: new mode ExclusiveWithFencing
Posted by Kezhu Wang <ke...@gmail.com>.
Nice improvement.
I was searching for similar functionalities to ban/fence old producers and
found a similar improvement/suggestion from [PIP 68 Suggestions](
https://github.com/apache/pulsar/issues/7584) in addition to PIP 68.
Best,
Kezhu Wang
On May 10, 2022 at 21:59:43, Enrico Olivelli (eolivelli@gmail.com) wrote:
I have renamed this to PIP-161 (there is some mess in the wiki page)
Enrico
Il giorno mar 10 mag 2022 alle ore 14:56 Enrico Olivelli
<eo...@gmail.com> ha scritto:
>
> Hello,
> I created a new PIP about a new AccessMode for the Producer.
> https://github.com/apache/pulsar/issues/15528
>
> This is the PR: https://github.com/apache/pulsar/pull/15488
>
> Honestly I don't like the name "ExclusiveWithFencing", any suggestion
> is really appreciated.
>
> Enrico
>
> Motivation
>
> In PIP-68 we introduced two access modes for the Producer:
>
> Exclusive: The producer is the only one who can publish to the topic.
> Fail if there is another Exclusive Producer connected to the topic
> while creating the new Producer.
> WaitForExclusive: Like Exclusive, but instead of Failing we are going
> to wait for the current Exclusive Producer to disconnect.
>
> Those two modes are very powerful and allow you to perform some kind
> of Locking on a topic.
>
> We are missing a third more, in which the Producer always succeeds to
> acquire the Exclusive lock on the topic by fencing out any other
> Producer that is connected, even the current Exclusive Producer and
> the other Producers waiting in WaitForExclusive mode.
>
> Goal
>
> The modes that are available with PIP-68 require a writer to acquire
> the lock and release it as soon as possible in order to allow other
> clients to write to the topic.
>
> With the new mode it will be possible to implement locking in another
> way: the Producer holds the lock until someone else steals it. This
> way when you have very low contention you can achieve better latency
> for writes because you don't have to acquire the lock every time you
> want to write,.
>
> API and Wire protocol Changes
>
> Changes:
>
> a new constant on the Wire Protocol for AccessMode
> a new constant in the Java Client API AccessMode#ExclusiveWithFencing
>
> Implementation
>
> The new mode will behave mostly like AccessMode#Exclusive but instead
> of failing in case of the presence of other Producers it will force
> all of the current connected Producers to be removed and invalidated
> (they will see ProducerFencedException).
>
> Reject Alternatives
>
> None
Re: [DISCUSS] PIP-157 Exclusive Producer: new mode ExclusiveWithFencing
Posted by Enrico Olivelli <eo...@gmail.com>.
I have renamed this to PIP-161 (there is some mess in the wiki page)
Enrico
Il giorno mar 10 mag 2022 alle ore 14:56 Enrico Olivelli
<eo...@gmail.com> ha scritto:
>
> Hello,
> I created a new PIP about a new AccessMode for the Producer.
> https://github.com/apache/pulsar/issues/15528
>
> This is the PR: https://github.com/apache/pulsar/pull/15488
>
> Honestly I don't like the name "ExclusiveWithFencing", any suggestion
> is really appreciated.
>
> Enrico
>
> Motivation
>
> In PIP-68 we introduced two access modes for the Producer:
>
> Exclusive: The producer is the only one who can publish to the topic.
> Fail if there is another Exclusive Producer connected to the topic
> while creating the new Producer.
> WaitForExclusive: Like Exclusive, but instead of Failing we are going
> to wait for the current Exclusive Producer to disconnect.
>
> Those two modes are very powerful and allow you to perform some kind
> of Locking on a topic.
>
> We are missing a third more, in which the Producer always succeeds to
> acquire the Exclusive lock on the topic by fencing out any other
> Producer that is connected, even the current Exclusive Producer and
> the other Producers waiting in WaitForExclusive mode.
>
> Goal
>
> The modes that are available with PIP-68 require a writer to acquire
> the lock and release it as soon as possible in order to allow other
> clients to write to the topic.
>
> With the new mode it will be possible to implement locking in another
> way: the Producer holds the lock until someone else steals it. This
> way when you have very low contention you can achieve better latency
> for writes because you don't have to acquire the lock every time you
> want to write,.
>
> API and Wire protocol Changes
>
> Changes:
>
> a new constant on the Wire Protocol for AccessMode
> a new constant in the Java Client API AccessMode#ExclusiveWithFencing
>
> Implementation
>
> The new mode will behave mostly like AccessMode#Exclusive but instead
> of failing in case of the presence of other Producers it will force
> all of the current connected Producers to be removed and invalidated
> (they will see ProducerFencedException).
>
> Reject Alternatives
>
> None
Re: [DISCUSS] PIP-157 Exclusive Producer: new mode ExclusiveWithFencing
Posted by Christophe Bornet <bo...@gmail.com>.
+1 for "Preemptive".
Le ven. 27 mai 2022 à 00:22, Neng Lu <fr...@gmail.com> a écrit :
> Hi All,
>
> Would "Preemptive" Mode make sense?
>
> On Wed, May 11, 2022 at 8:56 AM Matteo Merli <ma...@gmail.com>
> wrote:
>
> > +1
> >
> >
> > On Tue, May 10, 2022 at 5:56 AM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Hello,
> > > I created a new PIP about a new AccessMode for the Producer.
> > > https://github.com/apache/pulsar/issues/15528
> > >
> > > This is the PR: https://github.com/apache/pulsar/pull/15488
> > >
> > > Honestly I don't like the name "ExclusiveWithFencing", any suggestion
> > > is really appreciated.
> > >
> > > Enrico
> > >
> > > Motivation
> > >
> > > In PIP-68 we introduced two access modes for the Producer:
> > >
> > > Exclusive: The producer is the only one who can publish to the topic.
> > > Fail if there is another Exclusive Producer connected to the topic
> > > while creating the new Producer.
> > > WaitForExclusive: Like Exclusive, but instead of Failing we are going
> > > to wait for the current Exclusive Producer to disconnect.
> > >
> > > Those two modes are very powerful and allow you to perform some kind
> > > of Locking on a topic.
> > >
> > > We are missing a third more, in which the Producer always succeeds to
> > > acquire the Exclusive lock on the topic by fencing out any other
> > > Producer that is connected, even the current Exclusive Producer and
> > > the other Producers waiting in WaitForExclusive mode.
> > >
> > > Goal
> > >
> > > The modes that are available with PIP-68 require a writer to acquire
> > > the lock and release it as soon as possible in order to allow other
> > > clients to write to the topic.
> > >
> > > With the new mode it will be possible to implement locking in another
> > > way: the Producer holds the lock until someone else steals it. This
> > > way when you have very low contention you can achieve better latency
> > > for writes because you don't have to acquire the lock every time you
> > > want to write,.
> > >
> > > API and Wire protocol Changes
> > >
> > > Changes:
> > >
> > > a new constant on the Wire Protocol for AccessMode
> > > a new constant in the Java Client API AccessMode#ExclusiveWithFencing
> > >
> > > Implementation
> > >
> > > The new mode will behave mostly like AccessMode#Exclusive but instead
> > > of failing in case of the presence of other Producers it will force
> > > all of the current connected Producers to be removed and invalidated
> > > (they will see ProducerFencedException).
> > >
> > > Reject Alternatives
> > >
> > > None
> > >
> > --
> > --
> > Matteo Merli
> > <ma...@gmail.com>
> >
>
>
> --
> Best Regards,
> Neng
>
Re: [DISCUSS] PIP-157 Exclusive Producer: new mode ExclusiveWithFencing
Posted by Neng Lu <fr...@gmail.com>.
Hi All,
Would "Preemptive" Mode make sense?
On Wed, May 11, 2022 at 8:56 AM Matteo Merli <ma...@gmail.com> wrote:
> +1
>
>
> On Tue, May 10, 2022 at 5:56 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Hello,
> > I created a new PIP about a new AccessMode for the Producer.
> > https://github.com/apache/pulsar/issues/15528
> >
> > This is the PR: https://github.com/apache/pulsar/pull/15488
> >
> > Honestly I don't like the name "ExclusiveWithFencing", any suggestion
> > is really appreciated.
> >
> > Enrico
> >
> > Motivation
> >
> > In PIP-68 we introduced two access modes for the Producer:
> >
> > Exclusive: The producer is the only one who can publish to the topic.
> > Fail if there is another Exclusive Producer connected to the topic
> > while creating the new Producer.
> > WaitForExclusive: Like Exclusive, but instead of Failing we are going
> > to wait for the current Exclusive Producer to disconnect.
> >
> > Those two modes are very powerful and allow you to perform some kind
> > of Locking on a topic.
> >
> > We are missing a third more, in which the Producer always succeeds to
> > acquire the Exclusive lock on the topic by fencing out any other
> > Producer that is connected, even the current Exclusive Producer and
> > the other Producers waiting in WaitForExclusive mode.
> >
> > Goal
> >
> > The modes that are available with PIP-68 require a writer to acquire
> > the lock and release it as soon as possible in order to allow other
> > clients to write to the topic.
> >
> > With the new mode it will be possible to implement locking in another
> > way: the Producer holds the lock until someone else steals it. This
> > way when you have very low contention you can achieve better latency
> > for writes because you don't have to acquire the lock every time you
> > want to write,.
> >
> > API and Wire protocol Changes
> >
> > Changes:
> >
> > a new constant on the Wire Protocol for AccessMode
> > a new constant in the Java Client API AccessMode#ExclusiveWithFencing
> >
> > Implementation
> >
> > The new mode will behave mostly like AccessMode#Exclusive but instead
> > of failing in case of the presence of other Producers it will force
> > all of the current connected Producers to be removed and invalidated
> > (they will see ProducerFencedException).
> >
> > Reject Alternatives
> >
> > None
> >
> --
> --
> Matteo Merli
> <ma...@gmail.com>
>
--
Best Regards,
Neng
Re: [DISCUSS] PIP-157 Exclusive Producer: new mode ExclusiveWithFencing
Posted by Matteo Merli <ma...@gmail.com>.
+1
On Tue, May 10, 2022 at 5:56 AM Enrico Olivelli <eo...@gmail.com> wrote:
> Hello,
> I created a new PIP about a new AccessMode for the Producer.
> https://github.com/apache/pulsar/issues/15528
>
> This is the PR: https://github.com/apache/pulsar/pull/15488
>
> Honestly I don't like the name "ExclusiveWithFencing", any suggestion
> is really appreciated.
>
> Enrico
>
> Motivation
>
> In PIP-68 we introduced two access modes for the Producer:
>
> Exclusive: The producer is the only one who can publish to the topic.
> Fail if there is another Exclusive Producer connected to the topic
> while creating the new Producer.
> WaitForExclusive: Like Exclusive, but instead of Failing we are going
> to wait for the current Exclusive Producer to disconnect.
>
> Those two modes are very powerful and allow you to perform some kind
> of Locking on a topic.
>
> We are missing a third more, in which the Producer always succeeds to
> acquire the Exclusive lock on the topic by fencing out any other
> Producer that is connected, even the current Exclusive Producer and
> the other Producers waiting in WaitForExclusive mode.
>
> Goal
>
> The modes that are available with PIP-68 require a writer to acquire
> the lock and release it as soon as possible in order to allow other
> clients to write to the topic.
>
> With the new mode it will be possible to implement locking in another
> way: the Producer holds the lock until someone else steals it. This
> way when you have very low contention you can achieve better latency
> for writes because you don't have to acquire the lock every time you
> want to write,.
>
> API and Wire protocol Changes
>
> Changes:
>
> a new constant on the Wire Protocol for AccessMode
> a new constant in the Java Client API AccessMode#ExclusiveWithFencing
>
> Implementation
>
> The new mode will behave mostly like AccessMode#Exclusive but instead
> of failing in case of the presence of other Producers it will force
> all of the current connected Producers to be removed and invalidated
> (they will see ProducerFencedException).
>
> Reject Alternatives
>
> None
>
--
--
Matteo Merli
<ma...@gmail.com>