You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Enrico Olivelli <eo...@gmail.com> on 2021/04/21 11:23:49 UTC
Transaction Manager Coordinator Deployment
Hello,
Pulsar 2.8.0 is coming near to the release and we want transactions to
be "production ready".
If I understand correctly now we start the Transaction Coordinator
service inside the Broker.
Does it make sense to have the option to run the TC as a separate
service, as we do for the Functions Worker ?
For instance in a big cluster, with many brokers, an user may like to
dedicate machines to the role of Transaction Coordinator and do not
serve all of the usual Broker activities.
The same problem will also go into the Helm Chart, it would be better
to have the ability to configure the transaction related services with
specific options.
What do you think ?
Enrico
Re: Transaction Manager Coordinator Deployment
Posted by Devin Bost <de...@gmail.com>.
How computationally intensive is the responsibility of the Transaction
Coordinator?
--
Devin G. Bost
On Wed, Apr 21, 2021, 3:01 PM Sijie Guo <gu...@gmail.com> wrote:
> Enrico - I was suggesting not even making it a standalone. TC is a
> component that really tied to Pulsar protocol and logic. We shouldn't do it
> until it is really needed.
>
> - Sijie
>
> On Wed, Apr 21, 2021 at 1:21 PM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Il giorno mer 21 apr 2021 alle ore 21:57 Sijie Guo
> > <gu...@gmail.com> ha scritto:
> > >
> > > We build transaction components in modules. It provides the flexibility
> > for
> > > people to run those components independently.
> > >
> > > However, I don't think we should encourage this deployment at the
> > > beginning. Because most of the users don't have deep knowledge about
> how
> > > the transaction works and have no idea how to manage them if you run
> them
> > > separately. Hence my recommendation is that we don't add such a
> > deployment
> > > option in the documentation and helm chart. It will just confuse users.
> >
> > I am fine with not documenting how to do it for the 2.8 release, the
> > same for the official Apache Pulsar Helm Chart.
> >
> > but currently there is no way to deploy the Transaction Coordinator as
> > a standalone module,
> > if there is a way to launch an isolated TC please let me know
> > otherwise I will at least create an issue on github and hopefully we
> > can implement it for Pulsar 2.9.0, when transactions will be hopefully
> > considered fully stable.
> >
> > Enrico
> >
> >
> >
> >
> >
> > >
> > > - Sijie
> > >
> > > On Wed, Apr 21, 2021 at 8:59 AM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> > >
> > > > Il giorno mer 21 apr 2021 alle ore 16:18 PengHui Li
> > > > <co...@gmail.com> ha scritto:
> > > > >
> > > > > Hi Enrico,
> > > > >
> > > > > The current implementation of the transaction coordinator is run
> with
> > > > the broker, but the transaction coordinator
> > > > > Is a separate module in Pulsar. This is also mentioned in PIP-31
> > > > https://github.com/apache/pulsar/wiki/PIP-31%3A-Transaction-Support
> > > >
> > > > Great.
> > > > Is there any ongoing effort to add the ability to run the TC outside
> > > > of the broker process ?
> > > >
> > > > Enrico
> > > >
> > > > >
> > > > > Thanks,
> > > > > Penghui
> > > > > On Apr 21, 2021, 7:24 PM +0800, Enrico Olivelli <
> eolivelli@gmail.com
> > >,
> > > > wrote:
> > > > > > Hello,
> > > > > > Pulsar 2.8.0 is coming near to the release and we want
> > transactions to
> > > > > > be "production ready".
> > > > > >
> > > > > > If I understand correctly now we start the Transaction
> Coordinator
> > > > > > service inside the Broker.
> > > > > >
> > > > > > Does it make sense to have the option to run the TC as a separate
> > > > > > service, as we do for the Functions Worker ?
> > > > > >
> > > > > > For instance in a big cluster, with many brokers, an user may
> like
> > to
> > > > > > dedicate machines to the role of Transaction Coordinator and do
> not
> > > > > > serve all of the usual Broker activities.
> > > > > >
> > > > > > The same problem will also go into the Helm Chart, it would be
> > better
> > > > > > to have the ability to configure the transaction related services
> > with
> > > > > > specific options.
> > > > > >
> > > > > > What do you think ?
> > > > > >
> > > > > >
> > > > > > Enrico
> > > >
> >
>
Re: Transaction Manager Coordinator Deployment
Posted by Sijie Guo <gu...@gmail.com>.
Enrico - I was suggesting not even making it a standalone. TC is a
component that really tied to Pulsar protocol and logic. We shouldn't do it
until it is really needed.
- Sijie
On Wed, Apr 21, 2021 at 1:21 PM Enrico Olivelli <eo...@gmail.com> wrote:
> Il giorno mer 21 apr 2021 alle ore 21:57 Sijie Guo
> <gu...@gmail.com> ha scritto:
> >
> > We build transaction components in modules. It provides the flexibility
> for
> > people to run those components independently.
> >
> > However, I don't think we should encourage this deployment at the
> > beginning. Because most of the users don't have deep knowledge about how
> > the transaction works and have no idea how to manage them if you run them
> > separately. Hence my recommendation is that we don't add such a
> deployment
> > option in the documentation and helm chart. It will just confuse users.
>
> I am fine with not documenting how to do it for the 2.8 release, the
> same for the official Apache Pulsar Helm Chart.
>
> but currently there is no way to deploy the Transaction Coordinator as
> a standalone module,
> if there is a way to launch an isolated TC please let me know
> otherwise I will at least create an issue on github and hopefully we
> can implement it for Pulsar 2.9.0, when transactions will be hopefully
> considered fully stable.
>
> Enrico
>
>
>
>
>
> >
> > - Sijie
> >
> > On Wed, Apr 21, 2021 at 8:59 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
> >
> > > Il giorno mer 21 apr 2021 alle ore 16:18 PengHui Li
> > > <co...@gmail.com> ha scritto:
> > > >
> > > > Hi Enrico,
> > > >
> > > > The current implementation of the transaction coordinator is run with
> > > the broker, but the transaction coordinator
> > > > Is a separate module in Pulsar. This is also mentioned in PIP-31
> > > https://github.com/apache/pulsar/wiki/PIP-31%3A-Transaction-Support
> > >
> > > Great.
> > > Is there any ongoing effort to add the ability to run the TC outside
> > > of the broker process ?
> > >
> > > Enrico
> > >
> > > >
> > > > Thanks,
> > > > Penghui
> > > > On Apr 21, 2021, 7:24 PM +0800, Enrico Olivelli <eolivelli@gmail.com
> >,
> > > wrote:
> > > > > Hello,
> > > > > Pulsar 2.8.0 is coming near to the release and we want
> transactions to
> > > > > be "production ready".
> > > > >
> > > > > If I understand correctly now we start the Transaction Coordinator
> > > > > service inside the Broker.
> > > > >
> > > > > Does it make sense to have the option to run the TC as a separate
> > > > > service, as we do for the Functions Worker ?
> > > > >
> > > > > For instance in a big cluster, with many brokers, an user may like
> to
> > > > > dedicate machines to the role of Transaction Coordinator and do not
> > > > > serve all of the usual Broker activities.
> > > > >
> > > > > The same problem will also go into the Helm Chart, it would be
> better
> > > > > to have the ability to configure the transaction related services
> with
> > > > > specific options.
> > > > >
> > > > > What do you think ?
> > > > >
> > > > >
> > > > > Enrico
> > >
>
Re: Transaction Manager Coordinator Deployment
Posted by Enrico Olivelli <eo...@gmail.com>.
Il giorno mer 21 apr 2021 alle ore 21:57 Sijie Guo
<gu...@gmail.com> ha scritto:
>
> We build transaction components in modules. It provides the flexibility for
> people to run those components independently.
>
> However, I don't think we should encourage this deployment at the
> beginning. Because most of the users don't have deep knowledge about how
> the transaction works and have no idea how to manage them if you run them
> separately. Hence my recommendation is that we don't add such a deployment
> option in the documentation and helm chart. It will just confuse users.
I am fine with not documenting how to do it for the 2.8 release, the
same for the official Apache Pulsar Helm Chart.
but currently there is no way to deploy the Transaction Coordinator as
a standalone module,
if there is a way to launch an isolated TC please let me know
otherwise I will at least create an issue on github and hopefully we
can implement it for Pulsar 2.9.0, when transactions will be hopefully
considered fully stable.
Enrico
>
> - Sijie
>
> On Wed, Apr 21, 2021 at 8:59 AM Enrico Olivelli <eo...@gmail.com> wrote:
>
> > Il giorno mer 21 apr 2021 alle ore 16:18 PengHui Li
> > <co...@gmail.com> ha scritto:
> > >
> > > Hi Enrico,
> > >
> > > The current implementation of the transaction coordinator is run with
> > the broker, but the transaction coordinator
> > > Is a separate module in Pulsar. This is also mentioned in PIP-31
> > https://github.com/apache/pulsar/wiki/PIP-31%3A-Transaction-Support
> >
> > Great.
> > Is there any ongoing effort to add the ability to run the TC outside
> > of the broker process ?
> >
> > Enrico
> >
> > >
> > > Thanks,
> > > Penghui
> > > On Apr 21, 2021, 7:24 PM +0800, Enrico Olivelli <eo...@gmail.com>,
> > wrote:
> > > > Hello,
> > > > Pulsar 2.8.0 is coming near to the release and we want transactions to
> > > > be "production ready".
> > > >
> > > > If I understand correctly now we start the Transaction Coordinator
> > > > service inside the Broker.
> > > >
> > > > Does it make sense to have the option to run the TC as a separate
> > > > service, as we do for the Functions Worker ?
> > > >
> > > > For instance in a big cluster, with many brokers, an user may like to
> > > > dedicate machines to the role of Transaction Coordinator and do not
> > > > serve all of the usual Broker activities.
> > > >
> > > > The same problem will also go into the Helm Chart, it would be better
> > > > to have the ability to configure the transaction related services with
> > > > specific options.
> > > >
> > > > What do you think ?
> > > >
> > > >
> > > > Enrico
> >
Re: Transaction Manager Coordinator Deployment
Posted by Sijie Guo <gu...@gmail.com>.
We build transaction components in modules. It provides the flexibility for
people to run those components independently.
However, I don't think we should encourage this deployment at the
beginning. Because most of the users don't have deep knowledge about how
the transaction works and have no idea how to manage them if you run them
separately. Hence my recommendation is that we don't add such a deployment
option in the documentation and helm chart. It will just confuse users.
- Sijie
On Wed, Apr 21, 2021 at 8:59 AM Enrico Olivelli <eo...@gmail.com> wrote:
> Il giorno mer 21 apr 2021 alle ore 16:18 PengHui Li
> <co...@gmail.com> ha scritto:
> >
> > Hi Enrico,
> >
> > The current implementation of the transaction coordinator is run with
> the broker, but the transaction coordinator
> > Is a separate module in Pulsar. This is also mentioned in PIP-31
> https://github.com/apache/pulsar/wiki/PIP-31%3A-Transaction-Support
>
> Great.
> Is there any ongoing effort to add the ability to run the TC outside
> of the broker process ?
>
> Enrico
>
> >
> > Thanks,
> > Penghui
> > On Apr 21, 2021, 7:24 PM +0800, Enrico Olivelli <eo...@gmail.com>,
> wrote:
> > > Hello,
> > > Pulsar 2.8.0 is coming near to the release and we want transactions to
> > > be "production ready".
> > >
> > > If I understand correctly now we start the Transaction Coordinator
> > > service inside the Broker.
> > >
> > > Does it make sense to have the option to run the TC as a separate
> > > service, as we do for the Functions Worker ?
> > >
> > > For instance in a big cluster, with many brokers, an user may like to
> > > dedicate machines to the role of Transaction Coordinator and do not
> > > serve all of the usual Broker activities.
> > >
> > > The same problem will also go into the Helm Chart, it would be better
> > > to have the ability to configure the transaction related services with
> > > specific options.
> > >
> > > What do you think ?
> > >
> > >
> > > Enrico
>
Re: Transaction Manager Coordinator Deployment
Posted by Enrico Olivelli <eo...@gmail.com>.
Il giorno mer 21 apr 2021 alle ore 16:18 PengHui Li
<co...@gmail.com> ha scritto:
>
> Hi Enrico,
>
> The current implementation of the transaction coordinator is run with the broker, but the transaction coordinator
> Is a separate module in Pulsar. This is also mentioned in PIP-31 https://github.com/apache/pulsar/wiki/PIP-31%3A-Transaction-Support
Great.
Is there any ongoing effort to add the ability to run the TC outside
of the broker process ?
Enrico
>
> Thanks,
> Penghui
> On Apr 21, 2021, 7:24 PM +0800, Enrico Olivelli <eo...@gmail.com>, wrote:
> > Hello,
> > Pulsar 2.8.0 is coming near to the release and we want transactions to
> > be "production ready".
> >
> > If I understand correctly now we start the Transaction Coordinator
> > service inside the Broker.
> >
> > Does it make sense to have the option to run the TC as a separate
> > service, as we do for the Functions Worker ?
> >
> > For instance in a big cluster, with many brokers, an user may like to
> > dedicate machines to the role of Transaction Coordinator and do not
> > serve all of the usual Broker activities.
> >
> > The same problem will also go into the Helm Chart, it would be better
> > to have the ability to configure the transaction related services with
> > specific options.
> >
> > What do you think ?
> >
> >
> > Enrico
Re: Transaction Manager Coordinator Deployment
Posted by PengHui Li <co...@gmail.com>.
Hi Enrico,
The current implementation of the transaction coordinator is run with the broker, but the transaction coordinator
Is a separate module in Pulsar. This is also mentioned in PIP-31 https://github.com/apache/pulsar/wiki/PIP-31%3A-Transaction-Support
Thanks,
Penghui
On Apr 21, 2021, 7:24 PM +0800, Enrico Olivelli <eo...@gmail.com>, wrote:
> Hello,
> Pulsar 2.8.0 is coming near to the release and we want transactions to
> be "production ready".
>
> If I understand correctly now we start the Transaction Coordinator
> service inside the Broker.
>
> Does it make sense to have the option to run the TC as a separate
> service, as we do for the Functions Worker ?
>
> For instance in a big cluster, with many brokers, an user may like to
> dedicate machines to the role of Transaction Coordinator and do not
> serve all of the usual Broker activities.
>
> The same problem will also go into the Helm Chart, it would be better
> to have the ability to configure the transaction related services with
> specific options.
>
> What do you think ?
>
>
> Enrico
Re: Transaction Manager Coordinator Deployment
Posted by Chris Herzog <ch...@tibco.com.INVALID>.
Modularity like this is a huge plus for production deployments - it gives
the ability to scale more effectively and maintain a closer match between
resources and services where appropriate.
I think it's a solid idea.
We're almost certainly going to want to end up with this level of control
so it makes sense to ensure the framework is in place from the get go. That
will result in fewer potential design dead ends as well as cleaner upgrades
in the future because we'll have that many fewer significant paradigm
shifts.
On Wed, Apr 21, 2021 at 6:24 AM Enrico Olivelli <eo...@gmail.com> wrote:
> Hello,
> Pulsar 2.8.0 is coming near to the release and we want transactions to
> be "production ready".
>
> If I understand correctly now we start the Transaction Coordinator
> service inside the Broker.
>
> Does it make sense to have the option to run the TC as a separate
> service, as we do for the Functions Worker ?
>
> For instance in a big cluster, with many brokers, an user may like to
> dedicate machines to the role of Transaction Coordinator and do not
> serve all of the usual Broker activities.
>
> The same problem will also go into the Helm Chart, it would be better
> to have the ability to configure the transaction related services with
> specific options.
>
> What do you think ?
>
>
> Enrico
>
--
Chris Herzog Messaging Team | O 630 300 7718 | M 815 263 3764 |
www.tibco.com
<http://www.tibco.com/>