You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Yuri Mizushima <yu...@yahoo-corp.jp> on 2021/11/16 06:43:27 UTC
Re: [PIP-78] Reduce redundant producers from partitioned producer
Dear Pulsar community,
I have created a new PR https://github.com/apache/pulsar/pull/12401 for stats aggregation,
but I didn't discuss about the wire protocol change. I hope we will discuss it here.
Currently, partitioned producer can't aggregate by any key such as cnx, producerId, producerName, and so on.
I think we need to add any aggregation system.
Therefore, added new aggregation policy as producerName (with client side implementation).
New protocol field partial_producer_supported is not used for stats aggregation. It is used for backward compatibility.
https://github.com/apache/pulsar/pull/12401/files#diff-f29399fed32e0916cf28452ba71078a3ae5ed77bbaef9f52a925165d8ee66cfdR489
In my understanding, if introduce new stats aggregation key to client side,
need a way to determine whether the feature is enabled at client side.
For example, whether the producer has specific field or metadata,
the version (e.g. protocol version) is greater than threshold, etc.
Of course, if we can introduce aggregation feature without adding any new key or implementations from client side,
we can support the feature not only new client but also old one.
I'm looking forward to your opinions or suggestions to this PR.
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
On 2021/05/11 14:26, "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
Dear Pulsar Community,
> I will submit the next PR about PartitionedTopicStats later.
I submitted the next PR for this PIP. If you have any suggestions, please comment to this PR.
https://github.com/apache/pulsar/pull/10534
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
"Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
Dear Pulsar Community,
I submitted the PR for this PIP.
https://github.com/apache/pulsar/pull/10279
This is a part of implementations.
I will submit the next PR about PartitionedTopicStats later.
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
"Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
Sijie,
After sending previous mail, I watched meeting recording and understand about authn/authz issue.
Therefore, I updated the PIP document.
https://github.com/apache/pulsar/wiki/PIP-79%3A-Reduce-redundant-producers-from-partitioned-producer
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
"Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
Sijie,
> If the lazy-loading approach sounds attractive to you and you like it,
> maybe the next step is to update the PIP, what do you think?
I think so too. I will update the PIP after discussing the authn/authz issue.
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
"Sijie Guo" <gu...@gmail.com> wrote:
Hi Yuri,
Regarding the authn/authz issue, @Matteo Merli <mm...@apache.org> can
probably chime in more about that part.
If the lazy-loading approach sounds attractive to you and you like it,
maybe the next step is to update the PIP, what do you think?
- Sijie
On Mon, Feb 8, 2021 at 6:57 PM Yuri Mizushima <yu...@yahoo-corp.jp>
wrote:
> Michael,
>
> Thank you for your comment!
>
> > Which Pulsar Clients will benefit from this proposal?
> I think that this proposal will be useful to any clients.
> In my schedule, if this proposal is accepted then I will implement this
> feature to Java client.
> If needed, then implement same feature to other clients such as C++, Go,
> etc.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Michael Marshall" <mi...@gmail.com> wrote:
>
> Hi Yuri and Sijie,
>
> I definitely like the idea of lazily creating producers as well as
> introducing a way to provide custom routing logic.
>
> Which Pulsar Clients will benefit from this proposal? I’d love to see
> this feature in the go client.
>
> Thanks,
> Michael Marshall
>
> > On Feb 7, 2021, at 9:53 PM, Yuri Mizushima <yu...@yahoo-corp.jp>
> wrote:
> >
> > Sijie,
> >
> > Thank you for sharing!
> >
> > First, I considered your suggestion.
> > I think these implementations sound good.
> >
> > I think we should consider the State of partitioned producer: Ready,
> Connecting, etc.
> > Currently, partitioned producer gets "Ready" only when all producers
> connect to Broker correctly.
> >
> https://github.com/apache/pulsar/blob/fa41d02bebfd841767846240f3ae574047f118f0/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java#L146
> > It seems that we should change meaning of state (or change handling)
> if we introduce the lazy-load feature.
> > To guarantee the message ordering (e.g. using partitionKey),
> partitioned producer should stop (or don't send messages to be routed to
> unavailable partition) when producer can't connect to one of partition.
> >
> > Secondly, I considered Matteo's comments.
> > I couldn't understand well about issue of authn/authz. Please tell
> me more detail.
> >
> > I wrote "connection" as number of producers which connect to broker.
> Also, TCP connections between partitioned producer and broker will be less
> than or equal to current in some cases. I'll show a case below.
> >
> > Suppose
> > * cluster has Broker0, 1, 2
> > * partitioned topic has 5 partitions
> > * limit conf is 3 partitions
> > * loadbalance partitions as below
> > - Broker0: partition-0, partition-1
> > - Broker1: partition-2
> > - Broker2: partition-3, partition-4
> >
> > Currently, client will create 3 connections (Broker0, 1, 2). If
> client uses limit conf and elects partitions such as [0, 1, 2], then client
> will create 2 connections (Broker0, 1). Of course, if client elects
> partitions such as [0, 2, 3], then client will still create 3 connections.
> >
> > I'd like to decrease number of producers. I think that resources of
> broker will be improved slightly by this feature because broker has list of
> producers by some classes such as ServerCnx, AbstractTopic.
> >
> https://github.com/apache/pulsar/blob/fa41d02bebfd841767846240f3ae574047f118f0/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java#L1096-L1097
> >
> https://github.com/apache/pulsar/blob/fa41d02bebfd841767846240f3ae574047f118f0/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java#L577
> >
> > In my case, unspecified number of producers will connect to the same
> partitioned topic with different rate. We need to set the number of
> partitions according to the high-rate producer.
> > However, on the other hand, this number is excessively large for
> low-rate producers.
> > I want to reduce such redundant producers for resource efficiency.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Sijie Guo" <gu...@gmail.com> wrote:
> >
> > Hi Yuri,
> >
> > In today's community meeting, Matteo shared some of his thoughts
> about this
> > PIP.
> >
> > You can find some meeting notes here:
> >
> https://docs.google.com/document/d/19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE/edit#bookmark=id.rezbt4xmjxpz
> >
> > Matteo can also chime in as well.
> >
> > - Sijie
> >
> >> On Sun, Jan 31, 2021 at 7:21 PM Yuri Mizushima <
> yumizush@yahoo-corp.jp>
> >> wrote:
> >> Sijie,
> >> Thank you for your reply!
> >> I'll check it.
> >> Regards,
> >> --
> >> Yuri Mizushima
> >> yumizush@yahoo-corp.jp
> >> "Sijie Guo" <gu...@gmail.com> wrote:
> >> Yuri,
> >> Thank you for bringing this up! This is a super helpful proposal!
> >> The problem is very similar to what an RPC framework (like Finagle)
> >> with
> >> client-side load balancing has.
> >> An RPC framework with a client-side load-balancing mechanism needs
> to
> >> send
> >> requests across multiple nodes. If you have an RPC service that has
> >> thousands of nodes, there are thousands of clients connecting to
> that
> >> RPC
> >> service. How to reduce the connections and how to effectively load
> >> balance
> >> requests across thousands of nodes are the problems that a
> client-side
> >> loading technology needs to solve. If you think about "partition"
> as
> >> "node"
> >> and "partitioned producer" as "RPC client", the problem is exactly
> the
> >> same. Finagle (the Twitter RPC framework) has implemented a lot of
> >> client-side
> >> load-balancing algorithms
> >> <
> https://twitter.github.io/finagle/guide/Clients.html#load-balancing>
> >> and
> >> there are some great articles that you can reference
> >> <
> >>
> https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer.html
> >> .
> >> I agree with the direction of introducing a mechanism to reduce the
> >> number
> >> of producers in a partitioned topic producer. However, I have a
> concern
> >> about introducing `.numPartitionsLimit(10)` directly to the
> producer
> >> builder. It limits the possibility to implement different
> algorithms on
> >> selecting partitions.
> >> So instead of directly implementing the logic within the
> partitioned
> >> topic
> >> producer, I think the proposal can be broken into two parts:
> >> 1) Introduce some kind of lazy-loading mechanism in the partitioned
> >> producer to initialize the producers for partitions lazily. I.e.,
> only
> >> initialize a producer when the message router selects a partition.
> >> 2) Implement a message router that only selects one or N
> partitions.
> >> In this way, the partitioned producer is only responsible for
> managing
> >> a
> >> collection of producers, and the message router is responsible for
> >> selecting the partitions. This allows people to be able to
> implement
> >> different message routers. We can even adopt the client-side load
> >> balancing
> >> algorithms from Finagle.
> >> Thanks,
> >> Sijie
> >> On Wed, Jan 27, 2021 at 7:18 PM Yuri Mizushima <
> yumizush@yahoo-corp.jp
> >> wrote:
> >>> I notice that PIP-78 has already assigned to another issue.
> >>
> https://mail-archives.apache.org/mod_mbox/pulsar-dev/202101.mbox/%3CCAG%3DTQOrPH49v9ToDE_aeQzEiDC%2BEgSR61ERoqanpWfQGvEB_Vw%40mail.gmail.com%3E
> >>> So, I'll change the PIP number to 79.
> >>
> https://github.com/apache/pulsar/wiki/PIP-79%3A-Reduce-redundant-producers-from-partitioned-producer
> >>> Regards,
> >>> --
> >>> Yuri Mizushima
> >>> yumizush@yahoo-corp.jp
> >>> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
> >>> Dear Pulsar community,
> >>> When partitioned producer connects to partitioned topic,
> >>> sometimes doesn't need to connect to all of partitions depending
> >> on
> >>> rate, routing mode, etc.
> >>> So, I drafted a PIP about reducing redundant producers from
> >>> partitioned producer.
> >>> I'd like to use system resources (e.g. connections between
> >> Client and
> >>> Broker, memory usage of both Client and Broker)
> >>> more efficiently by this feature.
> >>
> https://github.com/apache/pulsar/wiki/PIP-78%3A-Reduce-redundant-producers-from-partitioned-producer
> >>> Feel free to ask me any questions or suggestions, etc.
> >>> Best regards,
> >>> --
> >>> Yuri Mizushima
> >>> yumizush@yahoo-corp.jp
>
>
Re: [PIP-78] Reduce redundant producers from partitioned producer
Posted by Yuri Mizushima <yu...@yahoo-corp.jp>.
Any thought on this?
--
Yuri Mizushima
yumizush@yahoo-corp.jp
On 2021/12/07 13:57, "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
Enrico,
Thank you for your comment.
> IIUC with this change the client will control which metrics are reported by
> the broker ?
From the protocol perspective, yes.
However, the main point of this change is not to "control" metrics by the client side,
but to make the broker aggregate partitioned topic's producer metrics explicitly.
Do you suggest adding a broker config that
configures whether partitioned producer stats are aggregated by producerName instead of
introducing a backward compatibility key (i.e., partial_producer_supported) on the client-side?
It is simple. However, I think we can't enable the config until all clients are updated.
What do you think?
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
On 2021/12/02 17:37, "Enrico Olivelli" <eo...@gmail.com> wrote:
Yuri,
IIUC with this change the client will control which metrics are reported by
the broker ?
I am not sure it is a good idea, because metrics are usually managed by the
owners of the brokers, who sometimes are not the same who run the clients.
Also, I am not sure if this way it is possible for the client to force the
Broker to create many metrics and create some kind of damage.
Would it be better to add a Broker configuration flag to turn on this
feature ? I mean to allow the client to select the type of metrics ?
Enrico
Il giorno gio 2 dic 2021 alle ore 03:00 Yuri Mizushima <
yumizush@yahoo-corp.jp> ha scritto:
> Do you have any comments?
> If there are no comments by Dec. 7, I will close the discussion and rebase
> the PR commit to current master.
>
> Regards,
>
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> On 2021/11/16 15:46, "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Dear Pulsar community,
>
> I have created a new PR
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709588901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qcUOpCoQZZH%2B%2FlobXMhgcNXmlaSyaahXMel9YBmGEWs%3D&reserved=0
> for stats aggregation,
> but I didn't discuss about the wire protocol change. I hope we will
> discuss it here.
>
> Currently, partitioned producer can't aggregate by any key such as
> cnx, producerId, producerName, and so on.
> I think we need to add any aggregation system.
> Therefore, added new aggregation policy as producerName (with client
> side implementation).
>
> New protocol field partial_producer_supported is not used for stats
> aggregation. It is used for backward compatibility.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401%2Ffiles%23diff-f29399fed32e0916cf28452ba71078a3ae5ed77bbaef9f52a925165d8ee66cfdR489&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709588901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wFBZwBwQ4yGUF9VlUdoOhKb4K54ZF0yGwv9kycvxoqI%3D&reserved=0
>
> In my understanding, if introduce new stats aggregation key to client
> side,
> need a way to determine whether the feature is enabled at client side.
> For example, whether the producer has specific field or metadata,
> the version (e.g. protocol version) is greater than threshold, etc.
>
> Of course, if we can introduce aggregation feature without adding any
> new key or implementations from client side,
> we can support the feature not only new client but also old one.
>
> I'm looking forward to your opinions or suggestions to this PR.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> On 2021/05/11 14:26, "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
>
> Dear Pulsar Community,
>
> > I will submit the next PR about PartitionedTopicStats later.
> I submitted the next PR for this PIP. If you have any suggestions,
> please comment to this PR.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10534&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rTJBlQb5ObE7SaAvTrTiM2HA3yhKyPS%2BxuPcWXv6luI%3D&reserved=0
>
> Regards,
>
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Dear Pulsar Community,
>
> I submitted the PR for this PIP.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10279&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=d20Pg8yDYGd2dVb0iJo9eeRZh2YymScPO7wwVCIiAL8%3D&reserved=0
>
> This is a part of implementations.
> I will submit the next PR about PartitionedTopicStats later.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Sijie,
>
> After sending previous mail, I watched meeting recording and
> understand about authn/authz issue.
> Therefore, I updated the PIP document.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=moRiB9I9Zwz%2FFcDAWxza1%2BV6Xcl1xf1pD524%2BMZB2Lk%3D&reserved=0
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Sijie,
>
> > If the lazy-loading approach sounds attractive to you
> and you like it,
> > maybe the next step is to update the PIP, what do you
> think?
>
> I think so too. I will update the PIP after discussing the
> authn/authz issue.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Sijie Guo" <gu...@gmail.com> wrote:
>
> Hi Yuri,
>
> Regarding the authn/authz issue, @Matteo Merli <
> mmerli@apache.org> can
> probably chime in more about that part.
>
> If the lazy-loading approach sounds attractive to you
> and you like it,
> maybe the next step is to update the PIP, what do you
> think?
>
> - Sijie
>
> On Mon, Feb 8, 2021 at 6:57 PM Yuri Mizushima <
> yumizush@yahoo-corp.jp>
> wrote:
>
> > Michael,
> >
> > Thank you for your comment!
> >
> > > Which Pulsar Clients will benefit from this
> proposal?
> > I think that this proposal will be useful to any
> clients.
> > In my schedule, if this proposal is accepted then I
> will implement this
> > feature to Java client.
> > If needed, then implement same feature to other
> clients such as C++, Go,
> > etc.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Michael Marshall" <mi...@gmail.com> wrote:
> >
> > Hi Yuri and Sijie,
> >
> > I definitely like the idea of lazily creating
> producers as well as
> > introducing a way to provide custom routing logic.
> >
> > Which Pulsar Clients will benefit from this
> proposal? I’d love to see
> > this feature in the go client.
> >
> > Thanks,
> > Michael Marshall
> >
> > > On Feb 7, 2021, at 9:53 PM, Yuri Mizushima <
> yumizush@yahoo-corp.jp>
> > wrote:
> > >
> > > Sijie,
> > >
> > > Thank you for sharing!
> > >
> > > First, I considered your suggestion.
> > > I think these implementations sound good.
> > >
> > > I think we should consider the State of
> partitioned producer: Ready,
> > Connecting, etc.
> > > Currently, partitioned producer gets "Ready"
> only when all producers
> > connect to Broker correctly.
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FPartitionedProducerImpl.java%23L146&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Wsfwi9SoiFizW1ATvt5nvb6%2BHFRGiWQzeKNH06hBt1I%3D&reserved=0
> > > It seems that we should change meaning of
> state (or change handling)
> > if we introduce the lazy-load feature.
> > > To guarantee the message ordering (e.g. using
> partitionKey),
> > partitioned producer should stop (or don't send
> messages to be routed to
> > unavailable partition) when producer can't connect
> to one of partition.
> > >
> > > Secondly, I considered Matteo's comments.
> > > I couldn't understand well about issue of
> authn/authz. Please tell
> > me more detail.
> > >
> > > I wrote "connection" as number of producers
> which connect to broker.
> > Also, TCP connections between partitioned producer
> and broker will be less
> > than or equal to current in some cases. I'll show a
> case below.
> > >
> > > Suppose
> > > * cluster has Broker0, 1, 2
> > > * partitioned topic has 5 partitions
> > > * limit conf is 3 partitions
> > > * loadbalance partitions as below
> > > - Broker0: partition-0, partition-1
> > > - Broker1: partition-2
> > > - Broker2: partition-3, partition-4
> > >
> > > Currently, client will create 3 connections
> (Broker0, 1, 2). If
> > client uses limit conf and elects partitions such as
> [0, 1, 2], then client
> > will create 2 connections (Broker0, 1). Of course,
> if client elects
> > partitions such as [0, 2, 3], then client will still
> create 3 connections.
> > >
> > > I'd like to decrease number of producers. I
> think that resources of
> > broker will be improved slightly by this feature
> because broker has list of
> > producers by some classes such as ServerCnx,
> AbstractTopic.
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FServerCnx.java%23L1096-L1097&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DVk%2BCh8F0VeWBay%2FiK7O%2BjCjzZrakOj7257POW8uEgA%3D&reserved=0
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FAbstractTopic.java%23L577&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=JkkE01hdRUB%2B%2BAwykUKuIZmcbn%2FBfBEHe2oeO7%2BbuQs%3D&reserved=0
> > >
> > > In my case, unspecified number of producers
> will connect to the same
> > partitioned topic with different rate. We need to
> set the number of
> > partitions according to the high-rate producer.
> > > However, on the other hand, this number is
> excessively large for
> > low-rate producers.
> > > I want to reduce such redundant producers for
> resource efficiency.
> > >
> > > Regards,
> > > --
> > > Yuri Mizushima
> > > yumizush@yahoo-corp.jp
> > >
> > >
> > > "Sijie Guo" <gu...@gmail.com> wrote:
> > >
> > > Hi Yuri,
> > >
> > > In today's community meeting, Matteo shared
> some of his thoughts
> > about this
> > > PIP.
> > >
> > > You can find some meeting notes here:
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE%2Fedit%23bookmark%3Did.rezbt4xmjxpz&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DkvO95bjPq75PtpbAKjATwmFonPWBqPn%2B38LukhiFXg%3D&reserved=0
> > >
> > > Matteo can also chime in as well.
> > >
> > > - Sijie
> > >
> > >> On Sun, Jan 31, 2021 at 7:21 PM Yuri
> Mizushima <
> > yumizush@yahoo-corp.jp>
> > >> wrote:
> > >> Sijie,
> > >> Thank you for your reply!
> > >> I'll check it.
> > >> Regards,
> > >> --
> > >> Yuri Mizushima
> > >> yumizush@yahoo-corp.jp
> > >> "Sijie Guo" <gu...@gmail.com> wrote:
> > >> Yuri,
> > >> Thank you for bringing this up! This is a
> super helpful proposal!
> > >> The problem is very similar to what an RPC
> framework (like Finagle)
> > >> with
> > >> client-side load balancing has.
> > >> An RPC framework with a client-side
> load-balancing mechanism needs
> > to
> > >> send
> > >> requests across multiple nodes. If you have
> an RPC service that has
> > >> thousands of nodes, there are thousands of
> clients connecting to
> > that
> > >> RPC
> > >> service. How to reduce the connections and
> how to effectively load
> > >> balance
> > >> requests across thousands of nodes are the
> problems that a
> > client-side
> > >> loading technology needs to solve. If you
> think about "partition"
> > as
> > >> "node"
> > >> and "partitioned producer" as "RPC client",
> the problem is exactly
> > the
> > >> same. Finagle (the Twitter RPC framework)
> has implemented a lot of
> > >> client-side
> > >> load-balancing algorithms
> > >> <
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.github.io%2Ffinagle%2Fguide%2FClients.html%23load-balancing&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6Kf70x0qgzsLC7Kzp7JtDF5TW9uS1WnU8FQW6OdtDHI%3D&reserved=0
> >
> > >> and
> > >> there are some great articles that you can
> reference
> > >> <
> > >>
> >
> https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer.html
> > >> .
> > >> I agree with the direction of introducing a
> mechanism to reduce the
> > >> number
> > >> of producers in a partitioned topic
> producer. However, I have a
> > concern
> > >> about introducing `.numPartitionsLimit(10)`
> directly to the
> > producer
> > >> builder. It limits the possibility to
> implement different
> > algorithms on
> > >> selecting partitions.
> > >> So instead of directly implementing the
> logic within the
> > partitioned
> > >> topic
> > >> producer, I think the proposal can be broken
> into two parts:
> > >> 1) Introduce some kind of lazy-loading
> mechanism in the partitioned
> > >> producer to initialize the producers for
> partitions lazily. I.e.,
> > only
> > >> initialize a producer when the message
> router selects a partition.
> > >> 2) Implement a message router that only
> selects one or N
> > partitions.
> > >> In this way, the partitioned producer is
> only responsible for
> > managing
> > >> a
> > >> collection of producers, and the message
> router is responsible for
> > >> selecting the partitions. This allows people
> to be able to
> > implement
> > >> different message routers. We can even adopt
> the client-side load
> > >> balancing
> > >> algorithms from Finagle.
> > >> Thanks,
> > >> Sijie
> > >> On Wed, Jan 27, 2021 at 7:18 PM Yuri
> Mizushima <
> > yumizush@yahoo-corp.jp
> > >> wrote:
> > >>> I notice that PIP-78 has already assigned to
> another issue.
> > >>
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fpulsar-dev%2F202101.mbox%2F%253CCAG%253DTQOrPH49v9ToDE_aeQzEiDC%252BEgSR61ERoqanpWfQGvEB_Vw%2540mail.gmail.com%253E&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dOyeyrkpxoQLJOEiWrXfqymuVP7eMa0oyeJLeUkdikU%3D&reserved=0
> > >>> So, I'll change the PIP number to 79.
> > >>
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=moRiB9I9Zwz%2FFcDAWxza1%2BV6Xcl1xf1pD524%2BMZB2Lk%3D&reserved=0
> > >>> Regards,
> > >>> --
> > >>> Yuri Mizushima
> > >>> yumizush@yahoo-corp.jp
> > >>> "Yuri Mizushima" <yu...@yahoo-corp.jp>
> wrote:
> > >>> Dear Pulsar community,
> > >>> When partitioned producer connects to
> partitioned topic,
> > >>> sometimes doesn't need to connect to all of
> partitions depending
> > >> on
> > >>> rate, routing mode, etc.
> > >>> So, I drafted a PIP about reducing
> redundant producers from
> > >>> partitioned producer.
> > >>> I'd like to use system resources (e.g.
> connections between
> > >> Client and
> > >>> Broker, memory usage of both Client and
> Broker)
> > >>> more efficiently by this feature.
> > >>
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-78%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C7e19573ee36d4edf3d6c08d9b93df247%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637744498709598895%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=oreRJxiW36BXaSjcJkdhpu1L%2FXpKxkhJfOtT7TTpssQ%3D&reserved=0
> > >>> Feel free to ask me any questions or
> suggestions, etc.
> > >>> Best regards,
> > >>> --
> > >>> Yuri Mizushima
> > >>> yumizush@yahoo-corp.jp
> >
> >
>
>
>
>
>
Re: [PIP-78] Reduce redundant producers from partitioned producer
Posted by Yuri Mizushima <yu...@yahoo-corp.jp>.
Enrico,
Thank you for your clarification.
> I would add a configuration parameter to allow the client to use this
> feature.
> So:
> - feature enabled -> everything works like in your patch
> - feature disabled -> the broker ignores *partial_producer_supported *and
> runs as before
Okay. That makes sense to me.
If you have no more concerns, I'll fix the change to address your comments.
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
On 2021/12/14 19:01, "Enrico Olivelli" <eo...@gmail.com> wrote:
sorry for the late reply, I missed this message.
Il giorno mar 7 dic 2021 alle ore 05:56 Yuri Mizushima <
yumizush@yahoo-corp.jp> ha scritto:
> Enrico,
> Thank you for your comment.
>
> > IIUC with this change the client will control which metrics are reported
> by
> > the broker ?
>
> From the protocol perspective, yes.
> However, the main point of this change is not to "control" metrics by the
> client side,
> but to make the broker aggregate partitioned topic's producer metrics
> explicitly.
>
> Do you suggest adding a broker config that
> configures whether partitioned producer stats are aggregated by
> producerName instead of
> introducing a backward compatibility key (i.e.,
> partial_producer_supported) on the client-side?
>
Not exactly.
I would add a configuration parameter to allow the client to use this
feature.
So:
- feature enabled -> everything works like in your patch
- feature disabled -> the broker ignores *partial_producer_supported *and
runs as before
This way if you want to let the client aggregate the stats you can do it
but by default clients are not able to interact with the metrics format.
In multi-tenancy environments, like in large clusters shared by many teams,
you do not want that the client is able to change the way the broker
reports its metrics
or in general that the client is able to alter the monitoring systems.
Enrico
>
> It is simple. However, I think we can't enable the config until all
> clients are updated.
>
> What do you think?
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> On 2021/12/02 17:37, "Enrico Olivelli" <eo...@gmail.com> wrote:
>
> Yuri,
> IIUC with this change the client will control which metrics are
> reported by
> the broker ?
>
> I am not sure it is a good idea, because metrics are usually managed
> by the
> owners of the brokers, who sometimes are not the same who run the
> clients.
>
> Also, I am not sure if this way it is possible for the client to force
> the
> Broker to create many metrics and create some kind of damage.
>
> Would it be better to add a Broker configuration flag to turn on this
> feature ? I mean to allow the client to select the type of metrics ?
>
>
> Enrico
>
>
> Il giorno gio 2 dic 2021 alle ore 03:00 Yuri Mizushima <
> yumizush@yahoo-corp.jp> ha scritto:
>
> > Do you have any comments?
> > If there are no comments by Dec. 7, I will close the discussion and
> rebase
> > the PR commit to current master.
> >
> > Regards,
> >
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > On 2021/11/16 15:46, "Yuri Mizushima" <yu...@yahoo-corp.jp>
> wrote:
> >
> > Dear Pulsar community,
> >
> > I have created a new PR
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=POhcguG9KtlOMqyrdZtiLD9KKBGYDBfbDiq8g88cKAo%3D&reserved=0
> > for stats aggregation,
> > but I didn't discuss about the wire protocol change. I hope we
> will
> > discuss it here.
> >
> > Currently, partitioned producer can't aggregate by any key such
> as
> > cnx, producerId, producerName, and so on.
> > I think we need to add any aggregation system.
> > Therefore, added new aggregation policy as producerName (with
> client
> > side implementation).
> >
> > New protocol field partial_producer_supported is not used for
> stats
> > aggregation. It is used for backward compatibility.
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401%2Ffiles%23diff-f29399fed32e0916cf28452ba71078a3ae5ed77bbaef9f52a925165d8ee66cfdR489&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=o7%2FrLHQCzDUD2ggjSZqBm5eG2RxJkI5oH95NGU59l2E%3D&reserved=0
> >
> > In my understanding, if introduce new stats aggregation key to
> client
> > side,
> > need a way to determine whether the feature is enabled at client
> side.
> > For example, whether the producer has specific field or metadata,
> > the version (e.g. protocol version) is greater than threshold,
> etc.
> >
> > Of course, if we can introduce aggregation feature without
> adding any
> > new key or implementations from client side,
> > we can support the feature not only new client but also old one.
> >
> > I'm looking forward to your opinions or suggestions to this PR.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > On 2021/05/11 14:26, "Yuri Mizushima" <yu...@yahoo-corp.jp>
> wrote:
> >
> >
> > Dear Pulsar Community,
> >
> > > I will submit the next PR about PartitionedTopicStats later.
> > I submitted the next PR for this PIP. If you have any
> suggestions,
> > please comment to this PR.
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10534&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Wcv8KeSv8gxceH10C%2B8LMZZdWbgK3FK5bAB%2BastT9To%3D&reserved=0
> >
> > Regards,
> >
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
> >
> > Dear Pulsar Community,
> >
> > I submitted the PR for this PIP.
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10279&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5cSZL6zu4F1ISWBKHfbCIgKCLCQkSVbOdZZnhEHu7Ng%3D&reserved=0
> >
> > This is a part of implementations.
> > I will submit the next PR about PartitionedTopicStats later.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
> >
> > Sijie,
> >
> > After sending previous mail, I watched meeting recording
> and
> > understand about authn/authz issue.
> > Therefore, I updated the PIP document.
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=t7FX1wGEb8g7Ev0FhP3x2E8Dx23AZIQ0PVUXNCu6pX8%3D&reserved=0
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
> >
> > Sijie,
> >
> > > If the lazy-loading approach sounds attractive to
> you
> > and you like it,
> > > maybe the next step is to update the PIP, what do
> you
> > think?
> >
> > I think so too. I will update the PIP after
> discussing the
> > authn/authz issue.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Sijie Guo" <gu...@gmail.com> wrote:
> >
> > Hi Yuri,
> >
> > Regarding the authn/authz issue, @Matteo Merli <
> > mmerli@apache.org> can
> > probably chime in more about that part.
> >
> > If the lazy-loading approach sounds attractive
> to you
> > and you like it,
> > maybe the next step is to update the PIP, what
> do you
> > think?
> >
> > - Sijie
> >
> > On Mon, Feb 8, 2021 at 6:57 PM Yuri Mizushima <
> > yumizush@yahoo-corp.jp>
> > wrote:
> >
> > > Michael,
> > >
> > > Thank you for your comment!
> > >
> > > > Which Pulsar Clients will benefit from this
> > proposal?
> > > I think that this proposal will be useful to
> any
> > clients.
> > > In my schedule, if this proposal is accepted
> then I
> > will implement this
> > > feature to Java client.
> > > If needed, then implement same feature to other
> > clients such as C++, Go,
> > > etc.
> > >
> > > Regards,
> > > --
> > > Yuri Mizushima
> > > yumizush@yahoo-corp.jp
> > >
> > >
> > > "Michael Marshall" <mi...@gmail.com>
> wrote:
> > >
> > > Hi Yuri and Sijie,
> > >
> > > I definitely like the idea of lazily
> creating
> > producers as well as
> > > introducing a way to provide custom routing
> logic.
> > >
> > > Which Pulsar Clients will benefit from this
> > proposal? I’d love to see
> > > this feature in the go client.
> > >
> > > Thanks,
> > > Michael Marshall
> > >
> > > > On Feb 7, 2021, at 9:53 PM, Yuri
> Mizushima <
> > yumizush@yahoo-corp.jp>
> > > wrote:
> > > >
> > > > Sijie,
> > > >
> > > > Thank you for sharing!
> > > >
> > > > First, I considered your suggestion.
> > > > I think these implementations sound good.
> > > >
> > > > I think we should consider the State of
> > partitioned producer: Ready,
> > > Connecting, etc.
> > > > Currently, partitioned producer gets
> "Ready"
> > only when all producers
> > > connect to Broker correctly.
> > > >
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FPartitionedProducerImpl.java%23L146&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=MqEoouv5zR4wPHtu8BSNP5Qn%2F5gBwcvHGt0qCYVlrHA%3D&reserved=0
> > > > It seems that we should change meaning of
> > state (or change handling)
> > > if we introduce the lazy-load feature.
> > > > To guarantee the message ordering (e.g.
> using
> > partitionKey),
> > > partitioned producer should stop (or don't send
> > messages to be routed to
> > > unavailable partition) when producer can't
> connect
> > to one of partition.
> > > >
> > > > Secondly, I considered Matteo's comments.
> > > > I couldn't understand well about issue of
> > authn/authz. Please tell
> > > me more detail.
> > > >
> > > > I wrote "connection" as number of
> producers
> > which connect to broker.
> > > Also, TCP connections between partitioned
> producer
> > and broker will be less
> > > than or equal to current in some cases. I'll
> show a
> > case below.
> > > >
> > > > Suppose
> > > > * cluster has Broker0, 1, 2
> > > > * partitioned topic has 5 partitions
> > > > * limit conf is 3 partitions
> > > > * loadbalance partitions as below
> > > > - Broker0: partition-0, partition-1
> > > > - Broker1: partition-2
> > > > - Broker2: partition-3, partition-4
> > > >
> > > > Currently, client will create 3
> connections
> > (Broker0, 1, 2). If
> > > client uses limit conf and elects partitions
> such as
> > [0, 1, 2], then client
> > > will create 2 connections (Broker0, 1). Of
> course,
> > if client elects
> > > partitions such as [0, 2, 3], then client will
> still
> > create 3 connections.
> > > >
> > > > I'd like to decrease number of
> producers. I
> > think that resources of
> > > broker will be improved slightly by this
> feature
> > because broker has list of
> > > producers by some classes such as ServerCnx,
> > AbstractTopic.
> > > >
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FServerCnx.java%23L1096-L1097&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=XArfKWYI5RBBA6O0F91nJQK6FA%2FSXDJsCaMa0uRFVLM%3D&reserved=0
> > > >
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FAbstractTopic.java%23L577&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DnjdSwsW%2FpoDZNjAjdc2Cyul4XNAfYUxOxeYUFetVOU%3D&reserved=0
> > > >
> > > > In my case, unspecified number of
> producers
> > will connect to the same
> > > partitioned topic with different rate. We need
> to
> > set the number of
> > > partitions according to the high-rate producer.
> > > > However, on the other hand, this number
> is
> > excessively large for
> > > low-rate producers.
> > > > I want to reduce such redundant
> producers for
> > resource efficiency.
> > > >
> > > > Regards,
> > > > --
> > > > Yuri Mizushima
> > > > yumizush@yahoo-corp.jp
> > > >
> > > >
> > > > "Sijie Guo" <gu...@gmail.com> wrote:
> > > >
> > > > Hi Yuri,
> > > >
> > > > In today's community meeting, Matteo
> shared
> > some of his thoughts
> > > about this
> > > > PIP.
> > > >
> > > > You can find some meeting notes here:
> > > >
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE%2Fedit%23bookmark%3Did.rezbt4xmjxpz&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Li5V1eYInUctd3XCE5GhGs0O%2F12skmhLQCzIq5PXosw%3D&reserved=0
> > > >
> > > > Matteo can also chime in as well.
> > > >
> > > > - Sijie
> > > >
> > > >> On Sun, Jan 31, 2021 at 7:21 PM Yuri
> > Mizushima <
> > > yumizush@yahoo-corp.jp>
> > > >> wrote:
> > > >> Sijie,
> > > >> Thank you for your reply!
> > > >> I'll check it.
> > > >> Regards,
> > > >> --
> > > >> Yuri Mizushima
> > > >> yumizush@yahoo-corp.jp
> > > >> "Sijie Guo" <gu...@gmail.com> wrote:
> > > >> Yuri,
> > > >> Thank you for bringing this up! This
> is a
> > super helpful proposal!
> > > >> The problem is very similar to what an
> RPC
> > framework (like Finagle)
> > > >> with
> > > >> client-side load balancing has.
> > > >> An RPC framework with a client-side
> > load-balancing mechanism needs
> > > to
> > > >> send
> > > >> requests across multiple nodes. If you
> have
> > an RPC service that has
> > > >> thousands of nodes, there are
> thousands of
> > clients connecting to
> > > that
> > > >> RPC
> > > >> service. How to reduce the connections
> and
> > how to effectively load
> > > >> balance
> > > >> requests across thousands of nodes are
> the
> > problems that a
> > > client-side
> > > >> loading technology needs to solve. If
> you
> > think about "partition"
> > > as
> > > >> "node"
> > > >> and "partitioned producer" as "RPC
> client",
> > the problem is exactly
> > > the
> > > >> same. Finagle (the Twitter RPC
> framework)
> > has implemented a lot of
> > > >> client-side
> > > >> load-balancing algorithms
> > > >> <
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.github.io%2Ffinagle%2Fguide%2FClients.html%23load-balancing&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=J6FM2AeFlnAdlsKjivTP0i2y9wy6ZLhPXuZeTapYD7s%3D&reserved=0
> > >
> > > >> and
> > > >> there are some great articles that you
> can
> > reference
> > > >> <
> > > >>
> > >
> >
> https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer.html
> > > >> .
> > > >> I agree with the direction of
> introducing a
> > mechanism to reduce the
> > > >> number
> > > >> of producers in a partitioned topic
> > producer. However, I have a
> > > concern
> > > >> about introducing
> `.numPartitionsLimit(10)`
> > directly to the
> > > producer
> > > >> builder. It limits the possibility to
> > implement different
> > > algorithms on
> > > >> selecting partitions.
> > > >> So instead of directly implementing the
> > logic within the
> > > partitioned
> > > >> topic
> > > >> producer, I think the proposal can be
> broken
> > into two parts:
> > > >> 1) Introduce some kind of lazy-loading
> > mechanism in the partitioned
> > > >> producer to initialize the producers
> for
> > partitions lazily. I.e.,
> > > only
> > > >> initialize a producer when the message
> > router selects a partition.
> > > >> 2) Implement a message router that only
> > selects one or N
> > > partitions.
> > > >> In this way, the partitioned producer
> is
> > only responsible for
> > > managing
> > > >> a
> > > >> collection of producers, and the
> message
> > router is responsible for
> > > >> selecting the partitions. This allows
> people
> > to be able to
> > > implement
> > > >> different message routers. We can even
> adopt
> > the client-side load
> > > >> balancing
> > > >> algorithms from Finagle.
> > > >> Thanks,
> > > >> Sijie
> > > >> On Wed, Jan 27, 2021 at 7:18 PM Yuri
> > Mizushima <
> > > yumizush@yahoo-corp.jp
> > > >> wrote:
> > > >>> I notice that PIP-78 has already
> assigned to
> > another issue.
> > > >>
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fpulsar-dev%2F202101.mbox%2F%253CCAG%253DTQOrPH49v9ToDE_aeQzEiDC%252BEgSR61ERoqanpWfQGvEB_Vw%2540mail.gmail.com%253E&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=duJLq8ne%2BpzggrqEhMtAlnDU%2BYcQ9ZmdCWTNaobbpFk%3D&reserved=0
> > > >>> So, I'll change the PIP number to 79.
> > > >>
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=t7FX1wGEb8g7Ev0FhP3x2E8Dx23AZIQ0PVUXNCu6pX8%3D&reserved=0
> > > >>> Regards,
> > > >>> --
> > > >>> Yuri Mizushima
> > > >>> yumizush@yahoo-corp.jp
> > > >>> "Yuri Mizushima" <
> yumizush@yahoo-corp.jp>
> > wrote:
> > > >>> Dear Pulsar community,
> > > >>> When partitioned producer connects to
> > partitioned topic,
> > > >>> sometimes doesn't need to connect to
> all of
> > partitions depending
> > > >> on
> > > >>> rate, routing mode, etc.
> > > >>> So, I drafted a PIP about reducing
> > redundant producers from
> > > >>> partitioned producer.
> > > >>> I'd like to use system resources (e.g.
> > connections between
> > > >> Client and
> > > >>> Broker, memory usage of both Client and
> > Broker)
> > > >>> more efficiently by this feature.
> > > >>
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-78%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7C14ab44acc2be4e8cd90e08d9bee8bf9b%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750729124731851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1FA7%2BdhD7bEySjrDlVgMTUKYskqgmXwWhAIoYDxjmR4%3D&reserved=0
> > > >>> Feel free to ask me any questions or
> > suggestions, etc.
> > > >>> Best regards,
> > > >>> --
> > > >>> Yuri Mizushima
> > > >>> yumizush@yahoo-corp.jp
> > >
> > >
> >
> >
> >
> >
> >
>
>
Re: [PIP-78] Reduce redundant producers from partitioned producer
Posted by Enrico Olivelli <eo...@gmail.com>.
sorry for the late reply, I missed this message.
Il giorno mar 7 dic 2021 alle ore 05:56 Yuri Mizushima <
yumizush@yahoo-corp.jp> ha scritto:
> Enrico,
> Thank you for your comment.
>
> > IIUC with this change the client will control which metrics are reported
> by
> > the broker ?
>
> From the protocol perspective, yes.
> However, the main point of this change is not to "control" metrics by the
> client side,
> but to make the broker aggregate partitioned topic's producer metrics
> explicitly.
>
> Do you suggest adding a broker config that
> configures whether partitioned producer stats are aggregated by
> producerName instead of
> introducing a backward compatibility key (i.e.,
> partial_producer_supported) on the client-side?
>
Not exactly.
I would add a configuration parameter to allow the client to use this
feature.
So:
- feature enabled -> everything works like in your patch
- feature disabled -> the broker ignores *partial_producer_supported *and
runs as before
This way if you want to let the client aggregate the stats you can do it
but by default clients are not able to interact with the metrics format.
In multi-tenancy environments, like in large clusters shared by many teams,
you do not want that the client is able to change the way the broker
reports its metrics
or in general that the client is able to alter the monitoring systems.
Enrico
>
> It is simple. However, I think we can't enable the config until all
> clients are updated.
>
> What do you think?
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> On 2021/12/02 17:37, "Enrico Olivelli" <eo...@gmail.com> wrote:
>
> Yuri,
> IIUC with this change the client will control which metrics are
> reported by
> the broker ?
>
> I am not sure it is a good idea, because metrics are usually managed
> by the
> owners of the brokers, who sometimes are not the same who run the
> clients.
>
> Also, I am not sure if this way it is possible for the client to force
> the
> Broker to create many metrics and create some kind of damage.
>
> Would it be better to add a Broker configuration flag to turn on this
> feature ? I mean to allow the client to select the type of metrics ?
>
>
> Enrico
>
>
> Il giorno gio 2 dic 2021 alle ore 03:00 Yuri Mizushima <
> yumizush@yahoo-corp.jp> ha scritto:
>
> > Do you have any comments?
> > If there are no comments by Dec. 7, I will close the discussion and
> rebase
> > the PR commit to current master.
> >
> > Regards,
> >
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > On 2021/11/16 15:46, "Yuri Mizushima" <yu...@yahoo-corp.jp>
> wrote:
> >
> > Dear Pulsar community,
> >
> > I have created a new PR
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655072887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=JVOJsF5wTRoAyvy%2F8kyddBJP1XsdBv3YcIt2FP4OkHI%3D&reserved=0
> > for stats aggregation,
> > but I didn't discuss about the wire protocol change. I hope we
> will
> > discuss it here.
> >
> > Currently, partitioned producer can't aggregate by any key such
> as
> > cnx, producerId, producerName, and so on.
> > I think we need to add any aggregation system.
> > Therefore, added new aggregation policy as producerName (with
> client
> > side implementation).
> >
> > New protocol field partial_producer_supported is not used for
> stats
> > aggregation. It is used for backward compatibility.
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401%2Ffiles%23diff-f29399fed32e0916cf28452ba71078a3ae5ed77bbaef9f52a925165d8ee66cfdR489&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655072887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=L%2B4JftwNnN7dUtck8wjWhwvqKhyFpyUcOcgsoinwyfQ%3D&reserved=0
> >
> > In my understanding, if introduce new stats aggregation key to
> client
> > side,
> > need a way to determine whether the feature is enabled at client
> side.
> > For example, whether the producer has specific field or metadata,
> > the version (e.g. protocol version) is greater than threshold,
> etc.
> >
> > Of course, if we can introduce aggregation feature without
> adding any
> > new key or implementations from client side,
> > we can support the feature not only new client but also old one.
> >
> > I'm looking forward to your opinions or suggestions to this PR.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > On 2021/05/11 14:26, "Yuri Mizushima" <yu...@yahoo-corp.jp>
> wrote:
> >
> >
> > Dear Pulsar Community,
> >
> > > I will submit the next PR about PartitionedTopicStats later.
> > I submitted the next PR for this PIP. If you have any
> suggestions,
> > please comment to this PR.
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10534&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655072887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8nKy4sWjl7mdvSgAgDRUwRWG7Czwyk%2F5oOA8zszlvRc%3D&reserved=0
> >
> > Regards,
> >
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
> >
> > Dear Pulsar Community,
> >
> > I submitted the PR for this PIP.
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10279&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RhxeAyS5PW0it0ilItjmfJZj4%2BmZAcEfqF86IbBgk9A%3D&reserved=0
> >
> > This is a part of implementations.
> > I will submit the next PR about PartitionedTopicStats later.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
> >
> > Sijie,
> >
> > After sending previous mail, I watched meeting recording
> and
> > understand about authn/authz issue.
> > Therefore, I updated the PIP document.
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DqqHO48yEXJV8Ci%2B7RJBVcZOgGLgXeNEuBMbNLUTtEw%3D&reserved=0
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
> >
> > Sijie,
> >
> > > If the lazy-loading approach sounds attractive to
> you
> > and you like it,
> > > maybe the next step is to update the PIP, what do
> you
> > think?
> >
> > I think so too. I will update the PIP after
> discussing the
> > authn/authz issue.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Sijie Guo" <gu...@gmail.com> wrote:
> >
> > Hi Yuri,
> >
> > Regarding the authn/authz issue, @Matteo Merli <
> > mmerli@apache.org> can
> > probably chime in more about that part.
> >
> > If the lazy-loading approach sounds attractive
> to you
> > and you like it,
> > maybe the next step is to update the PIP, what
> do you
> > think?
> >
> > - Sijie
> >
> > On Mon, Feb 8, 2021 at 6:57 PM Yuri Mizushima <
> > yumizush@yahoo-corp.jp>
> > wrote:
> >
> > > Michael,
> > >
> > > Thank you for your comment!
> > >
> > > > Which Pulsar Clients will benefit from this
> > proposal?
> > > I think that this proposal will be useful to
> any
> > clients.
> > > In my schedule, if this proposal is accepted
> then I
> > will implement this
> > > feature to Java client.
> > > If needed, then implement same feature to other
> > clients such as C++, Go,
> > > etc.
> > >
> > > Regards,
> > > --
> > > Yuri Mizushima
> > > yumizush@yahoo-corp.jp
> > >
> > >
> > > "Michael Marshall" <mi...@gmail.com>
> wrote:
> > >
> > > Hi Yuri and Sijie,
> > >
> > > I definitely like the idea of lazily
> creating
> > producers as well as
> > > introducing a way to provide custom routing
> logic.
> > >
> > > Which Pulsar Clients will benefit from this
> > proposal? I’d love to see
> > > this feature in the go client.
> > >
> > > Thanks,
> > > Michael Marshall
> > >
> > > > On Feb 7, 2021, at 9:53 PM, Yuri
> Mizushima <
> > yumizush@yahoo-corp.jp>
> > > wrote:
> > > >
> > > > Sijie,
> > > >
> > > > Thank you for sharing!
> > > >
> > > > First, I considered your suggestion.
> > > > I think these implementations sound good.
> > > >
> > > > I think we should consider the State of
> > partitioned producer: Ready,
> > > Connecting, etc.
> > > > Currently, partitioned producer gets
> "Ready"
> > only when all producers
> > > connect to Broker correctly.
> > > >
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FPartitionedProducerImpl.java%23L146&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Sq7Qw3nmVq725Bwy4rUdreqvhcbAn7qHXN5vEELqqNY%3D&reserved=0
> > > > It seems that we should change meaning of
> > state (or change handling)
> > > if we introduce the lazy-load feature.
> > > > To guarantee the message ordering (e.g.
> using
> > partitionKey),
> > > partitioned producer should stop (or don't send
> > messages to be routed to
> > > unavailable partition) when producer can't
> connect
> > to one of partition.
> > > >
> > > > Secondly, I considered Matteo's comments.
> > > > I couldn't understand well about issue of
> > authn/authz. Please tell
> > > me more detail.
> > > >
> > > > I wrote "connection" as number of
> producers
> > which connect to broker.
> > > Also, TCP connections between partitioned
> producer
> > and broker will be less
> > > than or equal to current in some cases. I'll
> show a
> > case below.
> > > >
> > > > Suppose
> > > > * cluster has Broker0, 1, 2
> > > > * partitioned topic has 5 partitions
> > > > * limit conf is 3 partitions
> > > > * loadbalance partitions as below
> > > > - Broker0: partition-0, partition-1
> > > > - Broker1: partition-2
> > > > - Broker2: partition-3, partition-4
> > > >
> > > > Currently, client will create 3
> connections
> > (Broker0, 1, 2). If
> > > client uses limit conf and elects partitions
> such as
> > [0, 1, 2], then client
> > > will create 2 connections (Broker0, 1). Of
> course,
> > if client elects
> > > partitions such as [0, 2, 3], then client will
> still
> > create 3 connections.
> > > >
> > > > I'd like to decrease number of
> producers. I
> > think that resources of
> > > broker will be improved slightly by this
> feature
> > because broker has list of
> > > producers by some classes such as ServerCnx,
> > AbstractTopic.
> > > >
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FServerCnx.java%23L1096-L1097&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2F0vpCDidkG38A4YyqhDZ2opj71OxPXHLFKfitPeMXto%3D&reserved=0
> > > >
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FAbstractTopic.java%23L577&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rKpJuX6im00enGDM0HVAYPw%2FjBgsotLcokaIVknVwaE%3D&reserved=0
> > > >
> > > > In my case, unspecified number of
> producers
> > will connect to the same
> > > partitioned topic with different rate. We need
> to
> > set the number of
> > > partitions according to the high-rate producer.
> > > > However, on the other hand, this number
> is
> > excessively large for
> > > low-rate producers.
> > > > I want to reduce such redundant
> producers for
> > resource efficiency.
> > > >
> > > > Regards,
> > > > --
> > > > Yuri Mizushima
> > > > yumizush@yahoo-corp.jp
> > > >
> > > >
> > > > "Sijie Guo" <gu...@gmail.com> wrote:
> > > >
> > > > Hi Yuri,
> > > >
> > > > In today's community meeting, Matteo
> shared
> > some of his thoughts
> > > about this
> > > > PIP.
> > > >
> > > > You can find some meeting notes here:
> > > >
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE%2Fedit%23bookmark%3Did.rezbt4xmjxpz&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Rst44OBvITL%2BbKigNhLVE6z428Laca6UmqDbG2V2IRs%3D&reserved=0
> > > >
> > > > Matteo can also chime in as well.
> > > >
> > > > - Sijie
> > > >
> > > >> On Sun, Jan 31, 2021 at 7:21 PM Yuri
> > Mizushima <
> > > yumizush@yahoo-corp.jp>
> > > >> wrote:
> > > >> Sijie,
> > > >> Thank you for your reply!
> > > >> I'll check it.
> > > >> Regards,
> > > >> --
> > > >> Yuri Mizushima
> > > >> yumizush@yahoo-corp.jp
> > > >> "Sijie Guo" <gu...@gmail.com> wrote:
> > > >> Yuri,
> > > >> Thank you for bringing this up! This
> is a
> > super helpful proposal!
> > > >> The problem is very similar to what an
> RPC
> > framework (like Finagle)
> > > >> with
> > > >> client-side load balancing has.
> > > >> An RPC framework with a client-side
> > load-balancing mechanism needs
> > > to
> > > >> send
> > > >> requests across multiple nodes. If you
> have
> > an RPC service that has
> > > >> thousands of nodes, there are
> thousands of
> > clients connecting to
> > > that
> > > >> RPC
> > > >> service. How to reduce the connections
> and
> > how to effectively load
> > > >> balance
> > > >> requests across thousands of nodes are
> the
> > problems that a
> > > client-side
> > > >> loading technology needs to solve. If
> you
> > think about "partition"
> > > as
> > > >> "node"
> > > >> and "partitioned producer" as "RPC
> client",
> > the problem is exactly
> > > the
> > > >> same. Finagle (the Twitter RPC
> framework)
> > has implemented a lot of
> > > >> client-side
> > > >> load-balancing algorithms
> > > >> <
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.github.io%2Ffinagle%2Fguide%2FClients.html%23load-balancing&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0qZmzlNFhR17Yl3ucmA57TTZEXN%2Bswrtj1Ib95P2TtU%3D&reserved=0
> > >
> > > >> and
> > > >> there are some great articles that you
> can
> > reference
> > > >> <
> > > >>
> > >
> >
> https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer.html
> > > >> .
> > > >> I agree with the direction of
> introducing a
> > mechanism to reduce the
> > > >> number
> > > >> of producers in a partitioned topic
> > producer. However, I have a
> > > concern
> > > >> about introducing
> `.numPartitionsLimit(10)`
> > directly to the
> > > producer
> > > >> builder. It limits the possibility to
> > implement different
> > > algorithms on
> > > >> selecting partitions.
> > > >> So instead of directly implementing the
> > logic within the
> > > partitioned
> > > >> topic
> > > >> producer, I think the proposal can be
> broken
> > into two parts:
> > > >> 1) Introduce some kind of lazy-loading
> > mechanism in the partitioned
> > > >> producer to initialize the producers
> for
> > partitions lazily. I.e.,
> > > only
> > > >> initialize a producer when the message
> > router selects a partition.
> > > >> 2) Implement a message router that only
> > selects one or N
> > > partitions.
> > > >> In this way, the partitioned producer
> is
> > only responsible for
> > > managing
> > > >> a
> > > >> collection of producers, and the
> message
> > router is responsible for
> > > >> selecting the partitions. This allows
> people
> > to be able to
> > > implement
> > > >> different message routers. We can even
> adopt
> > the client-side load
> > > >> balancing
> > > >> algorithms from Finagle.
> > > >> Thanks,
> > > >> Sijie
> > > >> On Wed, Jan 27, 2021 at 7:18 PM Yuri
> > Mizushima <
> > > yumizush@yahoo-corp.jp
> > > >> wrote:
> > > >>> I notice that PIP-78 has already
> assigned to
> > another issue.
> > > >>
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fpulsar-dev%2F202101.mbox%2F%253CCAG%253DTQOrPH49v9ToDE_aeQzEiDC%252BEgSR61ERoqanpWfQGvEB_Vw%2540mail.gmail.com%253E&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6pYP5laeruIaSJ7ASdp3hhRSIqUviMvUduq485hIUP0%3D&reserved=0
> > > >>> So, I'll change the PIP number to 79.
> > > >>
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DqqHO48yEXJV8Ci%2B7RJBVcZOgGLgXeNEuBMbNLUTtEw%3D&reserved=0
> > > >>> Regards,
> > > >>> --
> > > >>> Yuri Mizushima
> > > >>> yumizush@yahoo-corp.jp
> > > >>> "Yuri Mizushima" <
> yumizush@yahoo-corp.jp>
> > wrote:
> > > >>> Dear Pulsar community,
> > > >>> When partitioned producer connects to
> > partitioned topic,
> > > >>> sometimes doesn't need to connect to
> all of
> > partitions depending
> > > >> on
> > > >>> rate, routing mode, etc.
> > > >>> So, I drafted a PIP about reducing
> > redundant producers from
> > > >>> partitioned producer.
> > > >>> I'd like to use system resources (e.g.
> > connections between
> > > >> Client and
> > > >>> Broker, memory usage of both Client and
> > Broker)
> > > >>> more efficiently by this feature.
> > > >>
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-78%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=s%2BmbH9WWT3qihfOPc%2BToh%2BkWDNAiqEUv%2F%2B19oUKpeOY%3D&reserved=0
> > > >>> Feel free to ask me any questions or
> > suggestions, etc.
> > > >>> Best regards,
> > > >>> --
> > > >>> Yuri Mizushima
> > > >>> yumizush@yahoo-corp.jp
> > >
> > >
> >
> >
> >
> >
> >
>
>
Re: [PIP-78] Reduce redundant producers from partitioned producer
Posted by Yuri Mizushima <yu...@yahoo-corp.jp>.
Enrico,
Thank you for your comment.
> IIUC with this change the client will control which metrics are reported by
> the broker ?
From the protocol perspective, yes.
However, the main point of this change is not to "control" metrics by the client side,
but to make the broker aggregate partitioned topic's producer metrics explicitly.
Do you suggest adding a broker config that
configures whether partitioned producer stats are aggregated by producerName instead of
introducing a backward compatibility key (i.e., partial_producer_supported) on the client-side?
It is simple. However, I think we can't enable the config until all clients are updated.
What do you think?
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
On 2021/12/02 17:37, "Enrico Olivelli" <eo...@gmail.com> wrote:
Yuri,
IIUC with this change the client will control which metrics are reported by
the broker ?
I am not sure it is a good idea, because metrics are usually managed by the
owners of the brokers, who sometimes are not the same who run the clients.
Also, I am not sure if this way it is possible for the client to force the
Broker to create many metrics and create some kind of damage.
Would it be better to add a Broker configuration flag to turn on this
feature ? I mean to allow the client to select the type of metrics ?
Enrico
Il giorno gio 2 dic 2021 alle ore 03:00 Yuri Mizushima <
yumizush@yahoo-corp.jp> ha scritto:
> Do you have any comments?
> If there are no comments by Dec. 7, I will close the discussion and rebase
> the PR commit to current master.
>
> Regards,
>
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> On 2021/11/16 15:46, "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Dear Pulsar community,
>
> I have created a new PR
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655072887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=JVOJsF5wTRoAyvy%2F8kyddBJP1XsdBv3YcIt2FP4OkHI%3D&reserved=0
> for stats aggregation,
> but I didn't discuss about the wire protocol change. I hope we will
> discuss it here.
>
> Currently, partitioned producer can't aggregate by any key such as
> cnx, producerId, producerName, and so on.
> I think we need to add any aggregation system.
> Therefore, added new aggregation policy as producerName (with client
> side implementation).
>
> New protocol field partial_producer_supported is not used for stats
> aggregation. It is used for backward compatibility.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401%2Ffiles%23diff-f29399fed32e0916cf28452ba71078a3ae5ed77bbaef9f52a925165d8ee66cfdR489&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655072887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=L%2B4JftwNnN7dUtck8wjWhwvqKhyFpyUcOcgsoinwyfQ%3D&reserved=0
>
> In my understanding, if introduce new stats aggregation key to client
> side,
> need a way to determine whether the feature is enabled at client side.
> For example, whether the producer has specific field or metadata,
> the version (e.g. protocol version) is greater than threshold, etc.
>
> Of course, if we can introduce aggregation feature without adding any
> new key or implementations from client side,
> we can support the feature not only new client but also old one.
>
> I'm looking forward to your opinions or suggestions to this PR.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> On 2021/05/11 14:26, "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
>
> Dear Pulsar Community,
>
> > I will submit the next PR about PartitionedTopicStats later.
> I submitted the next PR for this PIP. If you have any suggestions,
> please comment to this PR.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10534&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655072887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8nKy4sWjl7mdvSgAgDRUwRWG7Czwyk%2F5oOA8zszlvRc%3D&reserved=0
>
> Regards,
>
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Dear Pulsar Community,
>
> I submitted the PR for this PIP.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10279&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RhxeAyS5PW0it0ilItjmfJZj4%2BmZAcEfqF86IbBgk9A%3D&reserved=0
>
> This is a part of implementations.
> I will submit the next PR about PartitionedTopicStats later.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Sijie,
>
> After sending previous mail, I watched meeting recording and
> understand about authn/authz issue.
> Therefore, I updated the PIP document.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DqqHO48yEXJV8Ci%2B7RJBVcZOgGLgXeNEuBMbNLUTtEw%3D&reserved=0
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Sijie,
>
> > If the lazy-loading approach sounds attractive to you
> and you like it,
> > maybe the next step is to update the PIP, what do you
> think?
>
> I think so too. I will update the PIP after discussing the
> authn/authz issue.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Sijie Guo" <gu...@gmail.com> wrote:
>
> Hi Yuri,
>
> Regarding the authn/authz issue, @Matteo Merli <
> mmerli@apache.org> can
> probably chime in more about that part.
>
> If the lazy-loading approach sounds attractive to you
> and you like it,
> maybe the next step is to update the PIP, what do you
> think?
>
> - Sijie
>
> On Mon, Feb 8, 2021 at 6:57 PM Yuri Mizushima <
> yumizush@yahoo-corp.jp>
> wrote:
>
> > Michael,
> >
> > Thank you for your comment!
> >
> > > Which Pulsar Clients will benefit from this
> proposal?
> > I think that this proposal will be useful to any
> clients.
> > In my schedule, if this proposal is accepted then I
> will implement this
> > feature to Java client.
> > If needed, then implement same feature to other
> clients such as C++, Go,
> > etc.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Michael Marshall" <mi...@gmail.com> wrote:
> >
> > Hi Yuri and Sijie,
> >
> > I definitely like the idea of lazily creating
> producers as well as
> > introducing a way to provide custom routing logic.
> >
> > Which Pulsar Clients will benefit from this
> proposal? I’d love to see
> > this feature in the go client.
> >
> > Thanks,
> > Michael Marshall
> >
> > > On Feb 7, 2021, at 9:53 PM, Yuri Mizushima <
> yumizush@yahoo-corp.jp>
> > wrote:
> > >
> > > Sijie,
> > >
> > > Thank you for sharing!
> > >
> > > First, I considered your suggestion.
> > > I think these implementations sound good.
> > >
> > > I think we should consider the State of
> partitioned producer: Ready,
> > Connecting, etc.
> > > Currently, partitioned producer gets "Ready"
> only when all producers
> > connect to Broker correctly.
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FPartitionedProducerImpl.java%23L146&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Sq7Qw3nmVq725Bwy4rUdreqvhcbAn7qHXN5vEELqqNY%3D&reserved=0
> > > It seems that we should change meaning of
> state (or change handling)
> > if we introduce the lazy-load feature.
> > > To guarantee the message ordering (e.g. using
> partitionKey),
> > partitioned producer should stop (or don't send
> messages to be routed to
> > unavailable partition) when producer can't connect
> to one of partition.
> > >
> > > Secondly, I considered Matteo's comments.
> > > I couldn't understand well about issue of
> authn/authz. Please tell
> > me more detail.
> > >
> > > I wrote "connection" as number of producers
> which connect to broker.
> > Also, TCP connections between partitioned producer
> and broker will be less
> > than or equal to current in some cases. I'll show a
> case below.
> > >
> > > Suppose
> > > * cluster has Broker0, 1, 2
> > > * partitioned topic has 5 partitions
> > > * limit conf is 3 partitions
> > > * loadbalance partitions as below
> > > - Broker0: partition-0, partition-1
> > > - Broker1: partition-2
> > > - Broker2: partition-3, partition-4
> > >
> > > Currently, client will create 3 connections
> (Broker0, 1, 2). If
> > client uses limit conf and elects partitions such as
> [0, 1, 2], then client
> > will create 2 connections (Broker0, 1). Of course,
> if client elects
> > partitions such as [0, 2, 3], then client will still
> create 3 connections.
> > >
> > > I'd like to decrease number of producers. I
> think that resources of
> > broker will be improved slightly by this feature
> because broker has list of
> > producers by some classes such as ServerCnx,
> AbstractTopic.
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FServerCnx.java%23L1096-L1097&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2F0vpCDidkG38A4YyqhDZ2opj71OxPXHLFKfitPeMXto%3D&reserved=0
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FAbstractTopic.java%23L577&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rKpJuX6im00enGDM0HVAYPw%2FjBgsotLcokaIVknVwaE%3D&reserved=0
> > >
> > > In my case, unspecified number of producers
> will connect to the same
> > partitioned topic with different rate. We need to
> set the number of
> > partitions according to the high-rate producer.
> > > However, on the other hand, this number is
> excessively large for
> > low-rate producers.
> > > I want to reduce such redundant producers for
> resource efficiency.
> > >
> > > Regards,
> > > --
> > > Yuri Mizushima
> > > yumizush@yahoo-corp.jp
> > >
> > >
> > > "Sijie Guo" <gu...@gmail.com> wrote:
> > >
> > > Hi Yuri,
> > >
> > > In today's community meeting, Matteo shared
> some of his thoughts
> > about this
> > > PIP.
> > >
> > > You can find some meeting notes here:
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE%2Fedit%23bookmark%3Did.rezbt4xmjxpz&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Rst44OBvITL%2BbKigNhLVE6z428Laca6UmqDbG2V2IRs%3D&reserved=0
> > >
> > > Matteo can also chime in as well.
> > >
> > > - Sijie
> > >
> > >> On Sun, Jan 31, 2021 at 7:21 PM Yuri
> Mizushima <
> > yumizush@yahoo-corp.jp>
> > >> wrote:
> > >> Sijie,
> > >> Thank you for your reply!
> > >> I'll check it.
> > >> Regards,
> > >> --
> > >> Yuri Mizushima
> > >> yumizush@yahoo-corp.jp
> > >> "Sijie Guo" <gu...@gmail.com> wrote:
> > >> Yuri,
> > >> Thank you for bringing this up! This is a
> super helpful proposal!
> > >> The problem is very similar to what an RPC
> framework (like Finagle)
> > >> with
> > >> client-side load balancing has.
> > >> An RPC framework with a client-side
> load-balancing mechanism needs
> > to
> > >> send
> > >> requests across multiple nodes. If you have
> an RPC service that has
> > >> thousands of nodes, there are thousands of
> clients connecting to
> > that
> > >> RPC
> > >> service. How to reduce the connections and
> how to effectively load
> > >> balance
> > >> requests across thousands of nodes are the
> problems that a
> > client-side
> > >> loading technology needs to solve. If you
> think about "partition"
> > as
> > >> "node"
> > >> and "partitioned producer" as "RPC client",
> the problem is exactly
> > the
> > >> same. Finagle (the Twitter RPC framework)
> has implemented a lot of
> > >> client-side
> > >> load-balancing algorithms
> > >> <
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.github.io%2Ffinagle%2Fguide%2FClients.html%23load-balancing&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0qZmzlNFhR17Yl3ucmA57TTZEXN%2Bswrtj1Ib95P2TtU%3D&reserved=0
> >
> > >> and
> > >> there are some great articles that you can
> reference
> > >> <
> > >>
> >
> https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer.html
> > >> .
> > >> I agree with the direction of introducing a
> mechanism to reduce the
> > >> number
> > >> of producers in a partitioned topic
> producer. However, I have a
> > concern
> > >> about introducing `.numPartitionsLimit(10)`
> directly to the
> > producer
> > >> builder. It limits the possibility to
> implement different
> > algorithms on
> > >> selecting partitions.
> > >> So instead of directly implementing the
> logic within the
> > partitioned
> > >> topic
> > >> producer, I think the proposal can be broken
> into two parts:
> > >> 1) Introduce some kind of lazy-loading
> mechanism in the partitioned
> > >> producer to initialize the producers for
> partitions lazily. I.e.,
> > only
> > >> initialize a producer when the message
> router selects a partition.
> > >> 2) Implement a message router that only
> selects one or N
> > partitions.
> > >> In this way, the partitioned producer is
> only responsible for
> > managing
> > >> a
> > >> collection of producers, and the message
> router is responsible for
> > >> selecting the partitions. This allows people
> to be able to
> > implement
> > >> different message routers. We can even adopt
> the client-side load
> > >> balancing
> > >> algorithms from Finagle.
> > >> Thanks,
> > >> Sijie
> > >> On Wed, Jan 27, 2021 at 7:18 PM Yuri
> Mizushima <
> > yumizush@yahoo-corp.jp
> > >> wrote:
> > >>> I notice that PIP-78 has already assigned to
> another issue.
> > >>
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fpulsar-dev%2F202101.mbox%2F%253CCAG%253DTQOrPH49v9ToDE_aeQzEiDC%252BEgSR61ERoqanpWfQGvEB_Vw%2540mail.gmail.com%253E&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6pYP5laeruIaSJ7ASdp3hhRSIqUviMvUduq485hIUP0%3D&reserved=0
> > >>> So, I'll change the PIP number to 79.
> > >>
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DqqHO48yEXJV8Ci%2B7RJBVcZOgGLgXeNEuBMbNLUTtEw%3D&reserved=0
> > >>> Regards,
> > >>> --
> > >>> Yuri Mizushima
> > >>> yumizush@yahoo-corp.jp
> > >>> "Yuri Mizushima" <yu...@yahoo-corp.jp>
> wrote:
> > >>> Dear Pulsar community,
> > >>> When partitioned producer connects to
> partitioned topic,
> > >>> sometimes doesn't need to connect to all of
> partitions depending
> > >> on
> > >>> rate, routing mode, etc.
> > >>> So, I drafted a PIP about reducing
> redundant producers from
> > >>> partitioned producer.
> > >>> I'd like to use system resources (e.g.
> connections between
> > >> Client and
> > >>> Broker, memory usage of both Client and
> Broker)
> > >>> more efficiently by this feature.
> > >>
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-78%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655082883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=s%2BmbH9WWT3qihfOPc%2BToh%2BkWDNAiqEUv%2F%2B19oUKpeOY%3D&reserved=0
> > >>> Feel free to ask me any questions or
> suggestions, etc.
> > >>> Best regards,
> > >>> --
> > >>> Yuri Mizushima
> > >>> yumizush@yahoo-corp.jp
> >
> >
>
>
>
>
>
Re: [PIP-78] Reduce redundant producers from partitioned producer
Posted by Enrico Olivelli <eo...@gmail.com>.
Yuri,
IIUC with this change the client will control which metrics are reported by
the broker ?
I am not sure it is a good idea, because metrics are usually managed by the
owners of the brokers, who sometimes are not the same who run the clients.
Also, I am not sure if this way it is possible for the client to force the
Broker to create many metrics and create some kind of damage.
Would it be better to add a Broker configuration flag to turn on this
feature ? I mean to allow the client to select the type of metrics ?
Enrico
Il giorno gio 2 dic 2021 alle ore 03:00 Yuri Mizushima <
yumizush@yahoo-corp.jp> ha scritto:
> Do you have any comments?
> If there are no comments by Dec. 7, I will close the discussion and rebase
> the PR commit to current master.
>
> Regards,
>
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> On 2021/11/16 15:46, "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Dear Pulsar community,
>
> I have created a new PR
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Sn57dCx1iItNEMArKWAXxE9frCbmQClw%2Fts0nmMkwyw%3D&reserved=0
> for stats aggregation,
> but I didn't discuss about the wire protocol change. I hope we will
> discuss it here.
>
> Currently, partitioned producer can't aggregate by any key such as
> cnx, producerId, producerName, and so on.
> I think we need to add any aggregation system.
> Therefore, added new aggregation policy as producerName (with client
> side implementation).
>
> New protocol field partial_producer_supported is not used for stats
> aggregation. It is used for backward compatibility.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401%2Ffiles%23diff-f29399fed32e0916cf28452ba71078a3ae5ed77bbaef9f52a925165d8ee66cfdR489&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QswqgZHSeHJArmgAdBx1wPrm6rX4l4lTf39hs57g6NI%3D&reserved=0
>
> In my understanding, if introduce new stats aggregation key to client
> side,
> need a way to determine whether the feature is enabled at client side.
> For example, whether the producer has specific field or metadata,
> the version (e.g. protocol version) is greater than threshold, etc.
>
> Of course, if we can introduce aggregation feature without adding any
> new key or implementations from client side,
> we can support the feature not only new client but also old one.
>
> I'm looking forward to your opinions or suggestions to this PR.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> On 2021/05/11 14:26, "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
>
> Dear Pulsar Community,
>
> > I will submit the next PR about PartitionedTopicStats later.
> I submitted the next PR for this PIP. If you have any suggestions,
> please comment to this PR.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10534&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cHqzaY1H4N%2FaQEemy5ZRQSBAiXa3FDXDuFLyoJ4dRtA%3D&reserved=0
>
> Regards,
>
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Dear Pulsar Community,
>
> I submitted the PR for this PIP.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10279&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AzmPY98arKmn7UM1%2F6FJMwOKOavPVC5xVG%2Br5FBJlH4%3D&reserved=0
>
> This is a part of implementations.
> I will submit the next PR about PartitionedTopicStats later.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Sijie,
>
> After sending previous mail, I watched meeting recording and
> understand about authn/authz issue.
> Therefore, I updated the PIP document.
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lZrLE4EhYfy7xUNmfUraIchYOkH%2B0HQaeVcKGaTBWLw%3D&reserved=0
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
>
> Sijie,
>
> > If the lazy-loading approach sounds attractive to you
> and you like it,
> > maybe the next step is to update the PIP, what do you
> think?
>
> I think so too. I will update the PIP after discussing the
> authn/authz issue.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Sijie Guo" <gu...@gmail.com> wrote:
>
> Hi Yuri,
>
> Regarding the authn/authz issue, @Matteo Merli <
> mmerli@apache.org> can
> probably chime in more about that part.
>
> If the lazy-loading approach sounds attractive to you
> and you like it,
> maybe the next step is to update the PIP, what do you
> think?
>
> - Sijie
>
> On Mon, Feb 8, 2021 at 6:57 PM Yuri Mizushima <
> yumizush@yahoo-corp.jp>
> wrote:
>
> > Michael,
> >
> > Thank you for your comment!
> >
> > > Which Pulsar Clients will benefit from this
> proposal?
> > I think that this proposal will be useful to any
> clients.
> > In my schedule, if this proposal is accepted then I
> will implement this
> > feature to Java client.
> > If needed, then implement same feature to other
> clients such as C++, Go,
> > etc.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Michael Marshall" <mi...@gmail.com> wrote:
> >
> > Hi Yuri and Sijie,
> >
> > I definitely like the idea of lazily creating
> producers as well as
> > introducing a way to provide custom routing logic.
> >
> > Which Pulsar Clients will benefit from this
> proposal? I’d love to see
> > this feature in the go client.
> >
> > Thanks,
> > Michael Marshall
> >
> > > On Feb 7, 2021, at 9:53 PM, Yuri Mizushima <
> yumizush@yahoo-corp.jp>
> > wrote:
> > >
> > > Sijie,
> > >
> > > Thank you for sharing!
> > >
> > > First, I considered your suggestion.
> > > I think these implementations sound good.
> > >
> > > I think we should consider the State of
> partitioned producer: Ready,
> > Connecting, etc.
> > > Currently, partitioned producer gets "Ready"
> only when all producers
> > connect to Broker correctly.
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FPartitionedProducerImpl.java%23L146&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EKgraLiuHSkiE9un5L6sWeS8PQLHLkM9nxiqGWt%2FFhw%3D&reserved=0
> > > It seems that we should change meaning of
> state (or change handling)
> > if we introduce the lazy-load feature.
> > > To guarantee the message ordering (e.g. using
> partitionKey),
> > partitioned producer should stop (or don't send
> messages to be routed to
> > unavailable partition) when producer can't connect
> to one of partition.
> > >
> > > Secondly, I considered Matteo's comments.
> > > I couldn't understand well about issue of
> authn/authz. Please tell
> > me more detail.
> > >
> > > I wrote "connection" as number of producers
> which connect to broker.
> > Also, TCP connections between partitioned producer
> and broker will be less
> > than or equal to current in some cases. I'll show a
> case below.
> > >
> > > Suppose
> > > * cluster has Broker0, 1, 2
> > > * partitioned topic has 5 partitions
> > > * limit conf is 3 partitions
> > > * loadbalance partitions as below
> > > - Broker0: partition-0, partition-1
> > > - Broker1: partition-2
> > > - Broker2: partition-3, partition-4
> > >
> > > Currently, client will create 3 connections
> (Broker0, 1, 2). If
> > client uses limit conf and elects partitions such as
> [0, 1, 2], then client
> > will create 2 connections (Broker0, 1). Of course,
> if client elects
> > partitions such as [0, 2, 3], then client will still
> create 3 connections.
> > >
> > > I'd like to decrease number of producers. I
> think that resources of
> > broker will be improved slightly by this feature
> because broker has list of
> > producers by some classes such as ServerCnx,
> AbstractTopic.
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FServerCnx.java%23L1096-L1097&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rx3Ov4r4dlL5YUpt4Bbl%2FhJGpmoeXNsMmS33TDcYrF8%3D&reserved=0
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FAbstractTopic.java%23L577&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=hx910MdAyt9jz7CfcYMxRuRFbkOJKUGe8mrpb67JmqA%3D&reserved=0
> > >
> > > In my case, unspecified number of producers
> will connect to the same
> > partitioned topic with different rate. We need to
> set the number of
> > partitions according to the high-rate producer.
> > > However, on the other hand, this number is
> excessively large for
> > low-rate producers.
> > > I want to reduce such redundant producers for
> resource efficiency.
> > >
> > > Regards,
> > > --
> > > Yuri Mizushima
> > > yumizush@yahoo-corp.jp
> > >
> > >
> > > "Sijie Guo" <gu...@gmail.com> wrote:
> > >
> > > Hi Yuri,
> > >
> > > In today's community meeting, Matteo shared
> some of his thoughts
> > about this
> > > PIP.
> > >
> > > You can find some meeting notes here:
> > >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE%2Fedit%23bookmark%3Did.rezbt4xmjxpz&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=f0bK%2Bzbd2NnxCgVLTYty%2BeIftpfM0%2BpBFC%2FySEBARoo%3D&reserved=0
> > >
> > > Matteo can also chime in as well.
> > >
> > > - Sijie
> > >
> > >> On Sun, Jan 31, 2021 at 7:21 PM Yuri
> Mizushima <
> > yumizush@yahoo-corp.jp>
> > >> wrote:
> > >> Sijie,
> > >> Thank you for your reply!
> > >> I'll check it.
> > >> Regards,
> > >> --
> > >> Yuri Mizushima
> > >> yumizush@yahoo-corp.jp
> > >> "Sijie Guo" <gu...@gmail.com> wrote:
> > >> Yuri,
> > >> Thank you for bringing this up! This is a
> super helpful proposal!
> > >> The problem is very similar to what an RPC
> framework (like Finagle)
> > >> with
> > >> client-side load balancing has.
> > >> An RPC framework with a client-side
> load-balancing mechanism needs
> > to
> > >> send
> > >> requests across multiple nodes. If you have
> an RPC service that has
> > >> thousands of nodes, there are thousands of
> clients connecting to
> > that
> > >> RPC
> > >> service. How to reduce the connections and
> how to effectively load
> > >> balance
> > >> requests across thousands of nodes are the
> problems that a
> > client-side
> > >> loading technology needs to solve. If you
> think about "partition"
> > as
> > >> "node"
> > >> and "partitioned producer" as "RPC client",
> the problem is exactly
> > the
> > >> same. Finagle (the Twitter RPC framework)
> has implemented a lot of
> > >> client-side
> > >> load-balancing algorithms
> > >> <
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.github.io%2Ffinagle%2Fguide%2FClients.html%23load-balancing&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FeyGeIXYMkZ9dEtEklKd3TEbN0rst2tyIEVAznbidXs%3D&reserved=0
> >
> > >> and
> > >> there are some great articles that you can
> reference
> > >> <
> > >>
> >
> https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer.html
> > >> .
> > >> I agree with the direction of introducing a
> mechanism to reduce the
> > >> number
> > >> of producers in a partitioned topic
> producer. However, I have a
> > concern
> > >> about introducing `.numPartitionsLimit(10)`
> directly to the
> > producer
> > >> builder. It limits the possibility to
> implement different
> > algorithms on
> > >> selecting partitions.
> > >> So instead of directly implementing the
> logic within the
> > partitioned
> > >> topic
> > >> producer, I think the proposal can be broken
> into two parts:
> > >> 1) Introduce some kind of lazy-loading
> mechanism in the partitioned
> > >> producer to initialize the producers for
> partitions lazily. I.e.,
> > only
> > >> initialize a producer when the message
> router selects a partition.
> > >> 2) Implement a message router that only
> selects one or N
> > partitions.
> > >> In this way, the partitioned producer is
> only responsible for
> > managing
> > >> a
> > >> collection of producers, and the message
> router is responsible for
> > >> selecting the partitions. This allows people
> to be able to
> > implement
> > >> different message routers. We can even adopt
> the client-side load
> > >> balancing
> > >> algorithms from Finagle.
> > >> Thanks,
> > >> Sijie
> > >> On Wed, Jan 27, 2021 at 7:18 PM Yuri
> Mizushima <
> > yumizush@yahoo-corp.jp
> > >> wrote:
> > >>> I notice that PIP-78 has already assigned to
> another issue.
> > >>
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fpulsar-dev%2F202101.mbox%2F%253CCAG%253DTQOrPH49v9ToDE_aeQzEiDC%252BEgSR61ERoqanpWfQGvEB_Vw%2540mail.gmail.com%253E&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EKRnaunNSAtXSWwCGayzd26SXG%2F6oKg15n%2BY1fUrZHA%3D&reserved=0
> > >>> So, I'll change the PIP number to 79.
> > >>
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EVNZADVJ8nsOwi8mS3sErIVkHJ0Cc8LXVlaIQEG0BbU%3D&reserved=0
> > >>> Regards,
> > >>> --
> > >>> Yuri Mizushima
> > >>> yumizush@yahoo-corp.jp
> > >>> "Yuri Mizushima" <yu...@yahoo-corp.jp>
> wrote:
> > >>> Dear Pulsar community,
> > >>> When partitioned producer connects to
> partitioned topic,
> > >>> sometimes doesn't need to connect to all of
> partitions depending
> > >> on
> > >>> rate, routing mode, etc.
> > >>> So, I drafted a PIP about reducing
> redundant producers from
> > >>> partitioned producer.
> > >>> I'd like to use system resources (e.g.
> connections between
> > >> Client and
> > >>> Broker, memory usage of both Client and
> Broker)
> > >>> more efficiently by this feature.
> > >>
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-78%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ncPm1G5ZCCtEfvR682J%2FYHH42G5Ns5eVJn7j%2BE40hR8%3D&reserved=0
> > >>> Feel free to ask me any questions or
> suggestions, etc.
> > >>> Best regards,
> > >>> --
> > >>> Yuri Mizushima
> > >>> yumizush@yahoo-corp.jp
> >
> >
>
>
>
>
>
Re: [PIP-78] Reduce redundant producers from partitioned producer
Posted by Yuri Mizushima <yu...@yahoo-corp.jp>.
Do you have any comments?
If there are no comments by Dec. 7, I will close the discussion and rebase the PR commit to current master.
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
On 2021/11/16 15:46, "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
Dear Pulsar community,
I have created a new PR https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Sn57dCx1iItNEMArKWAXxE9frCbmQClw%2Fts0nmMkwyw%3D&reserved=0 for stats aggregation,
but I didn't discuss about the wire protocol change. I hope we will discuss it here.
Currently, partitioned producer can't aggregate by any key such as cnx, producerId, producerName, and so on.
I think we need to add any aggregation system.
Therefore, added new aggregation policy as producerName (with client side implementation).
New protocol field partial_producer_supported is not used for stats aggregation. It is used for backward compatibility.
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401%2Ffiles%23diff-f29399fed32e0916cf28452ba71078a3ae5ed77bbaef9f52a925165d8ee66cfdR489&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QswqgZHSeHJArmgAdBx1wPrm6rX4l4lTf39hs57g6NI%3D&reserved=0
In my understanding, if introduce new stats aggregation key to client side,
need a way to determine whether the feature is enabled at client side.
For example, whether the producer has specific field or metadata,
the version (e.g. protocol version) is greater than threshold, etc.
Of course, if we can introduce aggregation feature without adding any new key or implementations from client side,
we can support the feature not only new client but also old one.
I'm looking forward to your opinions or suggestions to this PR.
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
On 2021/05/11 14:26, "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
Dear Pulsar Community,
> I will submit the next PR about PartitionedTopicStats later.
I submitted the next PR for this PIP. If you have any suggestions, please comment to this PR.
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10534&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cHqzaY1H4N%2FaQEemy5ZRQSBAiXa3FDXDuFLyoJ4dRtA%3D&reserved=0
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
"Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
Dear Pulsar Community,
I submitted the PR for this PIP.
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10279&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AzmPY98arKmn7UM1%2F6FJMwOKOavPVC5xVG%2Br5FBJlH4%3D&reserved=0
This is a part of implementations.
I will submit the next PR about PartitionedTopicStats later.
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
"Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
Sijie,
After sending previous mail, I watched meeting recording and understand about authn/authz issue.
Therefore, I updated the PIP document.
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lZrLE4EhYfy7xUNmfUraIchYOkH%2B0HQaeVcKGaTBWLw%3D&reserved=0
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
"Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
Sijie,
> If the lazy-loading approach sounds attractive to you and you like it,
> maybe the next step is to update the PIP, what do you think?
I think so too. I will update the PIP after discussing the authn/authz issue.
Regards,
--
Yuri Mizushima
yumizush@yahoo-corp.jp
"Sijie Guo" <gu...@gmail.com> wrote:
Hi Yuri,
Regarding the authn/authz issue, @Matteo Merli <mm...@apache.org> can
probably chime in more about that part.
If the lazy-loading approach sounds attractive to you and you like it,
maybe the next step is to update the PIP, what do you think?
- Sijie
On Mon, Feb 8, 2021 at 6:57 PM Yuri Mizushima <yu...@yahoo-corp.jp>
wrote:
> Michael,
>
> Thank you for your comment!
>
> > Which Pulsar Clients will benefit from this proposal?
> I think that this proposal will be useful to any clients.
> In my schedule, if this proposal is accepted then I will implement this
> feature to Java client.
> If needed, then implement same feature to other clients such as C++, Go,
> etc.
>
> Regards,
> --
> Yuri Mizushima
> yumizush@yahoo-corp.jp
>
>
> "Michael Marshall" <mi...@gmail.com> wrote:
>
> Hi Yuri and Sijie,
>
> I definitely like the idea of lazily creating producers as well as
> introducing a way to provide custom routing logic.
>
> Which Pulsar Clients will benefit from this proposal? I’d love to see
> this feature in the go client.
>
> Thanks,
> Michael Marshall
>
> > On Feb 7, 2021, at 9:53 PM, Yuri Mizushima <yu...@yahoo-corp.jp>
> wrote:
> >
> > Sijie,
> >
> > Thank you for sharing!
> >
> > First, I considered your suggestion.
> > I think these implementations sound good.
> >
> > I think we should consider the State of partitioned producer: Ready,
> Connecting, etc.
> > Currently, partitioned producer gets "Ready" only when all producers
> connect to Broker correctly.
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FPartitionedProducerImpl.java%23L146&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EKgraLiuHSkiE9un5L6sWeS8PQLHLkM9nxiqGWt%2FFhw%3D&reserved=0
> > It seems that we should change meaning of state (or change handling)
> if we introduce the lazy-load feature.
> > To guarantee the message ordering (e.g. using partitionKey),
> partitioned producer should stop (or don't send messages to be routed to
> unavailable partition) when producer can't connect to one of partition.
> >
> > Secondly, I considered Matteo's comments.
> > I couldn't understand well about issue of authn/authz. Please tell
> me more detail.
> >
> > I wrote "connection" as number of producers which connect to broker.
> Also, TCP connections between partitioned producer and broker will be less
> than or equal to current in some cases. I'll show a case below.
> >
> > Suppose
> > * cluster has Broker0, 1, 2
> > * partitioned topic has 5 partitions
> > * limit conf is 3 partitions
> > * loadbalance partitions as below
> > - Broker0: partition-0, partition-1
> > - Broker1: partition-2
> > - Broker2: partition-3, partition-4
> >
> > Currently, client will create 3 connections (Broker0, 1, 2). If
> client uses limit conf and elects partitions such as [0, 1, 2], then client
> will create 2 connections (Broker0, 1). Of course, if client elects
> partitions such as [0, 2, 3], then client will still create 3 connections.
> >
> > I'd like to decrease number of producers. I think that resources of
> broker will be improved slightly by this feature because broker has list of
> producers by some classes such as ServerCnx, AbstractTopic.
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FServerCnx.java%23L1096-L1097&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rx3Ov4r4dlL5YUpt4Bbl%2FhJGpmoeXNsMmS33TDcYrF8%3D&reserved=0
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FAbstractTopic.java%23L577&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=hx910MdAyt9jz7CfcYMxRuRFbkOJKUGe8mrpb67JmqA%3D&reserved=0
> >
> > In my case, unspecified number of producers will connect to the same
> partitioned topic with different rate. We need to set the number of
> partitions according to the high-rate producer.
> > However, on the other hand, this number is excessively large for
> low-rate producers.
> > I want to reduce such redundant producers for resource efficiency.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumizush@yahoo-corp.jp
> >
> >
> > "Sijie Guo" <gu...@gmail.com> wrote:
> >
> > Hi Yuri,
> >
> > In today's community meeting, Matteo shared some of his thoughts
> about this
> > PIP.
> >
> > You can find some meeting notes here:
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE%2Fedit%23bookmark%3Did.rezbt4xmjxpz&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=f0bK%2Bzbd2NnxCgVLTYty%2BeIftpfM0%2BpBFC%2FySEBARoo%3D&reserved=0
> >
> > Matteo can also chime in as well.
> >
> > - Sijie
> >
> >> On Sun, Jan 31, 2021 at 7:21 PM Yuri Mizushima <
> yumizush@yahoo-corp.jp>
> >> wrote:
> >> Sijie,
> >> Thank you for your reply!
> >> I'll check it.
> >> Regards,
> >> --
> >> Yuri Mizushima
> >> yumizush@yahoo-corp.jp
> >> "Sijie Guo" <gu...@gmail.com> wrote:
> >> Yuri,
> >> Thank you for bringing this up! This is a super helpful proposal!
> >> The problem is very similar to what an RPC framework (like Finagle)
> >> with
> >> client-side load balancing has.
> >> An RPC framework with a client-side load-balancing mechanism needs
> to
> >> send
> >> requests across multiple nodes. If you have an RPC service that has
> >> thousands of nodes, there are thousands of clients connecting to
> that
> >> RPC
> >> service. How to reduce the connections and how to effectively load
> >> balance
> >> requests across thousands of nodes are the problems that a
> client-side
> >> loading technology needs to solve. If you think about "partition"
> as
> >> "node"
> >> and "partitioned producer" as "RPC client", the problem is exactly
> the
> >> same. Finagle (the Twitter RPC framework) has implemented a lot of
> >> client-side
> >> load-balancing algorithms
> >> <
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.github.io%2Ffinagle%2Fguide%2FClients.html%23load-balancing&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FeyGeIXYMkZ9dEtEklKd3TEbN0rst2tyIEVAznbidXs%3D&reserved=0>
> >> and
> >> there are some great articles that you can reference
> >> <
> >>
> https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer.html
> >> .
> >> I agree with the direction of introducing a mechanism to reduce the
> >> number
> >> of producers in a partitioned topic producer. However, I have a
> concern
> >> about introducing `.numPartitionsLimit(10)` directly to the
> producer
> >> builder. It limits the possibility to implement different
> algorithms on
> >> selecting partitions.
> >> So instead of directly implementing the logic within the
> partitioned
> >> topic
> >> producer, I think the proposal can be broken into two parts:
> >> 1) Introduce some kind of lazy-loading mechanism in the partitioned
> >> producer to initialize the producers for partitions lazily. I.e.,
> only
> >> initialize a producer when the message router selects a partition.
> >> 2) Implement a message router that only selects one or N
> partitions.
> >> In this way, the partitioned producer is only responsible for
> managing
> >> a
> >> collection of producers, and the message router is responsible for
> >> selecting the partitions. This allows people to be able to
> implement
> >> different message routers. We can even adopt the client-side load
> >> balancing
> >> algorithms from Finagle.
> >> Thanks,
> >> Sijie
> >> On Wed, Jan 27, 2021 at 7:18 PM Yuri Mizushima <
> yumizush@yahoo-corp.jp
> >> wrote:
> >>> I notice that PIP-78 has already assigned to another issue.
> >>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fpulsar-dev%2F202101.mbox%2F%253CCAG%253DTQOrPH49v9ToDE_aeQzEiDC%252BEgSR61ERoqanpWfQGvEB_Vw%2540mail.gmail.com%253E&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EKRnaunNSAtXSWwCGayzd26SXG%2F6oKg15n%2BY1fUrZHA%3D&reserved=0
> >>> So, I'll change the PIP number to 79.
> >>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EVNZADVJ8nsOwi8mS3sErIVkHJ0Cc8LXVlaIQEG0BbU%3D&reserved=0
> >>> Regards,
> >>> --
> >>> Yuri Mizushima
> >>> yumizush@yahoo-corp.jp
> >>> "Yuri Mizushima" <yu...@yahoo-corp.jp> wrote:
> >>> Dear Pulsar community,
> >>> When partitioned producer connects to partitioned topic,
> >>> sometimes doesn't need to connect to all of partitions depending
> >> on
> >>> rate, routing mode, etc.
> >>> So, I drafted a PIP about reducing redundant producers from
> >>> partitioned producer.
> >>> I'd like to use system resources (e.g. connections between
> >> Client and
> >>> Broker, memory usage of both Client and Broker)
> >>> more efficiently by this feature.
> >>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-78%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ncPm1G5ZCCtEfvR682J%2FYHH42G5Ns5eVJn7j%2BE40hR8%3D&reserved=0
> >>> Feel free to ask me any questions or suggestions, etc.
> >>> Best regards,
> >>> --
> >>> Yuri Mizushima
> >>> yumizush@yahoo-corp.jp
>
>