You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by BALLADA Vincent <vi...@renault.com> on 2021/09/07 09:07:49 UTC
Re: [Proposal] => JmsIO auto scaling feature
Hi Luke,
Hope you are fine, back from holidays and consecutively to our discussion the 5th of august, I replied to your comments regarding the JmsIO auto-scaling proposal:
https://docs.google.com/document/d/1LMPpMpn9DByQcyCdRrTMBa-8ltByO4aqNYY4D9h3m18/edit?usp=sharing
To sum up :
* The number of initial split is not important for us (JmsIO creates one connection/consumer per split and we are ok with that),
* What is important is that the runner allocates more or less workers depending on the backlog size,
* We have added the getSplitBacklogBytes methos to the AutoScaler interface.
I will proceed with the pull request then.
Best regards.
Vincent BALLADA
On 2021/07/19 16:26:09, BALLADA Vincent <v....@renault.com>> wrote:
> Thanks, I enabled comment access.>
>
> De : Luke Cwik <lc...@google.com>>>
> Date : lundi, 19 juillet 2021 à 17:20>
> À : dev <de...@beam.apache.org>>>
> Objet : Re: [Proposal] => JmsIO auto scaling feature>
> Would you enable comment access on your doc?>
>
> On Mon, Jul 19, 2021 at 6:08 AM BALLADA Vincent <vi...@renault.com>>> wrote:>
> Hi,>
>
> I am Vincent from Renault Digital.>
> We are using JmsIO connector within our organization, and we are facing the following issue:>
>
> A lot of our company dataflow jobs use JmsIO connector, as our message bus widely use AMQP protocol.>
> JmsIO doesn't implement the getSplitBacklogBytes nor getTotalBacklogBytes methods, there is no backlog estimate, and dataflow auto scaler is not able to allocate workers depending on the backlog size (scale down and up).>
> Therefore we don't use autoscaling, and we have a fix number of workers.>
>
> I have created the following JIRA ticket:>
> https://issues.apache.org/jira/browse/BEAM-12634>
>
> And I would like to do the following proposal to implement auto scaling feature within JmsIO:>
> https://docs.google.com/document/d/1LMPpMpn9DByQcyCdRrTMBa-8ltByO4aqNYY4D9h3m18/edit?usp=sharing>
>
> Regards,>
>
> Vincent>
>
>
> Confidential C>
>
> -- Disclaimer ------------------------------------>
> Ce message ainsi que les eventuelles pieces jointes constituent une correspondance privee et confidentielle a l'attention exclusive du destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present message ou une personne susceptible de pouvoir le lui delivrer, il vous est signifie que toute divulgation, distribution ou copie de cette transmission est strictement interdite. Si vous avez recu ce message par erreur, nous vous remercions d'en informer l'expediteur par telephone ou de lui retourner le present message, puis d'effacer immediatement ce message de votre systeme.>
>
> *** This e-mail and any attachments is a confidential correspondence intended only for use of the individual or entity named above. If you are not the intended recipient or the agent responsible for delivering the message to the intended recipient, you are hereby notified that any disclosure, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender by phone or by replying this message, and then delete this message from your system.>
>
>
>
> Confidential C>
> -- Disclaimer ------------------------------------ >
> Ce message ainsi que les eventuelles pieces jointes constituent une correspondance privee et confidentielle a l'attention exclusive du destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present message ou une personne susceptible de pouvoir le lui delivrer, il vous est signifie que toute divulgation, distribution ou copie de cette transmission est strictement interdite. Si vous avez recu ce message par erreur, nous vous remercions d'en informer l'expediteur par telephone ou de lui retourner le present message, puis d'effacer immediatement ce message de votre systeme.>
>
> *** This e-mail and any attachments is a confidential correspondence intended only for use of the individual or entity named above. If you are not the intended recipient or the agent responsible for delivering the message to the intended recipient, you are hereby notified that any disclosure, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender by phone or by replying this message, and then delete this message from your system.>
>
Confidential C
-- Disclaimer ------------------------------------
Ce message ainsi que les eventuelles pieces jointes constituent une correspondance privee et confidentielle a l'attention exclusive du destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present message ou une personne susceptible de pouvoir le lui delivrer, il vous est signifie que toute divulgation, distribution ou copie de cette transmission est strictement interdite. Si vous avez recu ce message par erreur, nous vous remercions d'en informer l'expediteur par telephone ou de lui retourner le present message, puis d'effacer immediatement ce message de votre systeme.
*** This e-mail and any attachments is a confidential correspondence intended only for use of the individual or entity named above. If you are not the intended recipient or the agent responsible for delivering the message to the intended recipient, you are hereby notified that any disclosure, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender by phone or by replying this message, and then delete this message from your system.
Re: [Proposal] => JmsIO auto scaling feature
Posted by Luke Cwik <lc...@google.com>.
Sounds great.
On Tue, Sep 7, 2021 at 2:12 AM BALLADA Vincent <vi...@renault.com>
wrote:
> Hi Luke,
>
>
>
> Hope you are fine, back from holidays and consecutively to our discussion
> the 5th of august, I replied to your comments regarding the JmsIO
> auto-scaling proposal:
>
>
> https://docs.google.com/document/d/1LMPpMpn9DByQcyCdRrTMBa-8ltByO4aqNYY4D9h3m18/edit?usp=sharing
>
>
>
> To sum up :
>
> - The number of initial split is not important for us (JmsIO creates
> one connection/consumer per split and we are ok with that),
> - What is important is that the runner allocates more or less workers
> depending on the backlog size,
> - We have added the getSplitBacklogBytes methos to the AutoScaler
> interface.
>
>
>
> I will proceed with the pull request then.
>
>
>
> Best regards.
>
>
>
> Vincent BALLADA
>
> On 2021/07/19 16:26:09, BALLADA Vincent <v....@renault.com> wrote:
>
> > Thanks, I enabled comment access.>
>
> >
>
> > De : Luke Cwik <lc...@google.com>>
>
> > Date : lundi, 19 juillet 2021 à 17:20>
>
> > À : dev <de...@beam.apache.org>>
>
> > Objet : Re: [Proposal] => JmsIO auto scaling feature>
>
> > Would you enable comment access on your doc?>
>
> >
>
> > On Mon, Jul 19, 2021 at 6:08 AM BALLADA Vincent <vi...@renault.com>>
> wrote:>
>
> > Hi,>
>
> >
>
> > I am Vincent from Renault Digital.>
>
> > We are using JmsIO connector within our organization, and we are facing
> the following issue:>
>
> >
>
> > A lot of our company dataflow jobs use JmsIO connector, as our message
> bus widely use AMQP protocol.>
>
> > JmsIO doesn't implement the getSplitBacklogBytes nor
> getTotalBacklogBytes methods, there is no backlog estimate, and dataflow
> auto scaler is not able to allocate workers depending on the backlog size
> (scale down and up).>
>
> > Therefore we don't use autoscaling, and we have a fix number of
> workers.>
>
> >
>
> > I have created the following JIRA ticket:>
>
> > https://issues.apache.org/jira/browse/BEAM-12634>
>
> >
>
> > And I would like to do the following proposal to implement auto scaling
> feature within JmsIO:>
>
> >
> https://docs.google.com/document/d/1LMPpMpn9DByQcyCdRrTMBa-8ltByO4aqNYY4D9h3m18/edit?usp=sharing>
>
>
> >
>
> > Regards,>
>
> >
>
> > Vincent>
>
> >
>
> >
>
> > Confidential C>
>
> >
>
> > -- Disclaimer ------------------------------------>
>
> > Ce message ainsi que les eventuelles pieces jointes constituent une
> correspondance privee et confidentielle a l'attention exclusive du
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du
> present message ou une personne susceptible de pouvoir le lui delivrer, il
> vous est signifie que toute divulgation, distribution ou copie de cette
> transmission est strictement interdite. Si vous avez recu ce message par
> erreur, nous vous remercions d'en informer l'expediteur par telephone ou de
> lui retourner le present message, puis d'effacer immediatement ce message
> de votre systeme.>
>
> >
>
> > *** This e-mail and any attachments is a confidential correspondence
> intended only for use of the individual or entity named above. If you are
> not the intended recipient or the agent responsible for delivering the
> message to the intended recipient, you are hereby notified that any
> disclosure, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by phone or by replying this message, and then delete this
> message from your system.>
>
> >
>
> >
>
> >
>
> > Confidential C>
>
> > -- Disclaimer ------------------------------------ >
>
> > Ce message ainsi que les eventuelles pieces jointes constituent une
> correspondance privee et confidentielle a l'attention exclusive du
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du
> present message ou une personne susceptible de pouvoir le lui delivrer, il
> vous est signifie que toute divulgation, distribution ou copie de cette
> transmission est strictement interdite. Si vous avez recu ce message par
> erreur, nous vous remercions d'en informer l'expediteur par telephone ou de
> lui retourner le present message, puis d'effacer immediatement ce message
> de votre systeme.>
>
> >
>
> > *** This e-mail and any attachments is a confidential correspondence
> intended only for use of the individual or entity named above. If you are
> not the intended recipient or the agent responsible for delivering the
> message to the intended recipient, you are hereby notified that any
> disclosure, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by phone or by replying this message, and then delete this
> message from your system.>
>
> >
>
> Confidential C
>
> -- Disclaimer ------------------------------------
> Ce message ainsi que les eventuelles pieces jointes constituent une
> correspondance privee et confidentielle a l'attention exclusive du
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du
> present message ou une personne susceptible de pouvoir le lui delivrer, il
> vous est signifie que toute divulgation, distribution ou copie de cette
> transmission est strictement interdite. Si vous avez recu ce message par
> erreur, nous vous remercions d'en informer l'expediteur par telephone ou de
> lui retourner le present message, puis d'effacer immediatement ce message
> de votre systeme.
>
> *** This e-mail and any attachments is a confidential correspondence
> intended only for use of the individual or entity named above. If you are
> not the intended recipient or the agent responsible for delivering the
> message to the intended recipient, you are hereby notified that any
> disclosure, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by phone or by replying this message, and then delete this
> message from your system.
>