You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Olivier Mallassi <ol...@gmail.com> on 2015/05/20 12:00:51 UTC
how to scale (out) qpid
Hi all
my apologies in advance if I ask dummy questions and for this (maybe too)
long email but *i need some help in regards with "qpid clustering"*.
I have read some of the already asked questions on the ML and would like to
confirm (or not) some points.
*Overview*:
I wanna use the Java Broker (mainly because I need to support several OS -
linux, solaris, windows....) but I am wondering our to scale out (or how to
cluster) qpid (target throughput will be around 150k events per sec,
message size variable but less than 1kB, connections number is not known
but dozens.
I saw some numbers on the ML (like 70k events per sec for a single broker
etc...) so it appears I will have to distribute my load on 2 or 3 brokers.
for now, I am not talking about HA.
*1/ *I can cluster qipd using LVS or corosync. not ideal in my case because
it relies on OS specific solution
*2/* I noticed that the messaging api uses the first IP in the specified
broker list (amqp://guest:guest@
/default?failover='roundrobin?cyclecount='2''&brokerlist=")
So I can use a custom strategy (round robin, zookeeper based), coded in
java, to shuffle the brokers list and ensure to randomly distribute the
connections (when they are established). it works for fail over, can be
tricky if you want to be "elastic" but it should work.
what are your thought on that approach?
*3/* I am yet facing an issue because in some cases, I need to use node
affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I need
to ensure the same consumer jvm will get it.
so I need to ensure that for a given JMSXgroupID, publisher & consumer use
the same broker (which begin to be tricky, even using hashcode, in case of
failover etc...)
Does federation (Dispatch Router) helps in that case (knowing it will add
network hops, and so latency)? Is there any getting started/tutorial that
can help better understanding the fixedAdress and linkRoutePattern?
4/ what about HA? It appears I will finish with a topology like
| | A.Master --> A.Replica
Client | "hash" | B.Master --> B.Replica
| | C.Master --> C.Replica
Hope I was clear enough. many thanks for your help/feedbacks.
Regards
Re: how to scale (out) qpid
Posted by Rajith Muditha Attapattu <ra...@gmail.com>.
Having used Dispatch with some POCs, I'd agree with Jack that it's at least
good to investigate that route ;).
If you want to know more about how you could handle the 0-10 routing model
while using dispatch I could provide some suggestions there.
IIRC there's also an effort to support this natively in dispatch. But I'm
not sure about the status of that yet.
Regards,
Rajith
On Wed, May 20, 2015 at 4:52 PM, Gibson, Jack <ja...@paypal.com.invalid>
wrote:
> Olivier - look at qpid dispatch router that¹s how I'd handle it.
>
> Jack
>
> Jack Gibson
> Chief Architect
> Core Payments Platforms @ PayPal
>
>
>
>
>
>
> On 5/20/15, 5:31 AM, "Olivier Mallassi" <ol...@gmail.com>
> wrote:
>
> >Thank you rob. I will use non persitent messages for this.
> >
> >I like your idea of load-balancing at "message layer" instead of
> >"connection layer"! I need to figure out how to implement this, but this
> >is
> >clearly a more elegant solution (knowing we do not have thousands of
> >processes and connections)
> >
> >PS : for the record, we will also "functionally partitioned" : event of
> >type A goes to broker A (eg. a dedicated JVM), event of type B goes to
> >broker B
> >
> >On Wed, May 20, 2015 at 12:48 PM, Rob Godfrey <ro...@gmail.com>
> >wrote:
> >
> >> On 20 May 2015 at 12:00, Olivier Mallassi <ol...@gmail.com>
> >> wrote:
> >> > Hi all
> >> >
> >> > my apologies in advance if I ask dummy questions and for this (maybe
> >>too)
> >> > long email but *i need some help in regards with "qpid clustering"*.
> >> > I have read some of the already asked questions on the ML and would
> >>like
> >> to
> >> > confirm (or not) some points.
> >> >
> >> > *Overview*:
> >> > I wanna use the Java Broker (mainly because I need to support several
> >>OS
> >> -
> >> > linux, solaris, windows....) but I am wondering our to scale out (or
> >>how
> >> to
> >> > cluster) qpid (target throughput will be around 150k events per sec,
> >> > message size variable but less than 1kB, connections number is not
> >>known
> >> > but dozens.
> >> > I saw some numbers on the ML (like 70k events per sec for a single
> >>broker
> >> > etc...) so it appears I will have to distribute my load on 2 or 3
> >> brokers.
> >> >
> >>
> >> Are you using persistent or non persistent messages? The throughput
> >> the broker can achieve will obviously be very much tied to the type of
> >> messages and the sort of hardware you are using.
> >>
> >> > for now, I am not talking about HA.
> >> >
> >> > *1/ *I can cluster qipd using LVS or corosync. not ideal in my case
> >> because
> >> > it relies on OS specific solution
> >> >
> >> > *2/* I noticed that the messaging api uses the first IP in the
> >>specified
> >> > broker list (amqp://guest:guest@
> >> > /default?failover='roundrobin?cyclecount='2''&brokerlist=")
> >> >
> >> > So I can use a custom strategy (round robin, zookeeper based), coded
> >>in
> >> > java, to shuffle the brokers list and ensure to randomly distribute
> >>the
> >> > connections (when they are established). it works for fail over, can
> >>be
> >> > tricky if you want to be "elastic" but it should work.
> >> >
> >> > what are your thought on that approach?
> >>
> >> Depending on exactly how your application works, that sounds
> >> reasonable... you'd obviously need to ensure that every destination
> >> produced to had consumers on...
> >>
> >> >
> >> > *3/* I am yet facing an issue because in some cases, I need to use
> >>node
> >> > affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I
> >> need
> >> > to ensure the same consumer jvm will get it.
> >> > so I need to ensure that for a given JMSXgroupID, publisher & consumer
> >> use
> >> > the same broker (which begin to be tricky, even using hashcode, in
> >>case
> >> of
> >> > failover etc...)
> >>
> >>
> >> At this point it might seem like it would make more sense for every
> >> producer/consumer to have connections to every broker... and to do the
> >> round-robining at a higher lever than establishing the connection.
> >> The client can then make the choice on a per-message basis as to which
> >> broker to send to - I have seen this pattern used elsewhere.
> >>
> >> > Does federation (Dispatch Router) helps in that case (knowing it will
> >>add
> >> > network hops, and so latency)? Is there any getting started/tutorial
> >>that
> >> > can help better understanding the fixedAdress and linkRoutePattern?
> >> >
> >> > 4/ what about HA? It appears I will finish with a topology like
> >> > | | A.Master --> A.Replica
> >> > Client | "hash" | B.Master --> B.Replica
> >> > | | C.Master --> C.Replica
> >> >
> >>
> >> Unfortunately I'm not very well acquainted with Dispatch - so I'll let
> >> others step in here.
> >>
> >> -- Rob
> >>
> >> >
> >> > Hope I was clear enough. many thanks for your help/feedbacks.
> >> > Regards
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >> For additional commands, e-mail: users-help@qpid.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: how to scale (out) qpid
Posted by "Gibson, Jack" <ja...@paypal.com.INVALID>.
Olivier - look at qpid dispatch router that¹s how I'd handle it.
Jack
Jack Gibson
Chief Architect
Core Payments Platforms @ PayPal
On 5/20/15, 5:31 AM, "Olivier Mallassi" <ol...@gmail.com> wrote:
>Thank you rob. I will use non persitent messages for this.
>
>I like your idea of load-balancing at "message layer" instead of
>"connection layer"! I need to figure out how to implement this, but this
>is
>clearly a more elegant solution (knowing we do not have thousands of
>processes and connections)
>
>PS : for the record, we will also "functionally partitioned" : event of
>type A goes to broker A (eg. a dedicated JVM), event of type B goes to
>broker B
>
>On Wed, May 20, 2015 at 12:48 PM, Rob Godfrey <ro...@gmail.com>
>wrote:
>
>> On 20 May 2015 at 12:00, Olivier Mallassi <ol...@gmail.com>
>> wrote:
>> > Hi all
>> >
>> > my apologies in advance if I ask dummy questions and for this (maybe
>>too)
>> > long email but *i need some help in regards with "qpid clustering"*.
>> > I have read some of the already asked questions on the ML and would
>>like
>> to
>> > confirm (or not) some points.
>> >
>> > *Overview*:
>> > I wanna use the Java Broker (mainly because I need to support several
>>OS
>> -
>> > linux, solaris, windows....) but I am wondering our to scale out (or
>>how
>> to
>> > cluster) qpid (target throughput will be around 150k events per sec,
>> > message size variable but less than 1kB, connections number is not
>>known
>> > but dozens.
>> > I saw some numbers on the ML (like 70k events per sec for a single
>>broker
>> > etc...) so it appears I will have to distribute my load on 2 or 3
>> brokers.
>> >
>>
>> Are you using persistent or non persistent messages? The throughput
>> the broker can achieve will obviously be very much tied to the type of
>> messages and the sort of hardware you are using.
>>
>> > for now, I am not talking about HA.
>> >
>> > *1/ *I can cluster qipd using LVS or corosync. not ideal in my case
>> because
>> > it relies on OS specific solution
>> >
>> > *2/* I noticed that the messaging api uses the first IP in the
>>specified
>> > broker list (amqp://guest:guest@
>> > /default?failover='roundrobin?cyclecount='2''&brokerlist=")
>> >
>> > So I can use a custom strategy (round robin, zookeeper based), coded
>>in
>> > java, to shuffle the brokers list and ensure to randomly distribute
>>the
>> > connections (when they are established). it works for fail over, can
>>be
>> > tricky if you want to be "elastic" but it should work.
>> >
>> > what are your thought on that approach?
>>
>> Depending on exactly how your application works, that sounds
>> reasonable... you'd obviously need to ensure that every destination
>> produced to had consumers on...
>>
>> >
>> > *3/* I am yet facing an issue because in some cases, I need to use
>>node
>> > affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I
>> need
>> > to ensure the same consumer jvm will get it.
>> > so I need to ensure that for a given JMSXgroupID, publisher & consumer
>> use
>> > the same broker (which begin to be tricky, even using hashcode, in
>>case
>> of
>> > failover etc...)
>>
>>
>> At this point it might seem like it would make more sense for every
>> producer/consumer to have connections to every broker... and to do the
>> round-robining at a higher lever than establishing the connection.
>> The client can then make the choice on a per-message basis as to which
>> broker to send to - I have seen this pattern used elsewhere.
>>
>> > Does federation (Dispatch Router) helps in that case (knowing it will
>>add
>> > network hops, and so latency)? Is there any getting started/tutorial
>>that
>> > can help better understanding the fixedAdress and linkRoutePattern?
>> >
>> > 4/ what about HA? It appears I will finish with a topology like
>> > | | A.Master --> A.Replica
>> > Client | "hash" | B.Master --> B.Replica
>> > | | C.Master --> C.Replica
>> >
>>
>> Unfortunately I'm not very well acquainted with Dispatch - so I'll let
>> others step in here.
>>
>> -- Rob
>>
>> >
>> > Hope I was clear enough. many thanks for your help/feedbacks.
>> > Regards
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: how to scale (out) qpid
Posted by Olivier Mallassi <ol...@gmail.com>.
Thank you rob. I will use non persitent messages for this.
I like your idea of load-balancing at "message layer" instead of
"connection layer"! I need to figure out how to implement this, but this is
clearly a more elegant solution (knowing we do not have thousands of
processes and connections)
PS : for the record, we will also "functionally partitioned" : event of
type A goes to broker A (eg. a dedicated JVM), event of type B goes to
broker B
On Wed, May 20, 2015 at 12:48 PM, Rob Godfrey <ro...@gmail.com>
wrote:
> On 20 May 2015 at 12:00, Olivier Mallassi <ol...@gmail.com>
> wrote:
> > Hi all
> >
> > my apologies in advance if I ask dummy questions and for this (maybe too)
> > long email but *i need some help in regards with "qpid clustering"*.
> > I have read some of the already asked questions on the ML and would like
> to
> > confirm (or not) some points.
> >
> > *Overview*:
> > I wanna use the Java Broker (mainly because I need to support several OS
> -
> > linux, solaris, windows....) but I am wondering our to scale out (or how
> to
> > cluster) qpid (target throughput will be around 150k events per sec,
> > message size variable but less than 1kB, connections number is not known
> > but dozens.
> > I saw some numbers on the ML (like 70k events per sec for a single broker
> > etc...) so it appears I will have to distribute my load on 2 or 3
> brokers.
> >
>
> Are you using persistent or non persistent messages? The throughput
> the broker can achieve will obviously be very much tied to the type of
> messages and the sort of hardware you are using.
>
> > for now, I am not talking about HA.
> >
> > *1/ *I can cluster qipd using LVS or corosync. not ideal in my case
> because
> > it relies on OS specific solution
> >
> > *2/* I noticed that the messaging api uses the first IP in the specified
> > broker list (amqp://guest:guest@
> > /default?failover='roundrobin?cyclecount='2''&brokerlist=")
> >
> > So I can use a custom strategy (round robin, zookeeper based), coded in
> > java, to shuffle the brokers list and ensure to randomly distribute the
> > connections (when they are established). it works for fail over, can be
> > tricky if you want to be "elastic" but it should work.
> >
> > what are your thought on that approach?
>
> Depending on exactly how your application works, that sounds
> reasonable... you'd obviously need to ensure that every destination
> produced to had consumers on...
>
> >
> > *3/* I am yet facing an issue because in some cases, I need to use node
> > affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I
> need
> > to ensure the same consumer jvm will get it.
> > so I need to ensure that for a given JMSXgroupID, publisher & consumer
> use
> > the same broker (which begin to be tricky, even using hashcode, in case
> of
> > failover etc...)
>
>
> At this point it might seem like it would make more sense for every
> producer/consumer to have connections to every broker... and to do the
> round-robining at a higher lever than establishing the connection.
> The client can then make the choice on a per-message basis as to which
> broker to send to - I have seen this pattern used elsewhere.
>
> > Does federation (Dispatch Router) helps in that case (knowing it will add
> > network hops, and so latency)? Is there any getting started/tutorial that
> > can help better understanding the fixedAdress and linkRoutePattern?
> >
> > 4/ what about HA? It appears I will finish with a topology like
> > | | A.Master --> A.Replica
> > Client | "hash" | B.Master --> B.Replica
> > | | C.Master --> C.Replica
> >
>
> Unfortunately I'm not very well acquainted with Dispatch - so I'll let
> others step in here.
>
> -- Rob
>
> >
> > Hope I was clear enough. many thanks for your help/feedbacks.
> > Regards
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: how to scale (out) qpid
Posted by Rob Godfrey <ro...@gmail.com>.
On 20 May 2015 at 12:00, Olivier Mallassi <ol...@gmail.com> wrote:
> Hi all
>
> my apologies in advance if I ask dummy questions and for this (maybe too)
> long email but *i need some help in regards with "qpid clustering"*.
> I have read some of the already asked questions on the ML and would like to
> confirm (or not) some points.
>
> *Overview*:
> I wanna use the Java Broker (mainly because I need to support several OS -
> linux, solaris, windows....) but I am wondering our to scale out (or how to
> cluster) qpid (target throughput will be around 150k events per sec,
> message size variable but less than 1kB, connections number is not known
> but dozens.
> I saw some numbers on the ML (like 70k events per sec for a single broker
> etc...) so it appears I will have to distribute my load on 2 or 3 brokers.
>
Are you using persistent or non persistent messages? The throughput
the broker can achieve will obviously be very much tied to the type of
messages and the sort of hardware you are using.
> for now, I am not talking about HA.
>
> *1/ *I can cluster qipd using LVS or corosync. not ideal in my case because
> it relies on OS specific solution
>
> *2/* I noticed that the messaging api uses the first IP in the specified
> broker list (amqp://guest:guest@
> /default?failover='roundrobin?cyclecount='2''&brokerlist=")
>
> So I can use a custom strategy (round robin, zookeeper based), coded in
> java, to shuffle the brokers list and ensure to randomly distribute the
> connections (when they are established). it works for fail over, can be
> tricky if you want to be "elastic" but it should work.
>
> what are your thought on that approach?
Depending on exactly how your application works, that sounds
reasonable... you'd obviously need to ensure that every destination
produced to had consumers on...
>
> *3/* I am yet facing an issue because in some cases, I need to use node
> affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I need
> to ensure the same consumer jvm will get it.
> so I need to ensure that for a given JMSXgroupID, publisher & consumer use
> the same broker (which begin to be tricky, even using hashcode, in case of
> failover etc...)
At this point it might seem like it would make more sense for every
producer/consumer to have connections to every broker... and to do the
round-robining at a higher lever than establishing the connection.
The client can then make the choice on a per-message basis as to which
broker to send to - I have seen this pattern used elsewhere.
> Does federation (Dispatch Router) helps in that case (knowing it will add
> network hops, and so latency)? Is there any getting started/tutorial that
> can help better understanding the fixedAdress and linkRoutePattern?
>
> 4/ what about HA? It appears I will finish with a topology like
> | | A.Master --> A.Replica
> Client | "hash" | B.Master --> B.Replica
> | | C.Master --> C.Replica
>
Unfortunately I'm not very well acquainted with Dispatch - so I'll let
others step in here.
-- Rob
>
> Hope I was clear enough. many thanks for your help/feedbacks.
> Regards
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: how to scale (out) qpid
Posted by Olivier Mallassi <ol...@gmail.com>.
sorry for this long silence. but thank you all for sharing all of this.
On Sun, May 24, 2015 at 12:16 AM, Gibson, Jack <ja...@paypal.com.invalid>
wrote:
> Good point Chuck... here's the example attached to a jira request...
>
> https://issues.apache.org/jira/browse/DISPATCH-141
>
>
> Jack Gibson
> Chief Architect
> Core Payments Platforms @ PayPal
>
>
>
> On 5/21/15, 5:40 AM, "Chuck Rolke" <cr...@redhat.com> wrote:
>
> >The best way to "send an attachment" is to create a jira issue
> >https://issues.apache.org/ and attach the file there. This provides good
> >context for the patch/example/file making it easier to find in the
> >future, gives the author credit for the contribution, and implicitly
> >grants rights to use the file.
> >
> >----- Original Message -----
> >> From: "Olivier Mallassi" <ol...@gmail.com>
> >> To: users@qpid.apache.org
> >> Sent: Thursday, May 21, 2015 3:58:08 AM
> >> Subject: Re: how to scale (out) qpid
> >>
> >> Jack
> >>
> >> Thank you for you answer.
> >> I may miss something but I cannot find any attachment in the email. Am I
> >> wrong? sorry about that.
> >>
> >> Olivier.
> >>
> >> On Wed, May 20, 2015 at 10:57 PM, Gibson, Jack
> >><ja...@paypal.com.invalid>
> >> wrote:
> >>
> >> > Example attachedŠ
> >> >
> >> > Jack Gibson
> >> > Chief Architect
> >> > Core Payments Platforms @ PayPal
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On 5/20/15, 4:11 AM, "SUNDAY A. OLUTAYO" <ol...@sadeeb.com> wrote:
> >> >
> >> > >It will be great if those that have experience with Dispatch Router
> >> > >can help with tutorial, this will grow adoption rate.
> >> > >
> >> > >Thanks,
> >> > >
> >> > >Sunday Olutayo
> >> > >
> >> > >
> >> > >----- Original Message -----
> >> > >From: "Olivier Mallassi" <ol...@gmail.com>
> >> > >To: users@qpid.apache.org
> >> > >Sent: Wednesday, May 20, 2015 11:00:51 AM
> >> > >Subject: how to scale (out) qpid
> >> > >
> >> > >Hi all
> >> > >
> >> > >my apologies in advance if I ask dummy questions and for this (maybe
> >>too)
> >> > >long email but *i need some help in regards with "qpid clustering"*.
> >> > >I have read some of the already asked questions on the ML and would
> >>like
> >> > >to
> >> > >confirm (or not) some points.
> >> > >
> >> > >*Overview*:
> >> > >I wanna use the Java Broker (mainly because I need to support
> >>several OS -
> >> > >linux, solaris, windows....) but I am wondering our to scale out (or
> >>how
> >> > >to
> >> > >cluster) qpid (target throughput will be around 150k events per sec,
> >> > >message size variable but less than 1kB, connections number is not
> >>known
> >> > >but dozens.
> >> > >I saw some numbers on the ML (like 70k events per sec for a single
> >>broker
> >> > >etc...) so it appears I will have to distribute my load on 2 or 3
> >>brokers.
> >> > >
> >> > >for now, I am not talking about HA.
> >> > >
> >> > >*1/ *I can cluster qipd using LVS or corosync. not ideal in my case
> >> > >because
> >> > >it relies on OS specific solution
> >> > >
> >> > >*2/* I noticed that the messaging api uses the first IP in the
> >>specified
> >> > >broker list (amqp://guest:guest@
> >> > >/default?failover='roundrobin?cyclecount='2''&brokerlist=")
> >> > >
> >> > >So I can use a custom strategy (round robin, zookeeper based), coded
> >>in
> >> > >java, to shuffle the brokers list and ensure to randomly distribute
> >>the
> >> > >connections (when they are established). it works for fail over, can
> >>be
> >> > >tricky if you want to be "elastic" but it should work.
> >> > >
> >> > >what are your thought on that approach?
> >> > >
> >> > >*3/* I am yet facing an issue because in some cases, I need to use
> >>node
> >> > >affinity (based on JMSXGroupID...). so given a value of JMSXGroupID,
> >>I
> >> > >need
> >> > >to ensure the same consumer jvm will get it.
> >> > >so I need to ensure that for a given JMSXgroupID, publisher &
> >>consumer use
> >> > >the same broker (which begin to be tricky, even using hashcode, in
> >>case of
> >> > >failover etc...)
> >> > >
> >> > >Does federation (Dispatch Router) helps in that case (knowing it
> >>will add
> >> > >network hops, and so latency)? Is there any getting started/tutorial
> >>that
> >> > >can help better understanding the fixedAdress and linkRoutePattern?
> >> > >
> >> > >4/ what about HA? It appears I will finish with a topology like
> >> > > | | A.Master --> A.Replica
> >> > >Client | "hash" | B.Master --> B.Replica
> >> > > | | C.Master --> C.Replica
> >> > >
> >> > >
> >> > >Hope I was clear enough. many thanks for your help/feedbacks.
> >> > >Regards
> >> > >
> >> > >---------------------------------------------------------------------
> >> > >To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >> > >For additional commands, e-mail: users-help@qpid.apache.org
> >> > >
> >> >
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >> > For additional commands, e-mail: users-help@qpid.apache.org
> >> >
> >>
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >For additional commands, e-mail: users-help@qpid.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: how to scale (out) qpid
Posted by "Gibson, Jack" <ja...@paypal.com.INVALID>.
Good point Chuck... here's the example attached to a jira request...
https://issues.apache.org/jira/browse/DISPATCH-141
Jack Gibson
Chief Architect
Core Payments Platforms @ PayPal
On 5/21/15, 5:40 AM, "Chuck Rolke" <cr...@redhat.com> wrote:
>The best way to "send an attachment" is to create a jira issue
>https://issues.apache.org/ and attach the file there. This provides good
>context for the patch/example/file making it easier to find in the
>future, gives the author credit for the contribution, and implicitly
>grants rights to use the file.
>
>----- Original Message -----
>> From: "Olivier Mallassi" <ol...@gmail.com>
>> To: users@qpid.apache.org
>> Sent: Thursday, May 21, 2015 3:58:08 AM
>> Subject: Re: how to scale (out) qpid
>>
>> Jack
>>
>> Thank you for you answer.
>> I may miss something but I cannot find any attachment in the email. Am I
>> wrong? sorry about that.
>>
>> Olivier.
>>
>> On Wed, May 20, 2015 at 10:57 PM, Gibson, Jack
>><ja...@paypal.com.invalid>
>> wrote:
>>
>> > Example attachedŠ
>> >
>> > Jack Gibson
>> > Chief Architect
>> > Core Payments Platforms @ PayPal
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On 5/20/15, 4:11 AM, "SUNDAY A. OLUTAYO" <ol...@sadeeb.com> wrote:
>> >
>> > >It will be great if those that have experience with Dispatch Router
>> > >can help with tutorial, this will grow adoption rate.
>> > >
>> > >Thanks,
>> > >
>> > >Sunday Olutayo
>> > >
>> > >
>> > >----- Original Message -----
>> > >From: "Olivier Mallassi" <ol...@gmail.com>
>> > >To: users@qpid.apache.org
>> > >Sent: Wednesday, May 20, 2015 11:00:51 AM
>> > >Subject: how to scale (out) qpid
>> > >
>> > >Hi all
>> > >
>> > >my apologies in advance if I ask dummy questions and for this (maybe
>>too)
>> > >long email but *i need some help in regards with "qpid clustering"*.
>> > >I have read some of the already asked questions on the ML and would
>>like
>> > >to
>> > >confirm (or not) some points.
>> > >
>> > >*Overview*:
>> > >I wanna use the Java Broker (mainly because I need to support
>>several OS -
>> > >linux, solaris, windows....) but I am wondering our to scale out (or
>>how
>> > >to
>> > >cluster) qpid (target throughput will be around 150k events per sec,
>> > >message size variable but less than 1kB, connections number is not
>>known
>> > >but dozens.
>> > >I saw some numbers on the ML (like 70k events per sec for a single
>>broker
>> > >etc...) so it appears I will have to distribute my load on 2 or 3
>>brokers.
>> > >
>> > >for now, I am not talking about HA.
>> > >
>> > >*1/ *I can cluster qipd using LVS or corosync. not ideal in my case
>> > >because
>> > >it relies on OS specific solution
>> > >
>> > >*2/* I noticed that the messaging api uses the first IP in the
>>specified
>> > >broker list (amqp://guest:guest@
>> > >/default?failover='roundrobin?cyclecount='2''&brokerlist=")
>> > >
>> > >So I can use a custom strategy (round robin, zookeeper based), coded
>>in
>> > >java, to shuffle the brokers list and ensure to randomly distribute
>>the
>> > >connections (when they are established). it works for fail over, can
>>be
>> > >tricky if you want to be "elastic" but it should work.
>> > >
>> > >what are your thought on that approach?
>> > >
>> > >*3/* I am yet facing an issue because in some cases, I need to use
>>node
>> > >affinity (based on JMSXGroupID...). so given a value of JMSXGroupID,
>>I
>> > >need
>> > >to ensure the same consumer jvm will get it.
>> > >so I need to ensure that for a given JMSXgroupID, publisher &
>>consumer use
>> > >the same broker (which begin to be tricky, even using hashcode, in
>>case of
>> > >failover etc...)
>> > >
>> > >Does federation (Dispatch Router) helps in that case (knowing it
>>will add
>> > >network hops, and so latency)? Is there any getting started/tutorial
>>that
>> > >can help better understanding the fixedAdress and linkRoutePattern?
>> > >
>> > >4/ what about HA? It appears I will finish with a topology like
>> > > | | A.Master --> A.Replica
>> > >Client | "hash" | B.Master --> B.Replica
>> > > | | C.Master --> C.Replica
>> > >
>> > >
>> > >Hope I was clear enough. many thanks for your help/feedbacks.
>> > >Regards
>> > >
>> > >---------------------------------------------------------------------
>> > >To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> > >For additional commands, e-mail: users-help@qpid.apache.org
>> > >
>> >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> > For additional commands, e-mail: users-help@qpid.apache.org
>> >
>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: how to scale (out) qpid
Posted by Chuck Rolke <cr...@redhat.com>.
The best way to "send an attachment" is to create a jira issue https://issues.apache.org/ and attach the file there. This provides good context for the patch/example/file making it easier to find in the future, gives the author credit for the contribution, and implicitly grants rights to use the file.
----- Original Message -----
> From: "Olivier Mallassi" <ol...@gmail.com>
> To: users@qpid.apache.org
> Sent: Thursday, May 21, 2015 3:58:08 AM
> Subject: Re: how to scale (out) qpid
>
> Jack
>
> Thank you for you answer.
> I may miss something but I cannot find any attachment in the email. Am I
> wrong? sorry about that.
>
> Olivier.
>
> On Wed, May 20, 2015 at 10:57 PM, Gibson, Jack <ja...@paypal.com.invalid>
> wrote:
>
> > Example attachedŠ
> >
> > Jack Gibson
> > Chief Architect
> > Core Payments Platforms @ PayPal
> >
> >
> >
> >
> >
> >
> >
> > On 5/20/15, 4:11 AM, "SUNDAY A. OLUTAYO" <ol...@sadeeb.com> wrote:
> >
> > >It will be great if those that have experience with Dispatch Router
> > >can help with tutorial, this will grow adoption rate.
> > >
> > >Thanks,
> > >
> > >Sunday Olutayo
> > >
> > >
> > >----- Original Message -----
> > >From: "Olivier Mallassi" <ol...@gmail.com>
> > >To: users@qpid.apache.org
> > >Sent: Wednesday, May 20, 2015 11:00:51 AM
> > >Subject: how to scale (out) qpid
> > >
> > >Hi all
> > >
> > >my apologies in advance if I ask dummy questions and for this (maybe too)
> > >long email but *i need some help in regards with "qpid clustering"*.
> > >I have read some of the already asked questions on the ML and would like
> > >to
> > >confirm (or not) some points.
> > >
> > >*Overview*:
> > >I wanna use the Java Broker (mainly because I need to support several OS -
> > >linux, solaris, windows....) but I am wondering our to scale out (or how
> > >to
> > >cluster) qpid (target throughput will be around 150k events per sec,
> > >message size variable but less than 1kB, connections number is not known
> > >but dozens.
> > >I saw some numbers on the ML (like 70k events per sec for a single broker
> > >etc...) so it appears I will have to distribute my load on 2 or 3 brokers.
> > >
> > >for now, I am not talking about HA.
> > >
> > >*1/ *I can cluster qipd using LVS or corosync. not ideal in my case
> > >because
> > >it relies on OS specific solution
> > >
> > >*2/* I noticed that the messaging api uses the first IP in the specified
> > >broker list (amqp://guest:guest@
> > >/default?failover='roundrobin?cyclecount='2''&brokerlist=")
> > >
> > >So I can use a custom strategy (round robin, zookeeper based), coded in
> > >java, to shuffle the brokers list and ensure to randomly distribute the
> > >connections (when they are established). it works for fail over, can be
> > >tricky if you want to be "elastic" but it should work.
> > >
> > >what are your thought on that approach?
> > >
> > >*3/* I am yet facing an issue because in some cases, I need to use node
> > >affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I
> > >need
> > >to ensure the same consumer jvm will get it.
> > >so I need to ensure that for a given JMSXgroupID, publisher & consumer use
> > >the same broker (which begin to be tricky, even using hashcode, in case of
> > >failover etc...)
> > >
> > >Does federation (Dispatch Router) helps in that case (knowing it will add
> > >network hops, and so latency)? Is there any getting started/tutorial that
> > >can help better understanding the fixedAdress and linkRoutePattern?
> > >
> > >4/ what about HA? It appears I will finish with a topology like
> > > | | A.Master --> A.Replica
> > >Client | "hash" | B.Master --> B.Replica
> > > | | C.Master --> C.Replica
> > >
> > >
> > >Hope I was clear enough. many thanks for your help/feedbacks.
> > >Regards
> > >
> > >---------------------------------------------------------------------
> > >To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > >For additional commands, e-mail: users-help@qpid.apache.org
> > >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: how to scale (out) qpid
Posted by Olivier Mallassi <ol...@gmail.com>.
Jack
Thank you for you answer.
I may miss something but I cannot find any attachment in the email. Am I
wrong? sorry about that.
Olivier.
On Wed, May 20, 2015 at 10:57 PM, Gibson, Jack <ja...@paypal.com.invalid>
wrote:
> Example attachedŠ
>
> Jack Gibson
> Chief Architect
> Core Payments Platforms @ PayPal
>
>
>
>
>
>
>
> On 5/20/15, 4:11 AM, "SUNDAY A. OLUTAYO" <ol...@sadeeb.com> wrote:
>
> >It will be great if those that have experience with Dispatch Router
> >can help with tutorial, this will grow adoption rate.
> >
> >Thanks,
> >
> >Sunday Olutayo
> >
> >
> >----- Original Message -----
> >From: "Olivier Mallassi" <ol...@gmail.com>
> >To: users@qpid.apache.org
> >Sent: Wednesday, May 20, 2015 11:00:51 AM
> >Subject: how to scale (out) qpid
> >
> >Hi all
> >
> >my apologies in advance if I ask dummy questions and for this (maybe too)
> >long email but *i need some help in regards with "qpid clustering"*.
> >I have read some of the already asked questions on the ML and would like
> >to
> >confirm (or not) some points.
> >
> >*Overview*:
> >I wanna use the Java Broker (mainly because I need to support several OS -
> >linux, solaris, windows....) but I am wondering our to scale out (or how
> >to
> >cluster) qpid (target throughput will be around 150k events per sec,
> >message size variable but less than 1kB, connections number is not known
> >but dozens.
> >I saw some numbers on the ML (like 70k events per sec for a single broker
> >etc...) so it appears I will have to distribute my load on 2 or 3 brokers.
> >
> >for now, I am not talking about HA.
> >
> >*1/ *I can cluster qipd using LVS or corosync. not ideal in my case
> >because
> >it relies on OS specific solution
> >
> >*2/* I noticed that the messaging api uses the first IP in the specified
> >broker list (amqp://guest:guest@
> >/default?failover='roundrobin?cyclecount='2''&brokerlist=")
> >
> >So I can use a custom strategy (round robin, zookeeper based), coded in
> >java, to shuffle the brokers list and ensure to randomly distribute the
> >connections (when they are established). it works for fail over, can be
> >tricky if you want to be "elastic" but it should work.
> >
> >what are your thought on that approach?
> >
> >*3/* I am yet facing an issue because in some cases, I need to use node
> >affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I
> >need
> >to ensure the same consumer jvm will get it.
> >so I need to ensure that for a given JMSXgroupID, publisher & consumer use
> >the same broker (which begin to be tricky, even using hashcode, in case of
> >failover etc...)
> >
> >Does federation (Dispatch Router) helps in that case (knowing it will add
> >network hops, and so latency)? Is there any getting started/tutorial that
> >can help better understanding the fixedAdress and linkRoutePattern?
> >
> >4/ what about HA? It appears I will finish with a topology like
> > | | A.Master --> A.Replica
> >Client | "hash" | B.Master --> B.Replica
> > | | C.Master --> C.Replica
> >
> >
> >Hope I was clear enough. many thanks for your help/feedbacks.
> >Regards
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >For additional commands, e-mail: users-help@qpid.apache.org
> >
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
Re: how to scale (out) qpid
Posted by "Gibson, Jack" <ja...@paypal.com.INVALID>.
Example attached�
Jack Gibson
Chief Architect
Core Payments Platforms @ PayPal
On 5/20/15, 4:11 AM, "SUNDAY A. OLUTAYO" <ol...@sadeeb.com> wrote:
>It will be great if those that have experience with Dispatch Router
>can help with tutorial, this will grow adoption rate.
>
>Thanks,
>
>Sunday Olutayo
>
>
>----- Original Message -----
>From: "Olivier Mallassi" <ol...@gmail.com>
>To: users@qpid.apache.org
>Sent: Wednesday, May 20, 2015 11:00:51 AM
>Subject: how to scale (out) qpid
>
>Hi all
>
>my apologies in advance if I ask dummy questions and for this (maybe too)
>long email but *i need some help in regards with "qpid clustering"*.
>I have read some of the already asked questions on the ML and would like
>to
>confirm (or not) some points.
>
>*Overview*:
>I wanna use the Java Broker (mainly because I need to support several OS -
>linux, solaris, windows....) but I am wondering our to scale out (or how
>to
>cluster) qpid (target throughput will be around 150k events per sec,
>message size variable but less than 1kB, connections number is not known
>but dozens.
>I saw some numbers on the ML (like 70k events per sec for a single broker
>etc...) so it appears I will have to distribute my load on 2 or 3 brokers.
>
>for now, I am not talking about HA.
>
>*1/ *I can cluster qipd using LVS or corosync. not ideal in my case
>because
>it relies on OS specific solution
>
>*2/* I noticed that the messaging api uses the first IP in the specified
>broker list (amqp://guest:guest@
>/default?failover='roundrobin?cyclecount='2''&brokerlist=")
>
>So I can use a custom strategy (round robin, zookeeper based), coded in
>java, to shuffle the brokers list and ensure to randomly distribute the
>connections (when they are established). it works for fail over, can be
>tricky if you want to be "elastic" but it should work.
>
>what are your thought on that approach?
>
>*3/* I am yet facing an issue because in some cases, I need to use node
>affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I
>need
>to ensure the same consumer jvm will get it.
>so I need to ensure that for a given JMSXgroupID, publisher & consumer use
>the same broker (which begin to be tricky, even using hashcode, in case of
>failover etc...)
>
>Does federation (Dispatch Router) helps in that case (knowing it will add
>network hops, and so latency)? Is there any getting started/tutorial that
>can help better understanding the fixedAdress and linkRoutePattern?
>
>4/ what about HA? It appears I will finish with a topology like
> | | A.Master --> A.Replica
>Client | "hash" | B.Master --> B.Replica
> | | C.Master --> C.Replica
>
>
>Hope I was clear enough. many thanks for your help/feedbacks.
>Regards
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>For additional commands, e-mail: users-help@qpid.apache.org
>
Re: how to scale (out) qpid
Posted by "SUNDAY A. OLUTAYO" <ol...@sadeeb.com>.
It will be great if those that have experience with Dispatch Router
can help with tutorial, this will grow adoption rate.
Thanks,
Sunday Olutayo
----- Original Message -----
From: "Olivier Mallassi" <ol...@gmail.com>
To: users@qpid.apache.org
Sent: Wednesday, May 20, 2015 11:00:51 AM
Subject: how to scale (out) qpid
Hi all
my apologies in advance if I ask dummy questions and for this (maybe too)
long email but *i need some help in regards with "qpid clustering"*.
I have read some of the already asked questions on the ML and would like to
confirm (or not) some points.
*Overview*:
I wanna use the Java Broker (mainly because I need to support several OS -
linux, solaris, windows....) but I am wondering our to scale out (or how to
cluster) qpid (target throughput will be around 150k events per sec,
message size variable but less than 1kB, connections number is not known
but dozens.
I saw some numbers on the ML (like 70k events per sec for a single broker
etc...) so it appears I will have to distribute my load on 2 or 3 brokers.
for now, I am not talking about HA.
*1/ *I can cluster qipd using LVS or corosync. not ideal in my case because
it relies on OS specific solution
*2/* I noticed that the messaging api uses the first IP in the specified
broker list (amqp://guest:guest@
/default?failover='roundrobin?cyclecount='2''&brokerlist=")
So I can use a custom strategy (round robin, zookeeper based), coded in
java, to shuffle the brokers list and ensure to randomly distribute the
connections (when they are established). it works for fail over, can be
tricky if you want to be "elastic" but it should work.
what are your thought on that approach?
*3/* I am yet facing an issue because in some cases, I need to use node
affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I need
to ensure the same consumer jvm will get it.
so I need to ensure that for a given JMSXgroupID, publisher & consumer use
the same broker (which begin to be tricky, even using hashcode, in case of
failover etc...)
Does federation (Dispatch Router) helps in that case (knowing it will add
network hops, and so latency)? Is there any getting started/tutorial that
can help better understanding the fixedAdress and linkRoutePattern?
4/ what about HA? It appears I will finish with a topology like
| | A.Master --> A.Replica
Client | "hash" | B.Master --> B.Replica
| | C.Master --> C.Replica
Hope I was clear enough. many thanks for your help/feedbacks.
Regards
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org