You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Praveen M <le...@gmail.com> on 2012/01/14 02:47:29 UTC
Qpid Java Broker High Availability solution?
Hi,
Are there any java broker high availability/clustering solutions that
are currently present? I tried googling around and didn't find anything to
my luck.
Can you please suggest a HA strategy that you've used working with the Qpid
Java Broker?
I found a Message Federation design proposal document, but I'm guessing
it's not implemented yet (Please correct me if I'm wrong).
I plan to spin off two brokers on two different machines and use a failover
connection model to route messages to one if the other goes down. This
works well for message enqueues.
But still, I'd run the risk of not being able to process the messages in
the broker that just went down (until it's back up). It will be nice to
know if someone had solved a similar problem by other
strategies/solutions available with the broker.
Also, has someone tried replicating the database used for
the persistent store to solve this problem (BDB/Derby ?)
Please do share your experience in this area.
Thank you,
--
-Praveen
Re: Qpid Java Broker High Availability solution?
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Meghna,
your question seems to be about building the C++ code, so you might be
better reposting this with a different subject line, as Praveen and I were
discussing the Java Broker.
In order to help the C++ guys help you it would probably also be helpful to
let them know which operating system / version you are trying to build this
on.
Cheers,
Rob
On 20 January 2012 10:45, Senthil Kumar, Sinduja (GE Healthcare) <
Sinduja.SenthilKumar@ge.com> wrote:
> Hi Praveen and Rob,
> I am new to QPID. Can anyone of you help me in creating QPID libraries
> and binaries. This is very urgent and important.
> I downloaded the QPID from the suggested mirror on the HOME website of
> QPID. I gave boostrap [.....qpid0.12/cpp]./bootstrap and then
> ./configure too.
> But finding some problem when giving make. [.....qpid0.12/cpp]make
> The errors are
> File "/home/guest/sinduja/QPID/qpid-0.12/qpid-0.12/python/qpid/ops.py",
> line 236, in load_types_from_xml
> spec = mllib.xml_parse(file)
> File
> "/home/guest/sinduja/QPID/qpid-0.12/qpid-0.12/python/mllib/__init__.py",
> line 80, in xml_parse
> p.parse(source)
> File "/usr/lib64/python2.6/site-packages/_xmlplus/sax/expatreader.py",
> line 109, in parse
> xmlreader.IncrementalParser.parse(self, source)
> File "/usr/lib64/python2.6/site-packages/_xmlplus/sax/xmlreader.py",
> line 123, in parse
> self.feed(buffer)
> File "/usr/lib64/python2.6/site-packages/_xmlplus/sax/expatreader.py",
> line 216, in feed
> self._parser.Parse(data, isFinal)
> File "/usr/lib64/python2.6/site-packages/_xmlplus/sax/expatreader.py",
> line 396, in external_entity_ref
> "")
> File "/usr/lib64/python2.6/site-packages/_xmlplus/sax/saxutils.py",
> line 523, in prepare_input_source
> f = urllib2.urlopen(source.getSystemId())
> File "/usr/lib64/python2.6/urllib2.py", line 124, in urlopen
> return _opener.open(url, data, timeout)
> File "/usr/lib64/python2.6/urllib2.py", line 381, in open
> protocol = req.get_type()
> File "/usr/lib64/python2.6/urllib2.py", line 242, in get_type
> raise ValueError, "unknown url type: %s" % self.__original
> ValueError: unknown url type:
> /home/guest/sinduja/QPID/qpid-0.12/qpid-0.12/python/qpid/specs/amqp-0-10
> .dtd
>
> Can someone help me on this....Am I doing the right version of QPID ?. I
> have just started with QPID. So only if I make this running , I can
> proceed further.
>
> Regards,
> Meghna
>
>
>
>
> -----Original Message-----
> From: Rob Godfrey [mailto:rob.j.godfrey@gmail.com]
> Sent: Friday, January 20, 2012 3:06 PM
> To: users@qpid.apache.org
> Subject: Re: Qpid Java Broker High Availability solution?
>
> Hi Praveen,
>
> On 14 January 2012 02:47, Praveen M <le...@gmail.com> wrote:
>
> > Hi,
> >
> > Are there any java broker high availability/clustering solutions
> > that are currently present? I tried googling around and didn't find
> > anything to my luck.
> >
> > Can you please suggest a HA strategy that you've used working with the
>
> > Qpid Java Broker?
> >
> >
> So where I work we have two separate strategies for "HA" and disaster
> recovery.
>
> For HA we use synchronous replication of the BDB store, with external
> software monitoring the availability of the primary broker machine. If
> the primary broker machine goes down, the external software starts up
> the secondary broker machine, which points to the synchronously
> replicated instance of the store... it can also handle reassignment of
> the IP address / DNS name.
>
> For DR we take regular snapshots of the BDB store files and ship these
> using an FTP-like mechanism to a DR site. Clearly with this solution
> you run the risk of loss as you only have a snapshot from a known point
> in time, not from the very moment the system went down.
>
>
> > I found a Message Federation design proposal document, but I'm
> > guessing it's not implemented yet (Please correct me if I'm wrong).
> >
> >
> There is an alpha/beta implementation of Message Federation in the Java
> Broker, which follows the same design as that in the C++ broker and uses
> the same toolset to create routes. This code is broken in the most
> recent releases of the Java Broker, but should work "better" from
> trunk... however I'm not going to give any guarantees on it's
> suitability for a production system right now (I hope to be doing some
> serious testing/fixing over the next couple of months).
>
>
> > I plan to spin off two brokers on two different machines and use a
> > failover connection model to route messages to one if the other goes
> > down. This works well for message enqueues.
> > But still, I'd run the risk of not being able to process the messages
> > in the broker that just went down (until it's back up). It will be
> > nice to know if someone had solved a similar problem by other
> > strategies/solutions available with the broker.
> >
> > Also, has someone tried replicating the database used for the
> > persistent store to solve this problem (BDB/Derby ?)
> >
> >
> As above, we use replication, but managed by hardware/external software.
> I've not yet tried using BDB's own HA solutions to provide replication.
>
>
> > Please do share your experience in this area.
> >
> >
> Hope this helps,
> Cheers,
> Rob
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
RE: Qpid Java Broker High Availability solution?
Posted by "Senthil Kumar, Sinduja (GE Healthcare)" <Si...@ge.com>.
Hi Praveen and Rob,
I am new to QPID. Can anyone of you help me in creating QPID libraries
and binaries. This is very urgent and important.
I downloaded the QPID from the suggested mirror on the HOME website of
QPID. I gave boostrap [.....qpid0.12/cpp]./bootstrap and then
./configure too.
But finding some problem when giving make. [.....qpid0.12/cpp]make
The errors are
File "/home/guest/sinduja/QPID/qpid-0.12/qpid-0.12/python/qpid/ops.py",
line 236, in load_types_from_xml
spec = mllib.xml_parse(file)
File
"/home/guest/sinduja/QPID/qpid-0.12/qpid-0.12/python/mllib/__init__.py",
line 80, in xml_parse
p.parse(source)
File "/usr/lib64/python2.6/site-packages/_xmlplus/sax/expatreader.py",
line 109, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib64/python2.6/site-packages/_xmlplus/sax/xmlreader.py",
line 123, in parse
self.feed(buffer)
File "/usr/lib64/python2.6/site-packages/_xmlplus/sax/expatreader.py",
line 216, in feed
self._parser.Parse(data, isFinal)
File "/usr/lib64/python2.6/site-packages/_xmlplus/sax/expatreader.py",
line 396, in external_entity_ref
"")
File "/usr/lib64/python2.6/site-packages/_xmlplus/sax/saxutils.py",
line 523, in prepare_input_source
f = urllib2.urlopen(source.getSystemId())
File "/usr/lib64/python2.6/urllib2.py", line 124, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib64/python2.6/urllib2.py", line 381, in open
protocol = req.get_type()
File "/usr/lib64/python2.6/urllib2.py", line 242, in get_type
raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type:
/home/guest/sinduja/QPID/qpid-0.12/qpid-0.12/python/qpid/specs/amqp-0-10
.dtd
Can someone help me on this....Am I doing the right version of QPID ?. I
have just started with QPID. So only if I make this running , I can
proceed further.
Regards,
Meghna
-----Original Message-----
From: Rob Godfrey [mailto:rob.j.godfrey@gmail.com]
Sent: Friday, January 20, 2012 3:06 PM
To: users@qpid.apache.org
Subject: Re: Qpid Java Broker High Availability solution?
Hi Praveen,
On 14 January 2012 02:47, Praveen M <le...@gmail.com> wrote:
> Hi,
>
> Are there any java broker high availability/clustering solutions
> that are currently present? I tried googling around and didn't find
> anything to my luck.
>
> Can you please suggest a HA strategy that you've used working with the
> Qpid Java Broker?
>
>
So where I work we have two separate strategies for "HA" and disaster
recovery.
For HA we use synchronous replication of the BDB store, with external
software monitoring the availability of the primary broker machine. If
the primary broker machine goes down, the external software starts up
the secondary broker machine, which points to the synchronously
replicated instance of the store... it can also handle reassignment of
the IP address / DNS name.
For DR we take regular snapshots of the BDB store files and ship these
using an FTP-like mechanism to a DR site. Clearly with this solution
you run the risk of loss as you only have a snapshot from a known point
in time, not from the very moment the system went down.
> I found a Message Federation design proposal document, but I'm
> guessing it's not implemented yet (Please correct me if I'm wrong).
>
>
There is an alpha/beta implementation of Message Federation in the Java
Broker, which follows the same design as that in the C++ broker and uses
the same toolset to create routes. This code is broken in the most
recent releases of the Java Broker, but should work "better" from
trunk... however I'm not going to give any guarantees on it's
suitability for a production system right now (I hope to be doing some
serious testing/fixing over the next couple of months).
> I plan to spin off two brokers on two different machines and use a
> failover connection model to route messages to one if the other goes
> down. This works well for message enqueues.
> But still, I'd run the risk of not being able to process the messages
> in the broker that just went down (until it's back up). It will be
> nice to know if someone had solved a similar problem by other
> strategies/solutions available with the broker.
>
> Also, has someone tried replicating the database used for the
> persistent store to solve this problem (BDB/Derby ?)
>
>
As above, we use replication, but managed by hardware/external software.
I've not yet tried using BDB's own HA solutions to provide replication.
> Please do share your experience in this area.
>
>
Hope this helps,
Cheers,
Rob
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Qpid Java Broker High Availability solution?
Posted by Praveen M <le...@gmail.com>.
Hi Rob,
That's great. Thanks for taking your time to investigate this.
Thanks,
Praveen
On Mon, Jan 23, 2012 at 6:47 AM, Rob Godfrey <ro...@gmail.com>wrote:
> On 23 January 2012 15:39, Alan Conway <ac...@redhat.com> wrote:
>
> > On 01/21/2012 03:15 PM, Rob Godfrey wrote:
> >
> >> So... don't get your hopes up too high... but I am going to look at
> >> utilising BDB's HA capabilities to implement some sort of Active-Passive
> >> HA
> >> solution... it looks like it shouldn't be *too* much work at first
> glance
> >> (non master nodes block on startup waiting to be elected master, and
> then
> >> configure themselves from the now-master BDB instance).
> >>
> >> At best this is going to be a bit of a hobby project for me as it's not
> >> something that is strictly necessary for my personal end users,
> >>
> >> Cheers,
> >> Rob
> >>
> >>
> > FYI: the new HA design for the C++ cluster is much more amenable to being
> > implemented in Java. It doesn't depend on corosync and uses only AMQP for
> > communication among members. I'll be posting an updated design doc later
> > today, will ping the list.
> >
>
>
> Yes - sorry - I meant to say something about that. I'm very keen to try to
> get that working between brokers once we've had a chance to digest/discuss
> the ideas. Definitely on my plans as well - but probably a little more
> work than the simple BDB HA solution,
>
> Cheers,
> Rob
>
--
-Praveen
Re: Qpid Java Broker High Availability solution?
Posted by Rob Godfrey <ro...@gmail.com>.
On 23 January 2012 15:39, Alan Conway <ac...@redhat.com> wrote:
> On 01/21/2012 03:15 PM, Rob Godfrey wrote:
>
>> So... don't get your hopes up too high... but I am going to look at
>> utilising BDB's HA capabilities to implement some sort of Active-Passive
>> HA
>> solution... it looks like it shouldn't be *too* much work at first glance
>> (non master nodes block on startup waiting to be elected master, and then
>> configure themselves from the now-master BDB instance).
>>
>> At best this is going to be a bit of a hobby project for me as it's not
>> something that is strictly necessary for my personal end users,
>>
>> Cheers,
>> Rob
>>
>>
> FYI: the new HA design for the C++ cluster is much more amenable to being
> implemented in Java. It doesn't depend on corosync and uses only AMQP for
> communication among members. I'll be posting an updated design doc later
> today, will ping the list.
>
Yes - sorry - I meant to say something about that. I'm very keen to try to
get that working between brokers once we've had a chance to digest/discuss
the ideas. Definitely on my plans as well - but probably a little more
work than the simple BDB HA solution,
Cheers,
Rob
Re: Qpid Java Broker High Availability solution?
Posted by Alan Conway <ac...@redhat.com>.
On 01/21/2012 03:15 PM, Rob Godfrey wrote:
> So... don't get your hopes up too high... but I am going to look at
> utilising BDB's HA capabilities to implement some sort of Active-Passive HA
> solution... it looks like it shouldn't be *too* much work at first glance
> (non master nodes block on startup waiting to be elected master, and then
> configure themselves from the now-master BDB instance).
>
> At best this is going to be a bit of a hobby project for me as it's not
> something that is strictly necessary for my personal end users,
>
> Cheers,
> Rob
>
FYI: the new HA design for the C++ cluster is much more amenable to being
implemented in Java. It doesn't depend on corosync and uses only AMQP for
communication among members. I'll be posting an updated design doc later today,
will ping the list.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Qpid Java Broker High Availability solution?
Posted by Rob Godfrey <ro...@gmail.com>.
So... don't get your hopes up too high... but I am going to look at
utilising BDB's HA capabilities to implement some sort of Active-Passive HA
solution... it looks like it shouldn't be *too* much work at first glance
(non master nodes block on startup waiting to be elected master, and then
configure themselves from the now-master BDB instance).
At best this is going to be a bit of a hobby project for me as it's not
something that is strictly necessary for my personal end users,
Cheers,
Rob
On 20 January 2012 17:34, Praveen M <le...@gmail.com> wrote:
> Ah. okie, got it :) I was wondering if you were using some replication
> software that augments BDB that I wasn't aware of.
>
> A SAN explains your architecture. Thanks a lot for writing back :)
>
> On Fri, Jan 20, 2012 at 8:29 AM, Rob Godfrey <rob.j.godfrey@gmail.com
> >wrote:
>
> > On 20 January 2012 17:13, Praveen M <le...@gmail.com> wrote:
> >
> > > Hi Rob,
> > >
> > > Thanks for writing. Please see inline.
> > >
> > > On Fri, Jan 20, 2012 at 1:35 AM, Rob Godfrey <rob.j.godfrey@gmail.com
> > > >wrote:
> > >
> > > > Hi Praveen,
> > > >
> > > > On 14 January 2012 02:47, Praveen M <le...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Are there any java broker high availability/clustering solutions
> > that
> > > > > are currently present? I tried googling around and didn't find
> > anything
> > > > to
> > > > > my luck.
> > > > >
> > > > > Can you please suggest a HA strategy that you've used working with
> > the
> > > > Qpid
> > > > > Java Broker?
> > > > >
> > > > >
> > > > So where I work we have two separate strategies for "HA" and disaster
> > > > recovery.
> > > >
> > > > For HA we use synchronous replication of the BDB store, with external
> > > > software monitoring the availability of the primary broker machine.
> If
> > > the
> > > > primary broker machine goes down, the external software starts up the
> > > > secondary broker machine, which points to the synchronously
> replicated
> > > > instance of the store... it can also handle reassignment of the IP
> > > address
> > > > / DNS name.
> > > >
> > >
> > > *Is there a reason that you use an external software to monitor the
> > > availability of the primary broker machine.?*
> > > *Shouldn't the connection failover model be sufficient enough for this?
> > Or
> > > does the failover model have any limitations? *
> > > *
> > >
> > >
> > The JMS clients failover automatically, the architectural design was not
> > driven by limits in the failover model... however the HA solution is not
> > focused solely on Qpid and aims to provide a service which is as seamless
> > as possible to end user applications
> >
> >
> > > *Also, you mention synchronous replication of BDB. Can you please
> write a
> > > bit about how you go about doing this? I think with syncCommit false,
> > sync
> > > replication could be something that could work for us too without
> > > really jeopardizing the enqueue latencies.*
> > >
> > >
> > >
> > The synchronous replication in our case is done at the "hardware" level.
> > The storage attached to the machines provides this replication.
> >
> >
> > > > For DR we take regular snapshots of the BDB store files and ship
> these
> > > > using an FTP-like mechanism to a DR site. Clearly with this solution
> > you
> > > > run the risk of loss as you only have a snapshot from a known point
> in
> > > > time, not from the very moment the system went down.
> > > >
> > > > *Ah yes, this runs the risk of losing messages. Did you not consider
> a
> > > synchronous replication in this case too?*
> > >
> >
> > DR sites are necessarily far enough away from primary sites to make
> > synchronous replication (at least at the storage level) impractical.
> >
> >
> > > *Or is it because of the distance of the DR site that could contribute
> to
> > > high latency round trips. Just curious.*
> > >
> > >
> > Exactly.
> >
> > In general the message broker forms only one part of an application, in a
> > DR scenario many different components with their own stores will have to
> be
> > restarted. At this point the application design needs to be able to
> > recover - most importantly applications need to tolerate duplicates cause
> > by replaying from a point earlier in time than the point at which failure
> > occurred.
> >
> >
> > > In our model our transaction store which contains a copy of the message
> > > will be DR'ed.
> > >
> > >
> > > > > I found a Message Federation design proposal document, but I'm
> > guessing
> > > > > it's not implemented yet (Please correct me if I'm wrong).
> > > > >
> > > > >
> > > > There is an alpha/beta implementation of Message Federation in the
> Java
> > > > Broker, which follows the same design as that in the C++ broker and
> > uses
> > > > the same toolset to create routes. This code is broken in the most
> > > recent
> > > > releases of the Java Broker, but should work "better" from trunk...
> > > however
> > > > I'm not going to give any guarantees on it's suitability for a
> > production
> > > > system right now (I hope to be doing some serious testing/fixing over
> > the
> > > > next couple of months).
> > > >
> > > >
> > > > > I plan to spin off two brokers on two different machines and use a
> > > > failover
> > > > > connection model to route messages to one if the other goes down.
> > This
> > > > > works well for message enqueues.
> > > > > But still, I'd run the risk of not being able to process the
> messages
> > > in
> > > > > the broker that just went down (until it's back up). It will be
> nice
> > to
> > > > > know if someone had solved a similar problem by other
> > > > > strategies/solutions available with the broker.
> > > > >
> > > > > Also, has someone tried replicating the database used for
> > > > > the persistent store to solve this problem (BDB/Derby ?)
> > > > >
> > > > >
> > > > As above, we use replication, but managed by hardware/external
> > software.
> > > > I've not yet tried using BDB's own HA solutions to provide
> replication.
> > > >
> > > > *well. Is the replication too driven by an external software. I'm
> > > curious on how you go about doing a synchronous*
> > > *replication with BDB (as this is the route that we might want to
> take).
> > > Any tips here will be useful. *
> > > *
> > > *
> > >
> >
> > As above the replication I describe is at the storage level. Essentially
> > we're talking about facilities offered by certain Storage Area Network
> > products :-)
> >
> >
> > > *If you are allowed to talk about the hardware/external software piece
> > I'd
> > > love to hear more about your HA*
> > > *architecture. (I do understand sometimes NDAs might stop you. If so,
> > it's
> > > okie).*
> > >
> > >
> > >
> > We use a standard commercial High Availability Cluster software for this
> > purpose, I'm not really at liberty to say which of these products we use
> -
> > but I imagine that all are equally functional in this area.
> >
> > Cheers,
> > Rob
> >
>
>
>
> --
> -Praveen
>
Re: Qpid Java Broker High Availability solution?
Posted by Praveen M <le...@gmail.com>.
Ah. okie, got it :) I was wondering if you were using some replication
software that augments BDB that I wasn't aware of.
A SAN explains your architecture. Thanks a lot for writing back :)
On Fri, Jan 20, 2012 at 8:29 AM, Rob Godfrey <ro...@gmail.com>wrote:
> On 20 January 2012 17:13, Praveen M <le...@gmail.com> wrote:
>
> > Hi Rob,
> >
> > Thanks for writing. Please see inline.
> >
> > On Fri, Jan 20, 2012 at 1:35 AM, Rob Godfrey <rob.j.godfrey@gmail.com
> > >wrote:
> >
> > > Hi Praveen,
> > >
> > > On 14 January 2012 02:47, Praveen M <le...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Are there any java broker high availability/clustering solutions
> that
> > > > are currently present? I tried googling around and didn't find
> anything
> > > to
> > > > my luck.
> > > >
> > > > Can you please suggest a HA strategy that you've used working with
> the
> > > Qpid
> > > > Java Broker?
> > > >
> > > >
> > > So where I work we have two separate strategies for "HA" and disaster
> > > recovery.
> > >
> > > For HA we use synchronous replication of the BDB store, with external
> > > software monitoring the availability of the primary broker machine. If
> > the
> > > primary broker machine goes down, the external software starts up the
> > > secondary broker machine, which points to the synchronously replicated
> > > instance of the store... it can also handle reassignment of the IP
> > address
> > > / DNS name.
> > >
> >
> > *Is there a reason that you use an external software to monitor the
> > availability of the primary broker machine.?*
> > *Shouldn't the connection failover model be sufficient enough for this?
> Or
> > does the failover model have any limitations? *
> > *
> >
> >
> The JMS clients failover automatically, the architectural design was not
> driven by limits in the failover model... however the HA solution is not
> focused solely on Qpid and aims to provide a service which is as seamless
> as possible to end user applications
>
>
> > *Also, you mention synchronous replication of BDB. Can you please write a
> > bit about how you go about doing this? I think with syncCommit false,
> sync
> > replication could be something that could work for us too without
> > really jeopardizing the enqueue latencies.*
> >
> >
> >
> The synchronous replication in our case is done at the "hardware" level.
> The storage attached to the machines provides this replication.
>
>
> > > For DR we take regular snapshots of the BDB store files and ship these
> > > using an FTP-like mechanism to a DR site. Clearly with this solution
> you
> > > run the risk of loss as you only have a snapshot from a known point in
> > > time, not from the very moment the system went down.
> > >
> > > *Ah yes, this runs the risk of losing messages. Did you not consider a
> > synchronous replication in this case too?*
> >
>
> DR sites are necessarily far enough away from primary sites to make
> synchronous replication (at least at the storage level) impractical.
>
>
> > *Or is it because of the distance of the DR site that could contribute to
> > high latency round trips. Just curious.*
> >
> >
> Exactly.
>
> In general the message broker forms only one part of an application, in a
> DR scenario many different components with their own stores will have to be
> restarted. At this point the application design needs to be able to
> recover - most importantly applications need to tolerate duplicates cause
> by replaying from a point earlier in time than the point at which failure
> occurred.
>
>
> > In our model our transaction store which contains a copy of the message
> > will be DR'ed.
> >
> >
> > > > I found a Message Federation design proposal document, but I'm
> guessing
> > > > it's not implemented yet (Please correct me if I'm wrong).
> > > >
> > > >
> > > There is an alpha/beta implementation of Message Federation in the Java
> > > Broker, which follows the same design as that in the C++ broker and
> uses
> > > the same toolset to create routes. This code is broken in the most
> > recent
> > > releases of the Java Broker, but should work "better" from trunk...
> > however
> > > I'm not going to give any guarantees on it's suitability for a
> production
> > > system right now (I hope to be doing some serious testing/fixing over
> the
> > > next couple of months).
> > >
> > >
> > > > I plan to spin off two brokers on two different machines and use a
> > > failover
> > > > connection model to route messages to one if the other goes down.
> This
> > > > works well for message enqueues.
> > > > But still, I'd run the risk of not being able to process the messages
> > in
> > > > the broker that just went down (until it's back up). It will be nice
> to
> > > > know if someone had solved a similar problem by other
> > > > strategies/solutions available with the broker.
> > > >
> > > > Also, has someone tried replicating the database used for
> > > > the persistent store to solve this problem (BDB/Derby ?)
> > > >
> > > >
> > > As above, we use replication, but managed by hardware/external
> software.
> > > I've not yet tried using BDB's own HA solutions to provide replication.
> > >
> > > *well. Is the replication too driven by an external software. I'm
> > curious on how you go about doing a synchronous*
> > *replication with BDB (as this is the route that we might want to take).
> > Any tips here will be useful. *
> > *
> > *
> >
>
> As above the replication I describe is at the storage level. Essentially
> we're talking about facilities offered by certain Storage Area Network
> products :-)
>
>
> > *If you are allowed to talk about the hardware/external software piece
> I'd
> > love to hear more about your HA*
> > *architecture. (I do understand sometimes NDAs might stop you. If so,
> it's
> > okie).*
> >
> >
> >
> We use a standard commercial High Availability Cluster software for this
> purpose, I'm not really at liberty to say which of these products we use -
> but I imagine that all are equally functional in this area.
>
> Cheers,
> Rob
>
--
-Praveen
Re: Qpid Java Broker High Availability solution?
Posted by Rob Godfrey <ro...@gmail.com>.
On 20 January 2012 17:13, Praveen M <le...@gmail.com> wrote:
> Hi Rob,
>
> Thanks for writing. Please see inline.
>
> On Fri, Jan 20, 2012 at 1:35 AM, Rob Godfrey <rob.j.godfrey@gmail.com
> >wrote:
>
> > Hi Praveen,
> >
> > On 14 January 2012 02:47, Praveen M <le...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Are there any java broker high availability/clustering solutions that
> > > are currently present? I tried googling around and didn't find anything
> > to
> > > my luck.
> > >
> > > Can you please suggest a HA strategy that you've used working with the
> > Qpid
> > > Java Broker?
> > >
> > >
> > So where I work we have two separate strategies for "HA" and disaster
> > recovery.
> >
> > For HA we use synchronous replication of the BDB store, with external
> > software monitoring the availability of the primary broker machine. If
> the
> > primary broker machine goes down, the external software starts up the
> > secondary broker machine, which points to the synchronously replicated
> > instance of the store... it can also handle reassignment of the IP
> address
> > / DNS name.
> >
>
> *Is there a reason that you use an external software to monitor the
> availability of the primary broker machine.?*
> *Shouldn't the connection failover model be sufficient enough for this? Or
> does the failover model have any limitations? *
> *
>
>
The JMS clients failover automatically, the architectural design was not
driven by limits in the failover model... however the HA solution is not
focused solely on Qpid and aims to provide a service which is as seamless
as possible to end user applications
> *Also, you mention synchronous replication of BDB. Can you please write a
> bit about how you go about doing this? I think with syncCommit false, sync
> replication could be something that could work for us too without
> really jeopardizing the enqueue latencies.*
>
>
>
The synchronous replication in our case is done at the "hardware" level.
The storage attached to the machines provides this replication.
> > For DR we take regular snapshots of the BDB store files and ship these
> > using an FTP-like mechanism to a DR site. Clearly with this solution you
> > run the risk of loss as you only have a snapshot from a known point in
> > time, not from the very moment the system went down.
> >
> > *Ah yes, this runs the risk of losing messages. Did you not consider a
> synchronous replication in this case too?*
>
DR sites are necessarily far enough away from primary sites to make
synchronous replication (at least at the storage level) impractical.
> *Or is it because of the distance of the DR site that could contribute to
> high latency round trips. Just curious.*
>
>
Exactly.
In general the message broker forms only one part of an application, in a
DR scenario many different components with their own stores will have to be
restarted. At this point the application design needs to be able to
recover - most importantly applications need to tolerate duplicates cause
by replaying from a point earlier in time than the point at which failure
occurred.
> In our model our transaction store which contains a copy of the message
> will be DR'ed.
>
>
> > > I found a Message Federation design proposal document, but I'm guessing
> > > it's not implemented yet (Please correct me if I'm wrong).
> > >
> > >
> > There is an alpha/beta implementation of Message Federation in the Java
> > Broker, which follows the same design as that in the C++ broker and uses
> > the same toolset to create routes. This code is broken in the most
> recent
> > releases of the Java Broker, but should work "better" from trunk...
> however
> > I'm not going to give any guarantees on it's suitability for a production
> > system right now (I hope to be doing some serious testing/fixing over the
> > next couple of months).
> >
> >
> > > I plan to spin off two brokers on two different machines and use a
> > failover
> > > connection model to route messages to one if the other goes down. This
> > > works well for message enqueues.
> > > But still, I'd run the risk of not being able to process the messages
> in
> > > the broker that just went down (until it's back up). It will be nice to
> > > know if someone had solved a similar problem by other
> > > strategies/solutions available with the broker.
> > >
> > > Also, has someone tried replicating the database used for
> > > the persistent store to solve this problem (BDB/Derby ?)
> > >
> > >
> > As above, we use replication, but managed by hardware/external software.
> > I've not yet tried using BDB's own HA solutions to provide replication.
> >
> > *well. Is the replication too driven by an external software. I'm
> curious on how you go about doing a synchronous*
> *replication with BDB (as this is the route that we might want to take).
> Any tips here will be useful. *
> *
> *
>
As above the replication I describe is at the storage level. Essentially
we're talking about facilities offered by certain Storage Area Network
products :-)
> *If you are allowed to talk about the hardware/external software piece I'd
> love to hear more about your HA*
> *architecture. (I do understand sometimes NDAs might stop you. If so, it's
> okie).*
>
>
>
We use a standard commercial High Availability Cluster software for this
purpose, I'm not really at liberty to say which of these products we use -
but I imagine that all are equally functional in this area.
Cheers,
Rob
Re: Qpid Java Broker High Availability solution?
Posted by Praveen M <le...@gmail.com>.
Hi Rob,
Thanks for writing. Please see inline.
On Fri, Jan 20, 2012 at 1:35 AM, Rob Godfrey <ro...@gmail.com>wrote:
> Hi Praveen,
>
> On 14 January 2012 02:47, Praveen M <le...@gmail.com> wrote:
>
> > Hi,
> >
> > Are there any java broker high availability/clustering solutions that
> > are currently present? I tried googling around and didn't find anything
> to
> > my luck.
> >
> > Can you please suggest a HA strategy that you've used working with the
> Qpid
> > Java Broker?
> >
> >
> So where I work we have two separate strategies for "HA" and disaster
> recovery.
>
> For HA we use synchronous replication of the BDB store, with external
> software monitoring the availability of the primary broker machine. If the
> primary broker machine goes down, the external software starts up the
> secondary broker machine, which points to the synchronously replicated
> instance of the store... it can also handle reassignment of the IP address
> / DNS name.
>
*Is there a reason that you use an external software to monitor the
availability of the primary broker machine.?*
*Shouldn't the connection failover model be sufficient enough for this? Or
does the failover model have any limitations? *
*
*
*Also, you mention synchronous replication of BDB. Can you please write a
bit about how you go about doing this? I think with syncCommit false, sync
replication could be something that could work for us too without
really jeopardizing the enqueue latencies.*
> For DR we take regular snapshots of the BDB store files and ship these
> using an FTP-like mechanism to a DR site. Clearly with this solution you
> run the risk of loss as you only have a snapshot from a known point in
> time, not from the very moment the system went down.
>
> *Ah yes, this runs the risk of losing messages. Did you not consider a
synchronous replication in this case too?*
*Or is it because of the distance of the DR site that could contribute to
high latency round trips. Just curious.*
In our model our transaction store which contains a copy of the message
will be DR'ed.
> > I found a Message Federation design proposal document, but I'm guessing
> > it's not implemented yet (Please correct me if I'm wrong).
> >
> >
> There is an alpha/beta implementation of Message Federation in the Java
> Broker, which follows the same design as that in the C++ broker and uses
> the same toolset to create routes. This code is broken in the most recent
> releases of the Java Broker, but should work "better" from trunk... however
> I'm not going to give any guarantees on it's suitability for a production
> system right now (I hope to be doing some serious testing/fixing over the
> next couple of months).
>
>
> > I plan to spin off two brokers on two different machines and use a
> failover
> > connection model to route messages to one if the other goes down. This
> > works well for message enqueues.
> > But still, I'd run the risk of not being able to process the messages in
> > the broker that just went down (until it's back up). It will be nice to
> > know if someone had solved a similar problem by other
> > strategies/solutions available with the broker.
> >
> > Also, has someone tried replicating the database used for
> > the persistent store to solve this problem (BDB/Derby ?)
> >
> >
> As above, we use replication, but managed by hardware/external software.
> I've not yet tried using BDB's own HA solutions to provide replication.
>
> *well. Is the replication too driven by an external software. I'm
curious on how you go about doing a synchronous*
*replication with BDB (as this is the route that we might want to take).
Any tips here will be useful. *
*
*
*If you are allowed to talk about the hardware/external software piece I'd
love to hear more about your HA*
*architecture. (I do understand sometimes NDAs might stop you. If so, it's
okie).*
> > Please do share your experience in this area.
> >
> >
> Hope this helps,
>
*It sure does. Thanks a lot for taking your time to write. *
> Cheers,
> Rob
>
--
-Praveen
Re: Qpid Java Broker High Availability solution?
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Praveen,
On 14 January 2012 02:47, Praveen M <le...@gmail.com> wrote:
> Hi,
>
> Are there any java broker high availability/clustering solutions that
> are currently present? I tried googling around and didn't find anything to
> my luck.
>
> Can you please suggest a HA strategy that you've used working with the Qpid
> Java Broker?
>
>
So where I work we have two separate strategies for "HA" and disaster
recovery.
For HA we use synchronous replication of the BDB store, with external
software monitoring the availability of the primary broker machine. If the
primary broker machine goes down, the external software starts up the
secondary broker machine, which points to the synchronously replicated
instance of the store... it can also handle reassignment of the IP address
/ DNS name.
For DR we take regular snapshots of the BDB store files and ship these
using an FTP-like mechanism to a DR site. Clearly with this solution you
run the risk of loss as you only have a snapshot from a known point in
time, not from the very moment the system went down.
> I found a Message Federation design proposal document, but I'm guessing
> it's not implemented yet (Please correct me if I'm wrong).
>
>
There is an alpha/beta implementation of Message Federation in the Java
Broker, which follows the same design as that in the C++ broker and uses
the same toolset to create routes. This code is broken in the most recent
releases of the Java Broker, but should work "better" from trunk... however
I'm not going to give any guarantees on it's suitability for a production
system right now (I hope to be doing some serious testing/fixing over the
next couple of months).
> I plan to spin off two brokers on two different machines and use a failover
> connection model to route messages to one if the other goes down. This
> works well for message enqueues.
> But still, I'd run the risk of not being able to process the messages in
> the broker that just went down (until it's back up). It will be nice to
> know if someone had solved a similar problem by other
> strategies/solutions available with the broker.
>
> Also, has someone tried replicating the database used for
> the persistent store to solve this problem (BDB/Derby ?)
>
>
As above, we use replication, but managed by hardware/external software.
I've not yet tried using BDB's own HA solutions to provide replication.
> Please do share your experience in this area.
>
>
Hope this helps,
Cheers,
Rob