question about Master/Slave shared-nothing and synchronization

[David Budworth <db...@gmail.com>]

I mistakenly hijacked another thread yesterday when asking this, so I'll
repost it as it's own thread.

If I understand the wiki and comments I've read on this list recently, AMQ
has no state synchronizer.

Meaning that if I have Master / Slave and the Master dies, i can't restart
it without stopping the slave, copying it's database over to the master then
restarting both.

The part I'm confused about is how does work when you have clients
attempting to send messages?

If I start up the master and clients reconnect and start sending messages,
then start the slave, what happens?

Do I need the clients to wait until both master and slave are both up before
it starts sending?

Or does Master -> Slave state sync work fine, it's just that you can't do it
the other way?

Reason I'm asking is that we currently use SonicMQ for their CAA feature
which allows transparent failover / failback which is great.

We would like to use AMQ for some things that have special requirements we
don't get from SMQ but at the same time we don't want to give up the HA
aspects of having the clients not have to be stopped/started in the event we
have a broker failure and want to eventually get back to normal Master/Slave
environment.

Thanks,
-David

Re: question about Master/Slave shared-nothing and synchronization

[David Budworth <db...@gmail.com>]

slight followup to my own question here,

assuming we had a pure master/slave setup and the master dies, we then:

1) stop slave
2) copy files from slave to master
3) delete files from slave?
4) start master
5) (since clients were sitting the retrying, they'll connect and start
sending before slave can be started)
6) start slave

If I understand correctly, since there is no existing state synchronization
that any messages posted to the master's queues at step #5 would be unknown
by the slave and thus 'lost' if the master were to fail again.

Also, if I didn't do step #3, and left the data files as is on the slave and
then at #5 the clients start sending before the slave starts up, we'll have
the issue of lost acks and possible duplicate deliveries in the event of a
failover?

And one more thing, I saw that the slave doesn't allow connections.  But the
socket is open and responds when you connect in to it.  And I see no way
from JMX to check to see if the slave is acting 'master'.
My concern here is that it appears that the only way to tell if the slave is
acting master is to possibly try to send a test message directly to the
slave and see if it throws? (didn't try it, but I'd guess it would)  Or
maybe the connect would through (can't tell just by telnetting into a
server).

On the issue of startup state synchronization, I don't suppose there's a
means (internal api wise) to block all client connections in AMQ?  If we go
the AMQ route for our stuff, we may be able to implement the Master/Slave
synchronization  piece ourselves and contribute it back, assuming it doesn't
involve some kind of fundamental change to the system.  Possibly a
synchronizing hook above the persistence mechanism that would just block
persistence writes while state sync occurs.  Would require the master to
know about the slave though so it can sync back (or be master/slave in the
longest-up-is-master way)

Thanks,
-David



On 5/4/07, David Budworth <db...@gmail.com> wrote:
>
> true, but I was shooting for zero message loss and reasonably high
> performance with no single points of failure
>
> shared db will put us right back at single point of failure(db) that we're
> looking to avoid with MasterSlave
>
> On 5/4/07, Hiram Chirino <hi...@hiramchirino.com> wrote:
> >
> > If you use a shared DB or file system you should be able to failover
> > and fail back without a problem.
> >
> > On 5/4/07, David Budworth <db...@gmail.com> wrote:
> > > I mistakenly hijacked another thread yesterday when asking this, so
> > I'll
> > > repost it as it's own thread.
> > >
> > > If I understand the wiki and comments I've read on this list recently,
> > AMQ
> > > has no state synchronizer.
> > >
> > > Meaning that if I have Master / Slave and the Master dies, i can't
> > restart
> > > it without stopping the slave, copying it's database over to the
> > master then
> > > restarting both.
> > >
> > > The part I'm confused about is how does work when you have clients
> > > attempting to send messages?
> > >
> > > If I start up the master and clients reconnect and start sending
> > messages,
> > > then start the slave, what happens?
> > >
> > > Do I need the clients to wait until both master and slave are both up
> > before
> > > it starts sending?
> > >
> > > Or does Master -> Slave state sync work fine, it's just that you can't
> > do it
> > > the other way?
> > >
> > > Reason I'm asking is that we currently use SonicMQ for their CAA
> > feature
> > > which allows transparent failover / failback which is great.
> > >
> > > We would like to use AMQ for some things that have special
> > requirements we
> > > don't get from SMQ but at the same time we don't want to give up the
> > HA
> > > aspects of having the clients not have to be stopped/started in the
> > event we
> > > have a broker failure and want to eventually get back to normal
> > Master/Slave
> > > environment.
> > >
> > > Thanks,
> > > -David
> > >
> >
> >
> > --
> > Regards,
> > Hiram
> >
> > Blog: http://hiramchirino.com
> >
>
>

Re: question about Master/Slave shared-nothing and synchronization

[David Budworth <db...@gmail.com>]

true, but I was shooting for zero message loss and reasonably high
performance with no single points of failure

shared db will put us right back at single point of failure(db) that we're
looking to avoid with MasterSlave

On 5/4/07, Hiram Chirino <hi...@hiramchirino.com> wrote:
>
> If you use a shared DB or file system you should be able to failover
> and fail back without a problem.
>
> On 5/4/07, David Budworth <db...@gmail.com> wrote:
> > I mistakenly hijacked another thread yesterday when asking this, so I'll
> > repost it as it's own thread.
> >
> > If I understand the wiki and comments I've read on this list recently,
> AMQ
> > has no state synchronizer.
> >
> > Meaning that if I have Master / Slave and the Master dies, i can't
> restart
> > it without stopping the slave, copying it's database over to the master
> then
> > restarting both.
> >
> > The part I'm confused about is how does work when you have clients
> > attempting to send messages?
> >
> > If I start up the master and clients reconnect and start sending
> messages,
> > then start the slave, what happens?
> >
> > Do I need the clients to wait until both master and slave are both up
> before
> > it starts sending?
> >
> > Or does Master -> Slave state sync work fine, it's just that you can't
> do it
> > the other way?
> >
> > Reason I'm asking is that we currently use SonicMQ for their CAA feature
> > which allows transparent failover / failback which is great.
> >
> > We would like to use AMQ for some things that have special requirements
> we
> > don't get from SMQ but at the same time we don't want to give up the HA
> > aspects of having the clients not have to be stopped/started in the
> event we
> > have a broker failure and want to eventually get back to normal
> Master/Slave
> > environment.
> >
> > Thanks,
> > -David
> >
>
>
> --
> Regards,
> Hiram
>
> Blog: http://hiramchirino.com
>

Re: question about Master/Slave shared-nothing and synchronization

[Hiram Chirino <hi...@hiramchirino.com>]

If you use a shared DB or file system you should be able to failover
and fail back without a problem.

On 5/4/07, David Budworth <db...@gmail.com> wrote:
> I mistakenly hijacked another thread yesterday when asking this, so I'll
> repost it as it's own thread.
>
> If I understand the wiki and comments I've read on this list recently, AMQ
> has no state synchronizer.
>
> Meaning that if I have Master / Slave and the Master dies, i can't restart
> it without stopping the slave, copying it's database over to the master then
> restarting both.
>
> The part I'm confused about is how does work when you have clients
> attempting to send messages?
>
> If I start up the master and clients reconnect and start sending messages,
> then start the slave, what happens?
>
> Do I need the clients to wait until both master and slave are both up before
> it starts sending?
>
> Or does Master -> Slave state sync work fine, it's just that you can't do it
> the other way?
>
> Reason I'm asking is that we currently use SonicMQ for their CAA feature
> which allows transparent failover / failback which is great.
>
> We would like to use AMQ for some things that have special requirements we
> don't get from SMQ but at the same time we don't want to give up the HA
> aspects of having the clients not have to be stopped/started in the event we
> have a broker failure and want to eventually get back to normal Master/Slave
> environment.
>
> Thanks,
> -David
>


-- 
Regards,
Hiram

Blog: http://hiramchirino.com