Distributed Management Operations

[Rawad Assaf <ra...@gmail.com>]

Hello,

I have a use case where the dispatch router should be used with an array of
brokers to provide a scalable distributed work queue.

The queue should be created at runtime using a management API that creates
the queue across all the brokers and advertise it at the level of the
Dispatch router.

My question concerns the transactionality of the above creation. What
happens if the operation succeeds only on a subset of the brokers? Should
we rollback the operation to ensure all the brokers remain in a consistant
state?

There's a good chance this is an "application" concern but I was wondering
if I you have any recommendation for the above question.

Best regards,
Rawad

Re: Distributed Management Operations

[Rob Godfrey <ro...@gmail.com>]

If you wanted to get really fancy... for the Java Broker, one possibility
is to write a virtual host implementation which pulls its config from
something like zookeeper... and have the host push changes to zookeeper /
and some sort of mechanism by which changes made to the zookeeper master
are noticed by the vhost and applied.  For that you'd probably want to
"version" the config of each object (which may not be a bad thing for us to
add in general).

If anyone wants to go down that route and try to implement something,
that'd be a really cool feature to add :-)


-- Rob


On 6 February 2017 at 11:47, Rawad Assaf <ra...@gmail.com> wrote:

> Hello,
>
> These are exactly the type of questions we had when I initiated the thread.
> I was actually wondering if there were any solution for the management of a
> distributed network of amqp components.
>
> In case of lack of sources of inspiration we had an idea to store the
> "target" state in HA store and use it to:
> - update newly joining components.
> - re-attempt the failing operations on brokers that disagree
>
> We are also studying the impact of having inconsistencies between the state
> of components. The cardinality of possible scenarios is problematic. In
> some cases (mostly failures on brokers) the inconsistency is not harmful,
> the side effect being less performance and a higher troubleshooting effort.
> In some other cases (mostly failures on routers) it's less tolerable
> (broken network).
>
> I thought it was worth checking if someone went down this road before and
> would like to share some experience back.
>
> Thanks,
> Rawad
>
>
>
> On Mon, Feb 6, 2017 at 11:04 AM, Rob Godfrey <ro...@gmail.com>
> wrote:
>
> > While this does seem like an application concern... I think rolling back
> > probably makes sense.  More generally how are you planning on dealing
> with
> > inconsistent state between brokers?  What if one of the instances is
> > temporarily off-line when a new queue is created... or what if you want
> to
> > add a new broker to your cluster?  Are you planning on building tooling
> to
> > bring brokers into line with the existing state?  How are you determining
> > the "current" state if you have brokers that disagree?
> >
> > -- Rob
> >
> > On 4 February 2017 at 07:10, Rawad Assaf <ra...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > I have a use case where the dispatch router should be used with an
> array
> > of
> > > brokers to provide a scalable distributed work queue.
> > >
> > > The queue should be created at runtime using a management API that
> > creates
> > > the queue across all the brokers and advertise it at the level of the
> > > Dispatch router.
> > >
> > > My question concerns the transactionality of the above creation. What
> > > happens if the operation succeeds only on a subset of the brokers?
> Should
> > > we rollback the operation to ensure all the brokers remain in a
> > consistant
> > > state?
> > >
> > > There's a good chance this is an "application" concern but I was
> > wondering
> > > if I you have any recommendation for the above question.
> > >
> > > Best regards,
> > > Rawad
> > >
> >
>
>
>
> --
> Rawad.
>

Re: Distributed Management Operations

[Alan Conway <ac...@redhat.com>]

On Mon, 2017-02-06 at 12:47 +0200, Rawad Assaf wrote:
> Hello,
> 
> These are exactly the type of questions we had when I initiated the
> thread.
> I was actually wondering if there were any solution for the
> management of a
> distributed network of amqp components.
> 
> In case of lack of sources of inspiration we had an idea to store the
> "target" state in HA store and use it to:
> - update newly joining components.
> - re-attempt the failing operations on brokers that disagree
> 
> We are also studying the impact of having inconsistencies between the
> state
> of components. The cardinality of possible scenarios is problematic.
> In
> some cases (mostly failures on brokers) the inconsistency is not
> harmful,
> the side effect being less performance and a higher troubleshooting
> effort.
> In some other cases (mostly failures on routers) it's less tolerable
> (broken network).
> 
> I thought it was worth checking if someone went down this road before
> and
> would like to share some experience back.

You are not alone but AFAIK there aren't any complete solutions
available yet. The router itself will probably not grow features to
support transactional/stored/HA config but we are very interested in
making it easy to distribute configuration from such sources over a
dispatch network.

The router network does "self heal" in that address and route
information is automatically propagated to new routers as they join,
but right now that does not extend to brokers or other things attached
to the network.

CC'd gsim for comment, he has been doing some work on combining routers
with external config stores to propagate broker configuration.

> 
> Thanks,
> Rawad
> 
> 
> 
> On Mon, Feb 6, 2017 at 11:04 AM, Rob Godfrey <rob.j.godfrey@gmail.com
> >
> wrote:
> 
> > While this does seem like an application concern... I think rolling
> > back
> > probably makes sense.��More generally how are you planning on
> > dealing with
> > inconsistent state between brokers?��What if one of the instances
> > is
> > temporarily off-line when a new queue is created... or what if you
> > want to
> > add a new broker to your cluster?��Are you planning on building
> > tooling to
> > bring brokers into line with the existing state?��How are you
> > determining
> > the "current" state if you have brokers that disagree?
> > 
> > -- Rob
> > 
> > On 4 February 2017 at 07:10, Rawad Assaf <ra...@gmail.com>
> > wrote:
> > 
> > > Hello,
> > > 
> > > I have a use case where the dispatch router should be used with
> > > an array
> > 
> > of
> > > brokers to provide a scalable distributed work queue.
> > > 
> > > The queue should be created at runtime using a management API
> > > that
> > 
> > creates
> > > the queue across all the brokers and advertise it at the level of
> > > the
> > > Dispatch router.
> > > 
> > > My question concerns the transactionality of the above creation.
> > > What
> > > happens if the operation succeeds only on a subset of the
> > > brokers? Should
> > > we rollback the operation to ensure all the brokers remain in a
> > 
> > consistant
> > > state?
> > > 
> > > There's a good chance this is an "application" concern but I was
> > 
> > wondering
> > > if I you have any recommendation for the above question.
> > > 
> > > Best regards,
> > > Rawad
> > > 
> 
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Distributed Management Operations

[Rawad Assaf <ra...@gmail.com>]

Hello,

These are exactly the type of questions we had when I initiated the thread.
I was actually wondering if there were any solution for the management of a
distributed network of amqp components.

In case of lack of sources of inspiration we had an idea to store the
"target" state in HA store and use it to:
- update newly joining components.
- re-attempt the failing operations on brokers that disagree

We are also studying the impact of having inconsistencies between the state
of components. The cardinality of possible scenarios is problematic. In
some cases (mostly failures on brokers) the inconsistency is not harmful,
the side effect being less performance and a higher troubleshooting effort.
In some other cases (mostly failures on routers) it's less tolerable
(broken network).

I thought it was worth checking if someone went down this road before and
would like to share some experience back.

Thanks,
Rawad



On Mon, Feb 6, 2017 at 11:04 AM, Rob Godfrey <ro...@gmail.com>
wrote:

> While this does seem like an application concern... I think rolling back
> probably makes sense.  More generally how are you planning on dealing with
> inconsistent state between brokers?  What if one of the instances is
> temporarily off-line when a new queue is created... or what if you want to
> add a new broker to your cluster?  Are you planning on building tooling to
> bring brokers into line with the existing state?  How are you determining
> the "current" state if you have brokers that disagree?
>
> -- Rob
>
> On 4 February 2017 at 07:10, Rawad Assaf <ra...@gmail.com> wrote:
>
> > Hello,
> >
> > I have a use case where the dispatch router should be used with an array
> of
> > brokers to provide a scalable distributed work queue.
> >
> > The queue should be created at runtime using a management API that
> creates
> > the queue across all the brokers and advertise it at the level of the
> > Dispatch router.
> >
> > My question concerns the transactionality of the above creation. What
> > happens if the operation succeeds only on a subset of the brokers? Should
> > we rollback the operation to ensure all the brokers remain in a
> consistant
> > state?
> >
> > There's a good chance this is an "application" concern but I was
> wondering
> > if I you have any recommendation for the above question.
> >
> > Best regards,
> > Rawad
> >
>



-- 
Rawad.

Re: Distributed Management Operations

[Rob Godfrey <ro...@gmail.com>]

While this does seem like an application concern... I think rolling back
probably makes sense.  More generally how are you planning on dealing with
inconsistent state between brokers?  What if one of the instances is
temporarily off-line when a new queue is created... or what if you want to
add a new broker to your cluster?  Are you planning on building tooling to
bring brokers into line with the existing state?  How are you determining
the "current" state if you have brokers that disagree?

-- Rob

On 4 February 2017 at 07:10, Rawad Assaf <ra...@gmail.com> wrote:

> Hello,
>
> I have a use case where the dispatch router should be used with an array of
> brokers to provide a scalable distributed work queue.
>
> The queue should be created at runtime using a management API that creates
> the queue across all the brokers and advertise it at the level of the
> Dispatch router.
>
> My question concerns the transactionality of the above creation. What
> happens if the operation succeeds only on a subset of the brokers? Should
> we rollback the operation to ensure all the brokers remain in a consistant
> state?
>
> There's a good chance this is an "application" concern but I was wondering
> if I you have any recommendation for the above question.
>
> Best regards,
> Rawad
>