Dispatch Config Store (was: Re: [Broker-J 7.0.3] Memory configuration store and messages recovery)

[Rob Godfrey <ro...@gmail.com>]

Hi Olivier,

On Mon, 25 Jun 2018 at 09:40, VERMEULEN Olivier <Ol...@murex.com>
wrote:

> Ok so I must keep a persistent config store for the Broker.
> But if I'm not mistaken the Dispacth-Router only supports a non-persistent
> config store.
> So are there any plans to implement new config stores for the
> dispatch-router?
>
>
Updated the subject in the hopes that someone more familiar with the
current and future capabilities of Dispatch can answer your questions.

-- Rob


> Thanks,
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: vendredi 22 juin 2018 13:20
> To: users@qpid.apache.org
> Cc: Keith Wall <ke...@gmail.com>
> Subject: Re: [Broker-J 7.0.3] Memory configuration store and messages
> recovery
>
> Sure - but at that point you really need to provide the vhost config
> before the vhost itself starts up... would providing all the config
> (including the queue UUIDs) in a virtualHostInitialConfiguration provided
> to the VirtualHostNode on creation help I wonder...
>
> -- Rob
>
> On Fri, 22 Jun 2018 at 12:14, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Hello Rob,
> >
> > The main problem is to keep my version of the config in sync with the
> > one of each broker.
> > And knowing that all our brokers have the same config the added
> > complexity seems a bit overkill.
> > Especially when you start thinking about broker restart, scale up,
> > scale down...
> >
> > Olivier
> >
> > -----Original Message-----
> > From: Rob Godfrey <ro...@gmail.com>
> > Sent: vendredi 22 juin 2018 11:27
> > To: users@qpid.apache.org
> > Cc: Keith Wall <ke...@gmail.com>
> > Subject: Re: [Broker-J 7.0.3] Memory configuration store and messages
> > recovery
> >
> > In general, in order to be able to recover the queues from a message
> > store, the broker needs to know details of the queue - not only its
> > name, but also the type of the queue (standard, priority, LVQ, etc) as
> > well as other queue properties... this is precisely the data which is
> > stored in the configuration store, I don't think there's really any
> > way around that problem.  What sort of problems are you experiencing
> > by using a non-Memory configuration store in your setup?
> >
> > -- Rob
> >
> > On Fri, 22 Jun 2018 at 10:47, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com>
> > wrote:
> >
> > > Hello Keith,
> > >
> > > Thanks for the quick answer.
> > > Our target is a cluster of brokers and dispatch-routers.
> > > To configure it we created a management component that centralizes
> > > the configuration of all the Qpid components.
> > > With this approach, having the broker store its own configuration is
> > > redundant with what our management component does and causes some
> > problems.
> > > That's why I was looking into the memory configuration store which
> > > would have made the broker behave like the dispatch-router regarding
> > > dynamic configuration.
> > >
> > > Olivier
> > >
> > > -----Original Message-----
> > > From: Keith W <ke...@gmail.com>
> > > Sent: vendredi 22 juin 2018 10:14
> > > To: users@qpid.apache.org
> > > Subject: Re: [Broker-J 7.0.3] Memory configuration store and
> > > messages recovery
> > >
> > > Hello Olivier
> > >
> > > Let me say first that the memory store's primary use-case is to aid
> > > development and testing of Broker-J, by speeding ups the test cycle.
> > > It also has utility when using embedding Broker-J in the unit tests
> > > of another project and you want to discard all messaging state
> > > between
> > tests.
> > >
> > > You are correct in your analysis.   Queues (like all other objects)
> > > are allocated a random UUID on creation.  This UUID is stored in the
> > > configuration store together with the queue's name and all other
> > > attributes.  Within the message store, the message instance records
> > > refer to the queue's UUID rather than its name.  So if your set-up
> > > is a persistent message store and volatile configuration store, yes,
> > > you will lose messages.  This is expected.  The recovery phase of
> > > startup will clear the message store of the orphans.
> > >
> > > What problem are you actually trying to solve?
> > >
> > > Keith.
> > >
> > >
> > >
> > > On 22 June 2018 at 08:54, VERMEULEN Olivier
> > > <Ol...@murex.com>
> > > wrote:
> > > > Hello,
> > > >
> > > > I was running some tests on a Java Broker with 'Memory'
> > > > configuration
> > > store and 'DERBY' message store.
> > > > Is there a way with this configuration to recover the messages
> > > > stored in
> > > DB upon a broker restart?
> > > > Because it looks like the messages are mapped to the queue UUID
> > > > and that
> > > they are discarded directly during the broker startup if the queue
> > > with the specified UUID is not found.
> > > > And obviously since I'm using a 'Memory' configuration store, the
> > > > queue
> > > has not been recreated yet when the broker starts...
> > > > Have I missed something?
> > > >
> > > > Thanks,
> > > > Olivier
> > > > *******************************
> > > >
> > > > This e-mail contains information for the intended recipient only.
> > > > It may
> > > contain proprietary material or confidential information. If you are
> > > not the intended recipient you are not authorised to distribute,
> > > copy or use this e-mail or any attachment to it. Murex cannot
> > > guarantee that it is virus free and accepts no responsibility for
> > > any loss or damage arising from its use. If you have received this
> > > e-mail in error please notify immediately the sender and delete the
> > > original email received, any attachments and all copies from your
> system.
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> > > additional commands, e-mail: users-help@qpid.apache.org
> > >
> > > *******************************
> > >
> > > This e-mail contains information for the intended recipient only. It
> > > may contain proprietary material or confidential information. If you
> > > are not the intended recipient you are not authorised to distribute,
> > > copy or use this e-mail or any attachment to it. Murex cannot
> > > guarantee that it is virus free and accepts no responsibility for
> > > any loss or damage arising from its use. If you have received this
> > > e-mail in error please notify immediately the sender and delete the
> > > original email received, any attachments and all copies from your
> system.
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> > > additional commands, e-mail: users-help@qpid.apache.org
> > >
> > >
> > *******************************
> >
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorised to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for any
> > loss or damage arising from its use. If you have received this e-mail
> > in error please notify immediately the sender and delete the original
> > email received, any attachments and all copies from your system.
> >
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>

Re: Dispatch Config Store (was: Re: [Broker-J 7.0.3] Memory configuration store and messages recovery)

[Ted Ross <tr...@redhat.com>]

On Mon, Jun 25, 2018 at 4:19 AM, Rob Godfrey <ro...@gmail.com> wrote:
> Hi Olivier,
>
> On Mon, 25 Jun 2018 at 09:40, VERMEULEN Olivier <Ol...@murex.com>
> wrote:
>
>> Ok so I must keep a persistent config store for the Broker.
>> But if I'm not mistaken the Dispacth-Router only supports a non-persistent
>> config store.
>> So are there any plans to implement new config stores for the
>> dispatch-router?
>>
>>
> Updated the subject in the hopes that someone more familiar with the
> current and future capabilities of Dispatch can answer your questions.
>
> -- Rob

Thanks Rob,

Olivier is correct.  The configuration for Dispatch can be fully
captured in a configuration file.  However, changes made to the
configuration via the management protocol are not persistent.  The
router has no means for generating a configuration file based on its
current configuration.

I would be interested in gathering requirements for persistent
configuration for Qpid Dispatch Router.  This is something that I've
given a bit of thought to.  It should be noted that the Enmasse
project (Messaging as a Service in Kubernetes/OpenShift) addresses
this in a limited way.

I believe a persistent configuration solution for Qpid Dispatch Router
should provide the following:

  - Synchronization of configuration across a set of routers in a network;
  - Multi-tenant isolation (i.e. configuration access/storage
separated by tenant/vhost);

-Ted

>
>
>> Thanks,
>> Olivier
>>
>> -----Original Message-----
>> From: Rob Godfrey <ro...@gmail.com>
>> Sent: vendredi 22 juin 2018 13:20
>> To: users@qpid.apache.org
>> Cc: Keith Wall <ke...@gmail.com>
>> Subject: Re: [Broker-J 7.0.3] Memory configuration store and messages
>> recovery
>>
>> Sure - but at that point you really need to provide the vhost config
>> before the vhost itself starts up... would providing all the config
>> (including the queue UUIDs) in a virtualHostInitialConfiguration provided
>> to the VirtualHostNode on creation help I wonder...
>>
>> -- Rob
>>
>> On Fri, 22 Jun 2018 at 12:14, VERMEULEN Olivier <
>> Olivier.VERMEULEN@murex.com>
>> wrote:
>>
>> > Hello Rob,
>> >
>> > The main problem is to keep my version of the config in sync with the
>> > one of each broker.
>> > And knowing that all our brokers have the same config the added
>> > complexity seems a bit overkill.
>> > Especially when you start thinking about broker restart, scale up,
>> > scale down...
>> >
>> > Olivier
>> >
>> > -----Original Message-----
>> > From: Rob Godfrey <ro...@gmail.com>
>> > Sent: vendredi 22 juin 2018 11:27
>> > To: users@qpid.apache.org
>> > Cc: Keith Wall <ke...@gmail.com>
>> > Subject: Re: [Broker-J 7.0.3] Memory configuration store and messages
>> > recovery
>> >
>> > In general, in order to be able to recover the queues from a message
>> > store, the broker needs to know details of the queue - not only its
>> > name, but also the type of the queue (standard, priority, LVQ, etc) as
>> > well as other queue properties... this is precisely the data which is
>> > stored in the configuration store, I don't think there's really any
>> > way around that problem.  What sort of problems are you experiencing
>> > by using a non-Memory configuration store in your setup?
>> >
>> > -- Rob
>> >
>> > On Fri, 22 Jun 2018 at 10:47, VERMEULEN Olivier <
>> > Olivier.VERMEULEN@murex.com>
>> > wrote:
>> >
>> > > Hello Keith,
>> > >
>> > > Thanks for the quick answer.
>> > > Our target is a cluster of brokers and dispatch-routers.
>> > > To configure it we created a management component that centralizes
>> > > the configuration of all the Qpid components.
>> > > With this approach, having the broker store its own configuration is
>> > > redundant with what our management component does and causes some
>> > problems.
>> > > That's why I was looking into the memory configuration store which
>> > > would have made the broker behave like the dispatch-router regarding
>> > > dynamic configuration.
>> > >
>> > > Olivier
>> > >
>> > > -----Original Message-----
>> > > From: Keith W <ke...@gmail.com>
>> > > Sent: vendredi 22 juin 2018 10:14
>> > > To: users@qpid.apache.org
>> > > Subject: Re: [Broker-J 7.0.3] Memory configuration store and
>> > > messages recovery
>> > >
>> > > Hello Olivier
>> > >
>> > > Let me say first that the memory store's primary use-case is to aid
>> > > development and testing of Broker-J, by speeding ups the test cycle.
>> > > It also has utility when using embedding Broker-J in the unit tests
>> > > of another project and you want to discard all messaging state
>> > > between
>> > tests.
>> > >
>> > > You are correct in your analysis.   Queues (like all other objects)
>> > > are allocated a random UUID on creation.  This UUID is stored in the
>> > > configuration store together with the queue's name and all other
>> > > attributes.  Within the message store, the message instance records
>> > > refer to the queue's UUID rather than its name.  So if your set-up
>> > > is a persistent message store and volatile configuration store, yes,
>> > > you will lose messages.  This is expected.  The recovery phase of
>> > > startup will clear the message store of the orphans.
>> > >
>> > > What problem are you actually trying to solve?
>> > >
>> > > Keith.
>> > >
>> > >
>> > >
>> > > On 22 June 2018 at 08:54, VERMEULEN Olivier
>> > > <Ol...@murex.com>
>> > > wrote:
>> > > > Hello,
>> > > >
>> > > > I was running some tests on a Java Broker with 'Memory'
>> > > > configuration
>> > > store and 'DERBY' message store.
>> > > > Is there a way with this configuration to recover the messages
>> > > > stored in
>> > > DB upon a broker restart?
>> > > > Because it looks like the messages are mapped to the queue UUID
>> > > > and that
>> > > they are discarded directly during the broker startup if the queue
>> > > with the specified UUID is not found.
>> > > > And obviously since I'm using a 'Memory' configuration store, the
>> > > > queue
>> > > has not been recreated yet when the broker starts...
>> > > > Have I missed something?
>> > > >
>> > > > Thanks,
>> > > > Olivier
>> > > > *******************************
>> > > >
>> > > > This e-mail contains information for the intended recipient only.
>> > > > It may
>> > > contain proprietary material or confidential information. If you are
>> > > not the intended recipient you are not authorised to distribute,
>> > > copy or use this e-mail or any attachment to it. Murex cannot
>> > > guarantee that it is virus free and accepts no responsibility for
>> > > any loss or damage arising from its use. If you have received this
>> > > e-mail in error please notify immediately the sender and delete the
>> > > original email received, any attachments and all copies from your
>> system.
>> > >
>> > > --------------------------------------------------------------------
>> > > - To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
>> > > additional commands, e-mail: users-help@qpid.apache.org
>> > >
>> > > *******************************
>> > >
>> > > This e-mail contains information for the intended recipient only. It
>> > > may contain proprietary material or confidential information. If you
>> > > are not the intended recipient you are not authorised to distribute,
>> > > copy or use this e-mail or any attachment to it. Murex cannot
>> > > guarantee that it is virus free and accepts no responsibility for
>> > > any loss or damage arising from its use. If you have received this
>> > > e-mail in error please notify immediately the sender and delete the
>> > > original email received, any attachments and all copies from your
>> system.
>> > >
>> > > --------------------------------------------------------------------
>> > > - To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
>> > > additional commands, e-mail: users-help@qpid.apache.org
>> > >
>> > >
>> > *******************************
>> >
>> > This e-mail contains information for the intended recipient only. It
>> > may contain proprietary material or confidential information. If you
>> > are not the intended recipient you are not authorised to distribute,
>> > copy or use this e-mail or any attachment to it. Murex cannot
>> > guarantee that it is virus free and accepts no responsibility for any
>> > loss or damage arising from its use. If you have received this e-mail
>> > in error please notify immediately the sender and delete the original
>> > email received, any attachments and all copies from your system.
>> >
>> *******************************
>>
>> This e-mail contains information for the intended recipient only. It may
>> contain proprietary material or confidential information. If you are not
>> the intended recipient you are not authorised to distribute, copy or use
>> this e-mail or any attachment to it. Murex cannot guarantee that it is
>> virus free and accepts no responsibility for any loss or damage arising
>> from its use. If you have received this e-mail in error please notify
>> immediately the sender and delete the original email received, any
>> attachments and all copies from your system.
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org