[Broker-J] Claim-check pattern and MessageDeleteListener

[VERMEULEN Olivier <Ol...@murex.com>]

Hello,

We're trying to implement the claim-check pattern on top of our broker + dispatch-router topology.
So we have a blob store and the idea is for each message above 10KB to store the payload in it and to only keep an ID in the message.
The tricky part is the delete of the payload in the blob store, especially since some of our use cases are using multicast...
When looking at the Broker-J source code I found this MessageDeleteListener which would be really helpful for this use case:
https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2ae7a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/store/MessageStore.java#L84

My first question would be, is it ok to use it?
Second, so far the only way I found to register a new listener is to define my own VirtualHost, is there an easier way ?

Thanks,
Olivier
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

Re: [Broker-J] Claim-check pattern and MessageDeleteListener

[Rob Godfrey <ro...@gmail.com>]

So - the more I think about this, the more I think a custom store might be
the best way to approach this. For safety I think the approach I would take
is that on asking for the message to be removed from the DB, I would write
the blob id into a separate table (in the same transaction as the delete).
Then we can have a separate thread that is iterating over the table of
"blobs to be deleted", and for every row it finds it can call the blob
store to delete, and only remove the row from the "blobs to be deleted"
table when it gets confirmation that the blob has, indeed, been removed
from the external store.  In this way we can separate the deletion of the
message data from interaction with the external blob store (so, e.g. if the
blob store is unreachable it doesn't delay anything else), and on restart
the store can just start deleting the blobs by id of anything that is left
in the table.

-- Rob

On Wed, 11 Mar 2020 at 10:33, VERMEULEN Olivier <Ol...@murex.com>
wrote:

> from many queues on the same broker
> ________________________________
> From: Rob Godfrey <ro...@gmail.com>
> Sent: Wednesday, March 11, 2020 10:01:43 AM
> To: users@qpid.apache.org <us...@qpid.apache.org>
> Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
>
> On Wed, 11 Mar 2020 at 09:46, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Why don't we just trigger the existing MessageDeleteListeners at the
> > beginning of the remove method instead of doing it at the end ?
> >
> >
> https://github.com/apache/qpid-broker-j/blob/master/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670
> >
> >
> https://github.com/apache/qpid-broker-j/blob/master/bdbstore/src/main/java/org/apache/qpid/server/store/berkeleydb/AbstractBDBMessageStore.java#L1199
> >
> > This way we handle both the normal use case and the recovery one and the
> > headers will still be accessible from the StoredMessage instance (need to
> > add a getter in StorableMessageMetaData).
> > I can provide a pull-request if you want.
> >
>
> I still don't think this will work well for getting the header data - you
> really need to be operating on the AbstractServerMessage not the
> StoredMessage if you want to access the (protocol dependent implementation)
> headers / content.
>
> I think the current approach (and others that I have briefly thought about)
> will also fall down in the case of "orphaned" content bodies that are being
> deleted on broker restart (basically because the delete of the content -
> deliberately - not in a transaction with the delete of the per-queue
> enqueue records, an abrupt shutdown of the broker can leave message data in
> the store which is no longer referenced by any queues.  In this case the
> recovery code identifies the orphans and deletes them - but this is a
> database level operation rather than on the message model - and you can't
> assume the headers will be available there either (they are stored in a
> separate table).
>
> In a previous mail, you talked about "multicast" - is the expectation that
> the underlying blob may be being referenced from many queues on the same
> broker? or on queues on many different brokers?
>
> -- Rob
>
>
> >
> > WDYT?
> > Olivier
> >
> > From: VERMEULEN Olivier <Ol...@murex.com>
> > Sent: mardi 10 mars 2020 20:08
> > To: users@qpid.apache.org
> > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> >
> > The idea is to avoid any performance impact on the messaging when big
> > messages arrive.
> > So whenever a producer tries to send a big message (more than 10KB for
> us)
> > we put the payload in the blob store with a unique ID and we pass this ID
> > in the message instead of the original payload.
> > When a consumer receives such a message we then retrieve the payload from
> > the blob store.
> > The problem is that we can't expect the consumer to handle the cleanup of
> > the blob store since it might not be the only consumer receiving this
> > message (multicast for example or even a message being routed to 2 queues
> > through a topic).
> > The only one that really knows when the payload can be removed from the
> > blob store is the broker.
> > That's why I was looking for a hook that would be called whenever a
> > message is or will be deleted from the store (this includes the recovery
> > process).
> > After that whether the hook is at the queue or virtual host level doesn't
> > really matter...
> >
> > Olivier
> > ________________________________
> > From: Rob Godfrey <rob.j.godfrey@gmail.com<mailto:
> rob.j.godfrey@gmail.com
> > >>
> > Sent: Tuesday, March 10, 2020 6:26:39 PM
> > To: users@qpid.apache.org<ma...@qpid.apache.org> <
> > users@qpid.apache.org<ma...@qpid.apache.org>>
> > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> >
> > Yeah - I was wondering if it might be better to add hooks into the point
> at
> > which remove() is called (i.e.
> >
> >
> https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/message/AbstractServerMessageImpl.java#L127
> > ),
> > because a listener on the here could actually work with the message
> object
> > where the headers are available... However this would miss the other
> point
> > at which messages are removed from the store - in the store recovery
> > process (e.g.
> >
> >
> https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/virtualhost/AsynchronousMessageStoreRecoverer.java#L227
> > ).
> > Unfortunately at this point we wouldn't have access to the headers.
> >
> > Can you explain a bit more how your blob storage is supposed to work.
> > Perhaps there is another way that we can add a hook that would help here
> > (e.g. on a per queue basis have some sort of "dequeue" trigger which is
> > passed the message it is about to be permanently removed from the queue).
> >
> > -- Rob
> >
> > On Tue, 10 Mar 2020 at 17:59, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> > wrote:
> >
> > > The cast works fine in my case but the problem is that everything is
> > NULL,
> > > including the meta-data.
> > > Indeed the message is cleaned before the listeners are called...
> > >
> > >
> >
> https://github.com/apache/qpid-broker-j/blob/7193e26f6fad83f5ce81fd9ef855edb0fd5594ea/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670
> > >
> > > Olivier
> > >
> > > -----Original Message-----
> > > From: Rob Godfrey <rob.j.godfrey@gmail.com<mailto:
> > rob.j.godfrey@gmail.com>>
> > > Sent: lundi 9 mars 2020 09:58
> > > To: users@qpid.apache.org<ma...@qpid.apache.org>
> > > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> > >
> > > On Mon, 9 Mar 2020 at 09:36, VERMEULEN Olivier <
> > > Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> > > wrote:
> > >
> > > > Hello Rob,
> > > >
> > > > "Are you looking to have Broker-J trigger the deletion on the blob at
> > > > the point at which it believes it no longer needs the message?"
> > > > Yes that's exactly what I'm trying to do.
> > > > Note that I managed to do it by creating my own vhost, just wanted to
> > > > check if there was a less intrusive way of doing it...
> > > > I agree that we shouldn't be able to modify the message but today I
> > > > can't even read it easily.
> > > > What I'm interested in are the headers but from a StoredMessage
> > > > instance I have to go through internal classes and package-protected
> > > > classes to get
> > > > them:
> > > > ((InternalMessageMetaData) storedMessage.getMetaData()).getHeader();
> > > > getHeader being package-protected here...
> > > >
> > >
> > > Note that this will only work if the message is an "Internal" message
> > (i.e.
> > > the casting here would fail if the message were an AMQP 0-9, AMQP 0-10
> or
> > > AMQP 1.0 message sent over the wire).  The store API is basically built
> > to
> > > be as unaware of the form of the message as possible, and the api here
> is
> > > basically designed to allow the store to clean up a message which was
> > > potentially stored in multiple queues, but which has now been removed
> > from
> > > all of them.  As such it's not really designed to make it easy for you
> to
> > > access the headers (or content - since the "content" will include
> > structure
> > > for 1.0 messages).
> > >
> > >
> > > > Did I miss something?
> > > >
> > >
> > > No - I think to do this safely we'd have to expose some new API here.
> > >
> > > -- Rob
> > >
> > >
> > > >
> > > > Thanks,
> > > > Olivier
> > > >
> > > > -----Original Message-----
> > > > From: Rob Godfrey <rob.j.godfrey@gmail.com<mailto:
> > rob.j.godfrey@gmail.com>>
> > > > Sent: vendredi 6 mars 2020 17:54
> > > > To: users@qpid.apache.org<ma...@qpid.apache.org>
> > > > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> > > >
> > > > Hi Olivier
> > > >
> > > > I'm not entirely sure what your exact idea here is - are you looking
> > > > to have Broker-J trigger the deletion on the blob at the point at
> > > > which it believes it no longer needs the message?  Or maybe to update
> > > > the blob store to make clear that the broker no longer needs to
> retain
> > > > a copy of the message?
> > > >
> > > > Clearly the API is not designed for allowing extensions here - and I
> > > > think if we were to try to make it an extension / interception point
> > > > then we would have to be careful in limiting what could be done here
> > > > (e.g. offering only some read-only view of the message).
> > > >
> > > > Without such an interception extension mechanism, you are correct I
> > > > think that you would need to create a new vhost type (potentially
> > > > using a new store type of your own implementation)
> > > >
> > > > -- Rob
> > > >
> > > > On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier <
> > > > Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > We're trying to implement the claim-check pattern on top of our
> > > > > broker
> > > > > + dispatch-router topology.
> > > > > So we have a blob store and the idea is for each message above 10KB
> > > > > to store the payload in it and to only keep an ID in the message.
> > > > > The tricky part is the delete of the payload in the blob store,
> > > > > especially since some of our use cases are using multicast...
> > > > > When looking at the Broker-J source code I found this
> > > > > MessageDeleteListener which would be really helpful for this use
> > case:
> > > > >
> > > > >
> https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2a
> > > > > e7
> > > > >
> a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/sto
> > > > > re
> > > > > /MessageStore.java#L84
> > > > >
> > > > > My first question would be, is it ok to use it?
> > > > > Second, so far the only way I found to register a new listener is
> to
> > > > > define my own VirtualHost, is there an easier way ?
> > > > >
> > > > > Thanks,
> > > > > Olivier
> > > > > *******************************
> > > > > This e-mail contains information for the intended recipient only.
> It
> > > > > may contain proprietary material or confidential information. If
> you
> > > > > are not the intended recipient you are not authorized to
> distribute,
> > > > > copy or use this e-mail or any attachment to it. Murex cannot
> > > > > guarantee that it is virus free and accepts no responsibility for
> > > > > any loss or damage arising from its use. If you have received this
> > > > > e-mail in error please notify immediately the sender and delete the
> > > > > original email received, any attachments and all copies from your
> > > system.
> > > > >
> > > > *******************************
> > > > This e-mail contains information for the intended recipient only. It
> > > > may contain proprietary material or confidential information. If you
> > > > are not the intended recipient you are not authorized to distribute,
> > > > copy or use this e-mail or any attachment to it. Murex cannot
> > > > guarantee that it is virus free and accepts no responsibility for any
> > > > loss or damage arising from its use. If you have received this e-mail
> > > > in error please notify immediately the sender and delete the original
> > > > email received, any attachments and all copies from your system.
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org<mailto:
> > users-unsubscribe@qpid.apache.org> For
> > > > additional commands, e-mail: users-help@qpid.apache.org<mailto:
> > users-help@qpid.apache.org>
> > > >
> > > >
> > > *******************************
> > > This e-mail contains information for the intended recipient only. It
> may
> > > contain proprietary material or confidential information. If you are
> not
> > > the intended recipient you are not authorized to distribute, copy or
> use
> > > this e-mail or any attachment to it. Murex cannot guarantee that it is
> > > virus free and accepts no responsibility for any loss or damage arising
> > > from its use. If you have received this e-mail in error please notify
> > > immediately the sender and delete the original email received, any
> > > attachments and all copies from your system.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org<mailto:
> > users-unsubscribe@qpid.apache.org>
> > > For additional commands, e-mail: users-help@qpid.apache.org<mailto:
> > users-help@qpid.apache.org>
> > >
> > >
> > *******************************
> > This e-mail contains information for the intended recipient only. It may
> > contain proprietary material or confidential information. If you are not
> > the intended recipient you are not authorized to distribute, copy or use
> > this e-mail or any attachment to it. Murex cannot guarantee that it is
> > virus free and accepts no responsibility for any loss or damage arising
> > from its use. If you have received this e-mail in error please notify
> > immediately the sender and delete the original email received, any
> > attachments and all copies from your system.
> >
> *******************************
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>

Re: [Broker-J] Claim-check pattern and MessageDeleteListener

[VERMEULEN Olivier <Ol...@murex.com>]

from many queues on the same broker
________________________________
From: Rob Godfrey <ro...@gmail.com>
Sent: Wednesday, March 11, 2020 10:01:43 AM
To: users@qpid.apache.org <us...@qpid.apache.org>
Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener

On Wed, 11 Mar 2020 at 09:46, VERMEULEN Olivier <Ol...@murex.com>
wrote:

> Why don't we just trigger the existing MessageDeleteListeners at the
> beginning of the remove method instead of doing it at the end ?
>
> https://github.com/apache/qpid-broker-j/blob/master/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670
>
> https://github.com/apache/qpid-broker-j/blob/master/bdbstore/src/main/java/org/apache/qpid/server/store/berkeleydb/AbstractBDBMessageStore.java#L1199
>
> This way we handle both the normal use case and the recovery one and the
> headers will still be accessible from the StoredMessage instance (need to
> add a getter in StorableMessageMetaData).
> I can provide a pull-request if you want.
>

I still don't think this will work well for getting the header data - you
really need to be operating on the AbstractServerMessage not the
StoredMessage if you want to access the (protocol dependent implementation)
headers / content.

I think the current approach (and others that I have briefly thought about)
will also fall down in the case of "orphaned" content bodies that are being
deleted on broker restart (basically because the delete of the content -
deliberately - not in a transaction with the delete of the per-queue
enqueue records, an abrupt shutdown of the broker can leave message data in
the store which is no longer referenced by any queues.  In this case the
recovery code identifies the orphans and deletes them - but this is a
database level operation rather than on the message model - and you can't
assume the headers will be available there either (they are stored in a
separate table).

In a previous mail, you talked about "multicast" - is the expectation that
the underlying blob may be being referenced from many queues on the same
broker? or on queues on many different brokers?

-- Rob


>
> WDYT?
> Olivier
>
> From: VERMEULEN Olivier <Ol...@murex.com>
> Sent: mardi 10 mars 2020 20:08
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
>
> The idea is to avoid any performance impact on the messaging when big
> messages arrive.
> So whenever a producer tries to send a big message (more than 10KB for us)
> we put the payload in the blob store with a unique ID and we pass this ID
> in the message instead of the original payload.
> When a consumer receives such a message we then retrieve the payload from
> the blob store.
> The problem is that we can't expect the consumer to handle the cleanup of
> the blob store since it might not be the only consumer receiving this
> message (multicast for example or even a message being routed to 2 queues
> through a topic).
> The only one that really knows when the payload can be removed from the
> blob store is the broker.
> That's why I was looking for a hook that would be called whenever a
> message is or will be deleted from the store (this includes the recovery
> process).
> After that whether the hook is at the queue or virtual host level doesn't
> really matter...
>
> Olivier
> ________________________________
> From: Rob Godfrey <rob.j.godfrey@gmail.com<mailto:rob.j.godfrey@gmail.com
> >>
> Sent: Tuesday, March 10, 2020 6:26:39 PM
> To: users@qpid.apache.org<ma...@qpid.apache.org> <
> users@qpid.apache.org<ma...@qpid.apache.org>>
> Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
>
> Yeah - I was wondering if it might be better to add hooks into the point at
> which remove() is called (i.e.
>
> https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/message/AbstractServerMessageImpl.java#L127
> ),
> because a listener on the here could actually work with the message object
> where the headers are available... However this would miss the other point
> at which messages are removed from the store - in the store recovery
> process (e.g.
>
> https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/virtualhost/AsynchronousMessageStoreRecoverer.java#L227
> ).
> Unfortunately at this point we wouldn't have access to the headers.
>
> Can you explain a bit more how your blob storage is supposed to work.
> Perhaps there is another way that we can add a hook that would help here
> (e.g. on a per queue basis have some sort of "dequeue" trigger which is
> passed the message it is about to be permanently removed from the queue).
>
> -- Rob
>
> On Tue, 10 Mar 2020 at 17:59, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> wrote:
>
> > The cast works fine in my case but the problem is that everything is
> NULL,
> > including the meta-data.
> > Indeed the message is cleaned before the listeners are called...
> >
> >
> https://github.com/apache/qpid-broker-j/blob/7193e26f6fad83f5ce81fd9ef855edb0fd5594ea/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670
> >
> > Olivier
> >
> > -----Original Message-----
> > From: Rob Godfrey <rob.j.godfrey@gmail.com<mailto:
> rob.j.godfrey@gmail.com>>
> > Sent: lundi 9 mars 2020 09:58
> > To: users@qpid.apache.org<ma...@qpid.apache.org>
> > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> >
> > On Mon, 9 Mar 2020 at 09:36, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> > wrote:
> >
> > > Hello Rob,
> > >
> > > "Are you looking to have Broker-J trigger the deletion on the blob at
> > > the point at which it believes it no longer needs the message?"
> > > Yes that's exactly what I'm trying to do.
> > > Note that I managed to do it by creating my own vhost, just wanted to
> > > check if there was a less intrusive way of doing it...
> > > I agree that we shouldn't be able to modify the message but today I
> > > can't even read it easily.
> > > What I'm interested in are the headers but from a StoredMessage
> > > instance I have to go through internal classes and package-protected
> > > classes to get
> > > them:
> > > ((InternalMessageMetaData) storedMessage.getMetaData()).getHeader();
> > > getHeader being package-protected here...
> > >
> >
> > Note that this will only work if the message is an "Internal" message
> (i.e.
> > the casting here would fail if the message were an AMQP 0-9, AMQP 0-10 or
> > AMQP 1.0 message sent over the wire).  The store API is basically built
> to
> > be as unaware of the form of the message as possible, and the api here is
> > basically designed to allow the store to clean up a message which was
> > potentially stored in multiple queues, but which has now been removed
> from
> > all of them.  As such it's not really designed to make it easy for you to
> > access the headers (or content - since the "content" will include
> structure
> > for 1.0 messages).
> >
> >
> > > Did I miss something?
> > >
> >
> > No - I think to do this safely we'd have to expose some new API here.
> >
> > -- Rob
> >
> >
> > >
> > > Thanks,
> > > Olivier
> > >
> > > -----Original Message-----
> > > From: Rob Godfrey <rob.j.godfrey@gmail.com<mailto:
> rob.j.godfrey@gmail.com>>
> > > Sent: vendredi 6 mars 2020 17:54
> > > To: users@qpid.apache.org<ma...@qpid.apache.org>
> > > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> > >
> > > Hi Olivier
> > >
> > > I'm not entirely sure what your exact idea here is - are you looking
> > > to have Broker-J trigger the deletion on the blob at the point at
> > > which it believes it no longer needs the message?  Or maybe to update
> > > the blob store to make clear that the broker no longer needs to retain
> > > a copy of the message?
> > >
> > > Clearly the API is not designed for allowing extensions here - and I
> > > think if we were to try to make it an extension / interception point
> > > then we would have to be careful in limiting what could be done here
> > > (e.g. offering only some read-only view of the message).
> > >
> > > Without such an interception extension mechanism, you are correct I
> > > think that you would need to create a new vhost type (potentially
> > > using a new store type of your own implementation)
> > >
> > > -- Rob
> > >
> > > On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier <
> > > Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > We're trying to implement the claim-check pattern on top of our
> > > > broker
> > > > + dispatch-router topology.
> > > > So we have a blob store and the idea is for each message above 10KB
> > > > to store the payload in it and to only keep an ID in the message.
> > > > The tricky part is the delete of the payload in the blob store,
> > > > especially since some of our use cases are using multicast...
> > > > When looking at the Broker-J source code I found this
> > > > MessageDeleteListener which would be really helpful for this use
> case:
> > > >
> > > > https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2a
> > > > e7
> > > > a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/sto
> > > > re
> > > > /MessageStore.java#L84
> > > >
> > > > My first question would be, is it ok to use it?
> > > > Second, so far the only way I found to register a new listener is to
> > > > define my own VirtualHost, is there an easier way ?
> > > >
> > > > Thanks,
> > > > Olivier
> > > > *******************************
> > > > This e-mail contains information for the intended recipient only. It
> > > > may contain proprietary material or confidential information. If you
> > > > are not the intended recipient you are not authorized to distribute,
> > > > copy or use this e-mail or any attachment to it. Murex cannot
> > > > guarantee that it is virus free and accepts no responsibility for
> > > > any loss or damage arising from its use. If you have received this
> > > > e-mail in error please notify immediately the sender and delete the
> > > > original email received, any attachments and all copies from your
> > system.
> > > >
> > > *******************************
> > > This e-mail contains information for the intended recipient only. It
> > > may contain proprietary material or confidential information. If you
> > > are not the intended recipient you are not authorized to distribute,
> > > copy or use this e-mail or any attachment to it. Murex cannot
> > > guarantee that it is virus free and accepts no responsibility for any
> > > loss or damage arising from its use. If you have received this e-mail
> > > in error please notify immediately the sender and delete the original
> > > email received, any attachments and all copies from your system.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org<mailto:
> users-unsubscribe@qpid.apache.org> For
> > > additional commands, e-mail: users-help@qpid.apache.org<mailto:
> users-help@qpid.apache.org>
> > >
> > >
> > *******************************
> > This e-mail contains information for the intended recipient only. It may
> > contain proprietary material or confidential information. If you are not
> > the intended recipient you are not authorized to distribute, copy or use
> > this e-mail or any attachment to it. Murex cannot guarantee that it is
> > virus free and accepts no responsibility for any loss or damage arising
> > from its use. If you have received this e-mail in error please notify
> > immediately the sender and delete the original email received, any
> > attachments and all copies from your system.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org<mailto:
> users-unsubscribe@qpid.apache.org>
> > For additional commands, e-mail: users-help@qpid.apache.org<mailto:
> users-help@qpid.apache.org>
> >
> >
> *******************************
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

Re: [Broker-J] Claim-check pattern and MessageDeleteListener

[Rob Godfrey <ro...@gmail.com>]

On Wed, 11 Mar 2020 at 09:46, VERMEULEN Olivier <Ol...@murex.com>
wrote:

> Why don't we just trigger the existing MessageDeleteListeners at the
> beginning of the remove method instead of doing it at the end ?
>
> https://github.com/apache/qpid-broker-j/blob/master/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670
>
> https://github.com/apache/qpid-broker-j/blob/master/bdbstore/src/main/java/org/apache/qpid/server/store/berkeleydb/AbstractBDBMessageStore.java#L1199
>
> This way we handle both the normal use case and the recovery one and the
> headers will still be accessible from the StoredMessage instance (need to
> add a getter in StorableMessageMetaData).
> I can provide a pull-request if you want.
>

I still don't think this will work well for getting the header data - you
really need to be operating on the AbstractServerMessage not the
StoredMessage if you want to access the (protocol dependent implementation)
headers / content.

I think the current approach (and others that I have briefly thought about)
will also fall down in the case of "orphaned" content bodies that are being
deleted on broker restart (basically because the delete of the content -
deliberately - not in a transaction with the delete of the per-queue
enqueue records, an abrupt shutdown of the broker can leave message data in
the store which is no longer referenced by any queues.  In this case the
recovery code identifies the orphans and deletes them - but this is a
database level operation rather than on the message model - and you can't
assume the headers will be available there either (they are stored in a
separate table).

In a previous mail, you talked about "multicast" - is the expectation that
the underlying blob may be being referenced from many queues on the same
broker? or on queues on many different brokers?

-- Rob


>
> WDYT?
> Olivier
>
> From: VERMEULEN Olivier <Ol...@murex.com>
> Sent: mardi 10 mars 2020 20:08
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
>
> The idea is to avoid any performance impact on the messaging when big
> messages arrive.
> So whenever a producer tries to send a big message (more than 10KB for us)
> we put the payload in the blob store with a unique ID and we pass this ID
> in the message instead of the original payload.
> When a consumer receives such a message we then retrieve the payload from
> the blob store.
> The problem is that we can't expect the consumer to handle the cleanup of
> the blob store since it might not be the only consumer receiving this
> message (multicast for example or even a message being routed to 2 queues
> through a topic).
> The only one that really knows when the payload can be removed from the
> blob store is the broker.
> That's why I was looking for a hook that would be called whenever a
> message is or will be deleted from the store (this includes the recovery
> process).
> After that whether the hook is at the queue or virtual host level doesn't
> really matter...
>
> Olivier
> ________________________________
> From: Rob Godfrey <rob.j.godfrey@gmail.com<mailto:rob.j.godfrey@gmail.com
> >>
> Sent: Tuesday, March 10, 2020 6:26:39 PM
> To: users@qpid.apache.org<ma...@qpid.apache.org> <
> users@qpid.apache.org<ma...@qpid.apache.org>>
> Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
>
> Yeah - I was wondering if it might be better to add hooks into the point at
> which remove() is called (i.e.
>
> https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/message/AbstractServerMessageImpl.java#L127
> ),
> because a listener on the here could actually work with the message object
> where the headers are available... However this would miss the other point
> at which messages are removed from the store - in the store recovery
> process (e.g.
>
> https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/virtualhost/AsynchronousMessageStoreRecoverer.java#L227
> ).
> Unfortunately at this point we wouldn't have access to the headers.
>
> Can you explain a bit more how your blob storage is supposed to work.
> Perhaps there is another way that we can add a hook that would help here
> (e.g. on a per queue basis have some sort of "dequeue" trigger which is
> passed the message it is about to be permanently removed from the queue).
>
> -- Rob
>
> On Tue, 10 Mar 2020 at 17:59, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> wrote:
>
> > The cast works fine in my case but the problem is that everything is
> NULL,
> > including the meta-data.
> > Indeed the message is cleaned before the listeners are called...
> >
> >
> https://github.com/apache/qpid-broker-j/blob/7193e26f6fad83f5ce81fd9ef855edb0fd5594ea/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670
> >
> > Olivier
> >
> > -----Original Message-----
> > From: Rob Godfrey <rob.j.godfrey@gmail.com<mailto:
> rob.j.godfrey@gmail.com>>
> > Sent: lundi 9 mars 2020 09:58
> > To: users@qpid.apache.org<ma...@qpid.apache.org>
> > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> >
> > On Mon, 9 Mar 2020 at 09:36, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> > wrote:
> >
> > > Hello Rob,
> > >
> > > "Are you looking to have Broker-J trigger the deletion on the blob at
> > > the point at which it believes it no longer needs the message?"
> > > Yes that's exactly what I'm trying to do.
> > > Note that I managed to do it by creating my own vhost, just wanted to
> > > check if there was a less intrusive way of doing it...
> > > I agree that we shouldn't be able to modify the message but today I
> > > can't even read it easily.
> > > What I'm interested in are the headers but from a StoredMessage
> > > instance I have to go through internal classes and package-protected
> > > classes to get
> > > them:
> > > ((InternalMessageMetaData) storedMessage.getMetaData()).getHeader();
> > > getHeader being package-protected here...
> > >
> >
> > Note that this will only work if the message is an "Internal" message
> (i.e.
> > the casting here would fail if the message were an AMQP 0-9, AMQP 0-10 or
> > AMQP 1.0 message sent over the wire).  The store API is basically built
> to
> > be as unaware of the form of the message as possible, and the api here is
> > basically designed to allow the store to clean up a message which was
> > potentially stored in multiple queues, but which has now been removed
> from
> > all of them.  As such it's not really designed to make it easy for you to
> > access the headers (or content - since the "content" will include
> structure
> > for 1.0 messages).
> >
> >
> > > Did I miss something?
> > >
> >
> > No - I think to do this safely we'd have to expose some new API here.
> >
> > -- Rob
> >
> >
> > >
> > > Thanks,
> > > Olivier
> > >
> > > -----Original Message-----
> > > From: Rob Godfrey <rob.j.godfrey@gmail.com<mailto:
> rob.j.godfrey@gmail.com>>
> > > Sent: vendredi 6 mars 2020 17:54
> > > To: users@qpid.apache.org<ma...@qpid.apache.org>
> > > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> > >
> > > Hi Olivier
> > >
> > > I'm not entirely sure what your exact idea here is - are you looking
> > > to have Broker-J trigger the deletion on the blob at the point at
> > > which it believes it no longer needs the message?  Or maybe to update
> > > the blob store to make clear that the broker no longer needs to retain
> > > a copy of the message?
> > >
> > > Clearly the API is not designed for allowing extensions here - and I
> > > think if we were to try to make it an extension / interception point
> > > then we would have to be careful in limiting what could be done here
> > > (e.g. offering only some read-only view of the message).
> > >
> > > Without such an interception extension mechanism, you are correct I
> > > think that you would need to create a new vhost type (potentially
> > > using a new store type of your own implementation)
> > >
> > > -- Rob
> > >
> > > On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier <
> > > Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > We're trying to implement the claim-check pattern on top of our
> > > > broker
> > > > + dispatch-router topology.
> > > > So we have a blob store and the idea is for each message above 10KB
> > > > to store the payload in it and to only keep an ID in the message.
> > > > The tricky part is the delete of the payload in the blob store,
> > > > especially since some of our use cases are using multicast...
> > > > When looking at the Broker-J source code I found this
> > > > MessageDeleteListener which would be really helpful for this use
> case:
> > > >
> > > > https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2a
> > > > e7
> > > > a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/sto
> > > > re
> > > > /MessageStore.java#L84
> > > >
> > > > My first question would be, is it ok to use it?
> > > > Second, so far the only way I found to register a new listener is to
> > > > define my own VirtualHost, is there an easier way ?
> > > >
> > > > Thanks,
> > > > Olivier
> > > > *******************************
> > > > This e-mail contains information for the intended recipient only. It
> > > > may contain proprietary material or confidential information. If you
> > > > are not the intended recipient you are not authorized to distribute,
> > > > copy or use this e-mail or any attachment to it. Murex cannot
> > > > guarantee that it is virus free and accepts no responsibility for
> > > > any loss or damage arising from its use. If you have received this
> > > > e-mail in error please notify immediately the sender and delete the
> > > > original email received, any attachments and all copies from your
> > system.
> > > >
> > > *******************************
> > > This e-mail contains information for the intended recipient only. It
> > > may contain proprietary material or confidential information. If you
> > > are not the intended recipient you are not authorized to distribute,
> > > copy or use this e-mail or any attachment to it. Murex cannot
> > > guarantee that it is virus free and accepts no responsibility for any
> > > loss or damage arising from its use. If you have received this e-mail
> > > in error please notify immediately the sender and delete the original
> > > email received, any attachments and all copies from your system.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org<mailto:
> users-unsubscribe@qpid.apache.org> For
> > > additional commands, e-mail: users-help@qpid.apache.org<mailto:
> users-help@qpid.apache.org>
> > >
> > >
> > *******************************
> > This e-mail contains information for the intended recipient only. It may
> > contain proprietary material or confidential information. If you are not
> > the intended recipient you are not authorized to distribute, copy or use
> > this e-mail or any attachment to it. Murex cannot guarantee that it is
> > virus free and accepts no responsibility for any loss or damage arising
> > from its use. If you have received this e-mail in error please notify
> > immediately the sender and delete the original email received, any
> > attachments and all copies from your system.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org<mailto:
> users-unsubscribe@qpid.apache.org>
> > For additional commands, e-mail: users-help@qpid.apache.org<mailto:
> users-help@qpid.apache.org>
> >
> >
> *******************************
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>

RE: [Broker-J] Claim-check pattern and MessageDeleteListener

[VERMEULEN Olivier <Ol...@murex.com>]

Why don't we just trigger the existing MessageDeleteListeners at the beginning of the remove method instead of doing it at the end ?
https://github.com/apache/qpid-broker-j/blob/master/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670
https://github.com/apache/qpid-broker-j/blob/master/bdbstore/src/main/java/org/apache/qpid/server/store/berkeleydb/AbstractBDBMessageStore.java#L1199

This way we handle both the normal use case and the recovery one and the headers will still be accessible from the StoredMessage instance (need to add a getter in StorableMessageMetaData).
I can provide a pull-request if you want.

WDYT?
Olivier

From: VERMEULEN Olivier <Ol...@murex.com>
Sent: mardi 10 mars 2020 20:08
To: users@qpid.apache.org
Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener

The idea is to avoid any performance impact on the messaging when big messages arrive.
So whenever a producer tries to send a big message (more than 10KB for us) we put the payload in the blob store with a unique ID and we pass this ID in the message instead of the original payload.
When a consumer receives such a message we then retrieve the payload from the blob store.
The problem is that we can't expect the consumer to handle the cleanup of the blob store since it might not be the only consumer receiving this message (multicast for example or even a message being routed to 2 queues through a topic).
The only one that really knows when the payload can be removed from the blob store is the broker.
That's why I was looking for a hook that would be called whenever a message is or will be deleted from the store (this includes the recovery process).
After that whether the hook is at the queue or virtual host level doesn't really matter...

Olivier
________________________________
From: Rob Godfrey <ro...@gmail.com>>
Sent: Tuesday, March 10, 2020 6:26:39 PM
To: users@qpid.apache.org<ma...@qpid.apache.org> <us...@qpid.apache.org>>
Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener

Yeah - I was wondering if it might be better to add hooks into the point at
which remove() is called (i.e.
https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/message/AbstractServerMessageImpl.java#L127
),
because a listener on the here could actually work with the message object
where the headers are available... However this would miss the other point
at which messages are removed from the store - in the store recovery
process (e.g.
https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/virtualhost/AsynchronousMessageStoreRecoverer.java#L227
).
Unfortunately at this point we wouldn't have access to the headers.

Can you explain a bit more how your blob storage is supposed to work.
Perhaps there is another way that we can add a hook that would help here
(e.g. on a per queue basis have some sort of "dequeue" trigger which is
passed the message it is about to be permanently removed from the queue).

-- Rob

On Tue, 10 Mar 2020 at 17:59, VERMEULEN Olivier <Ol...@murex.com>>
wrote:

> The cast works fine in my case but the problem is that everything is NULL,
> including the meta-data.
> Indeed the message is cleaned before the listeners are called...
>
> https://github.com/apache/qpid-broker-j/blob/7193e26f6fad83f5ce81fd9ef855edb0fd5594ea/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670
>
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>>
> Sent: lundi 9 mars 2020 09:58
> To: users@qpid.apache.org<ma...@qpid.apache.org>
> Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
>
> On Mon, 9 Mar 2020 at 09:36, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> wrote:
>
> > Hello Rob,
> >
> > "Are you looking to have Broker-J trigger the deletion on the blob at
> > the point at which it believes it no longer needs the message?"
> > Yes that's exactly what I'm trying to do.
> > Note that I managed to do it by creating my own vhost, just wanted to
> > check if there was a less intrusive way of doing it...
> > I agree that we shouldn't be able to modify the message but today I
> > can't even read it easily.
> > What I'm interested in are the headers but from a StoredMessage
> > instance I have to go through internal classes and package-protected
> > classes to get
> > them:
> > ((InternalMessageMetaData) storedMessage.getMetaData()).getHeader();
> > getHeader being package-protected here...
> >
>
> Note that this will only work if the message is an "Internal" message (i.e.
> the casting here would fail if the message were an AMQP 0-9, AMQP 0-10 or
> AMQP 1.0 message sent over the wire).  The store API is basically built to
> be as unaware of the form of the message as possible, and the api here is
> basically designed to allow the store to clean up a message which was
> potentially stored in multiple queues, but which has now been removed from
> all of them.  As such it's not really designed to make it easy for you to
> access the headers (or content - since the "content" will include structure
> for 1.0 messages).
>
>
> > Did I miss something?
> >
>
> No - I think to do this safely we'd have to expose some new API here.
>
> -- Rob
>
>
> >
> > Thanks,
> > Olivier
> >
> > -----Original Message-----
> > From: Rob Godfrey <ro...@gmail.com>>
> > Sent: vendredi 6 mars 2020 17:54
> > To: users@qpid.apache.org<ma...@qpid.apache.org>
> > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> >
> > Hi Olivier
> >
> > I'm not entirely sure what your exact idea here is - are you looking
> > to have Broker-J trigger the deletion on the blob at the point at
> > which it believes it no longer needs the message?  Or maybe to update
> > the blob store to make clear that the broker no longer needs to retain
> > a copy of the message?
> >
> > Clearly the API is not designed for allowing extensions here - and I
> > think if we were to try to make it an extension / interception point
> > then we would have to be careful in limiting what could be done here
> > (e.g. offering only some read-only view of the message).
> >
> > Without such an interception extension mechanism, you are correct I
> > think that you would need to create a new vhost type (potentially
> > using a new store type of your own implementation)
> >
> > -- Rob
> >
> > On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com<ma...@murex.com>>
> > wrote:
> >
> > > Hello,
> > >
> > > We're trying to implement the claim-check pattern on top of our
> > > broker
> > > + dispatch-router topology.
> > > So we have a blob store and the idea is for each message above 10KB
> > > to store the payload in it and to only keep an ID in the message.
> > > The tricky part is the delete of the payload in the blob store,
> > > especially since some of our use cases are using multicast...
> > > When looking at the Broker-J source code I found this
> > > MessageDeleteListener which would be really helpful for this use case:
> > >
> > > https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2a
> > > e7
> > > a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/sto
> > > re
> > > /MessageStore.java#L84
> > >
> > > My first question would be, is it ok to use it?
> > > Second, so far the only way I found to register a new listener is to
> > > define my own VirtualHost, is there an easier way ?
> > >
> > > Thanks,
> > > Olivier
> > > *******************************
> > > This e-mail contains information for the intended recipient only. It
> > > may contain proprietary material or confidential information. If you
> > > are not the intended recipient you are not authorized to distribute,
> > > copy or use this e-mail or any attachment to it. Murex cannot
> > > guarantee that it is virus free and accepts no responsibility for
> > > any loss or damage arising from its use. If you have received this
> > > e-mail in error please notify immediately the sender and delete the
> > > original email received, any attachments and all copies from your
> system.
> > >
> > *******************************
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorized to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for any
> > loss or damage arising from its use. If you have received this e-mail
> > in error please notify immediately the sender and delete the original
> > email received, any attachments and all copies from your system.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org<ma...@qpid.apache.org> For
> > additional commands, e-mail: users-help@qpid.apache.org<ma...@qpid.apache.org>
> >
> >
> *******************************
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org<ma...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org<ma...@qpid.apache.org>
>
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

Re: [Broker-J] Claim-check pattern and MessageDeleteListener

[VERMEULEN Olivier <Ol...@murex.com>]

The idea is to avoid any performance impact on the messaging when big messages arrive.
So whenever a producer tries to send a big message (more than 10KB for us) we put the payload in the blob store with a unique ID and we pass this ID in the message instead of the original payload.
When a consumer receives such a message we then retrieve the payload from the blob store.
The problem is that we can't expect the consumer to handle the cleanup of the blob store since it might not be the only consumer receiving this message (multicast for example or even a message being routed to 2 queues through a topic).
The only one that really knows when the payload can be removed from the blob store is the broker.
That's why I was looking for a hook that would be called whenever a message is or will be deleted from the store (this includes the recovery process).
After that whether the hook is at the queue or virtual host level doesn't really matter...

Olivier
________________________________
From: Rob Godfrey <ro...@gmail.com>
Sent: Tuesday, March 10, 2020 6:26:39 PM
To: users@qpid.apache.org <us...@qpid.apache.org>
Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener

Yeah - I was wondering if it might be better to add hooks into the point at
which remove() is called (i.e.
https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/message/AbstractServerMessageImpl.java#L127
),
because a listener on the here could actually work with the message object
where the headers are available... However this would miss the other point
at which messages are removed from the store - in the store recovery
process (e.g.
https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/virtualhost/AsynchronousMessageStoreRecoverer.java#L227
).
Unfortunately at this point we wouldn't have access to the headers.

Can you explain a bit more how your blob storage is supposed to work.
Perhaps there is another way that we can add a hook that would help here
(e.g. on a per queue basis have some sort of "dequeue" trigger which is
passed the message it is about to be permanently removed from the queue).

-- Rob

On Tue, 10 Mar 2020 at 17:59, VERMEULEN Olivier <Ol...@murex.com>
wrote:

> The cast works fine in my case but the problem is that everything is NULL,
> including the meta-data.
> Indeed the message is cleaned before the listeners are called...
>
> https://github.com/apache/qpid-broker-j/blob/7193e26f6fad83f5ce81fd9ef855edb0fd5594ea/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670
>
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: lundi 9 mars 2020 09:58
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
>
> On Mon, 9 Mar 2020 at 09:36, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Hello Rob,
> >
> > "Are you looking to have Broker-J trigger the deletion on the blob at
> > the point at which it believes it no longer needs the message?"
> > Yes that's exactly what I'm trying to do.
> > Note that I managed to do it by creating my own vhost, just wanted to
> > check if there was a less intrusive way of doing it...
> > I agree that we shouldn't be able to modify the message but today I
> > can't even read it easily.
> > What I'm interested in are the headers but from a StoredMessage
> > instance I have to go through internal classes and package-protected
> > classes to get
> > them:
> > ((InternalMessageMetaData) storedMessage.getMetaData()).getHeader();
> > getHeader being package-protected here...
> >
>
> Note that this will only work if the message is an "Internal" message (i.e.
> the casting here would fail if the message were an AMQP 0-9, AMQP 0-10 or
> AMQP 1.0 message sent over the wire).  The store API is basically built to
> be as unaware of the form of the message as possible, and the api here is
> basically designed to allow the store to clean up a message which was
> potentially stored in multiple queues, but which has now been removed from
> all of them.  As such it's not really designed to make it easy for you to
> access the headers (or content - since the "content" will include structure
> for 1.0 messages).
>
>
> > Did I miss something?
> >
>
> No - I think to do this safely we'd have to expose some new API here.
>
> -- Rob
>
>
> >
> > Thanks,
> > Olivier
> >
> > -----Original Message-----
> > From: Rob Godfrey <ro...@gmail.com>
> > Sent: vendredi 6 mars 2020 17:54
> > To: users@qpid.apache.org
> > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> >
> > Hi Olivier
> >
> > I'm not entirely sure what your exact idea here is - are you looking
> > to have Broker-J trigger the deletion on the blob at the point at
> > which it believes it no longer needs the message?  Or maybe to update
> > the blob store to make clear that the broker no longer needs to retain
> > a copy of the message?
> >
> > Clearly the API is not designed for allowing extensions here - and I
> > think if we were to try to make it an extension / interception point
> > then we would have to be careful in limiting what could be done here
> > (e.g. offering only some read-only view of the message).
> >
> > Without such an interception extension mechanism, you are correct I
> > think that you would need to create a new vhost type (potentially
> > using a new store type of your own implementation)
> >
> > -- Rob
> >
> > On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com>
> > wrote:
> >
> > > Hello,
> > >
> > > We're trying to implement the claim-check pattern on top of our
> > > broker
> > > + dispatch-router topology.
> > > So we have a blob store and the idea is for each message above 10KB
> > > to store the payload in it and to only keep an ID in the message.
> > > The tricky part is the delete of the payload in the blob store,
> > > especially since some of our use cases are using multicast...
> > > When looking at the Broker-J source code I found this
> > > MessageDeleteListener which would be really helpful for this use case:
> > >
> > > https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2a
> > > e7
> > > a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/sto
> > > re
> > > /MessageStore.java#L84
> > >
> > > My first question would be, is it ok to use it?
> > > Second, so far the only way I found to register a new listener is to
> > > define my own VirtualHost, is there an easier way ?
> > >
> > > Thanks,
> > > Olivier
> > > *******************************
> > > This e-mail contains information for the intended recipient only. It
> > > may contain proprietary material or confidential information. If you
> > > are not the intended recipient you are not authorized to distribute,
> > > copy or use this e-mail or any attachment to it. Murex cannot
> > > guarantee that it is virus free and accepts no responsibility for
> > > any loss or damage arising from its use. If you have received this
> > > e-mail in error please notify immediately the sender and delete the
> > > original email received, any attachments and all copies from your
> system.
> > >
> > *******************************
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorized to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for any
> > loss or damage arising from its use. If you have received this e-mail
> > in error please notify immediately the sender and delete the original
> > email received, any attachments and all copies from your system.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> > additional commands, e-mail: users-help@qpid.apache.org
> >
> >
> *******************************
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

Re: [Broker-J] Claim-check pattern and MessageDeleteListener

[Rob Godfrey <ro...@gmail.com>]

Yeah - I was wondering if it might be better to add hooks into the point at
which remove() is called (i.e.
https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/message/AbstractServerMessageImpl.java#L127
),
because a listener on the here could actually work with the message object
where the headers are available... However this would miss the other point
at which messages are removed from the store - in the store recovery
process (e.g.
https://github.com/apache/qpid-broker-j/blob/master/broker-core/src/main/java/org/apache/qpid/server/virtualhost/AsynchronousMessageStoreRecoverer.java#L227
).
Unfortunately at this point we wouldn't have access to the headers.

Can you explain a bit more how your blob storage is supposed to work.
Perhaps there is another way that we can add a hook that would help here
(e.g. on a per queue basis have some sort of "dequeue" trigger which is
passed the message it is about to be permanently removed from the queue).

-- Rob

On Tue, 10 Mar 2020 at 17:59, VERMEULEN Olivier <Ol...@murex.com>
wrote:

> The cast works fine in my case but the problem is that everything is NULL,
> including the meta-data.
> Indeed the message is cleaned before the listeners are called...
>
> https://github.com/apache/qpid-broker-j/blob/7193e26f6fad83f5ce81fd9ef855edb0fd5594ea/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670
>
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: lundi 9 mars 2020 09:58
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
>
> On Mon, 9 Mar 2020 at 09:36, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Hello Rob,
> >
> > "Are you looking to have Broker-J trigger the deletion on the blob at
> > the point at which it believes it no longer needs the message?"
> > Yes that's exactly what I'm trying to do.
> > Note that I managed to do it by creating my own vhost, just wanted to
> > check if there was a less intrusive way of doing it...
> > I agree that we shouldn't be able to modify the message but today I
> > can't even read it easily.
> > What I'm interested in are the headers but from a StoredMessage
> > instance I have to go through internal classes and package-protected
> > classes to get
> > them:
> > ((InternalMessageMetaData) storedMessage.getMetaData()).getHeader();
> > getHeader being package-protected here...
> >
>
> Note that this will only work if the message is an "Internal" message (i.e.
> the casting here would fail if the message were an AMQP 0-9, AMQP 0-10 or
> AMQP 1.0 message sent over the wire).  The store API is basically built to
> be as unaware of the form of the message as possible, and the api here is
> basically designed to allow the store to clean up a message which was
> potentially stored in multiple queues, but which has now been removed from
> all of them.  As such it's not really designed to make it easy for you to
> access the headers (or content - since the "content" will include structure
> for 1.0 messages).
>
>
> > Did I miss something?
> >
>
> No - I think to do this safely we'd have to expose some new API here.
>
> -- Rob
>
>
> >
> > Thanks,
> > Olivier
> >
> > -----Original Message-----
> > From: Rob Godfrey <ro...@gmail.com>
> > Sent: vendredi 6 mars 2020 17:54
> > To: users@qpid.apache.org
> > Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
> >
> > Hi Olivier
> >
> > I'm not entirely sure what your exact idea here is - are you looking
> > to have Broker-J trigger the deletion on the blob at the point at
> > which it believes it no longer needs the message?  Or maybe to update
> > the blob store to make clear that the broker no longer needs to retain
> > a copy of the message?
> >
> > Clearly the API is not designed for allowing extensions here - and I
> > think if we were to try to make it an extension / interception point
> > then we would have to be careful in limiting what could be done here
> > (e.g. offering only some read-only view of the message).
> >
> > Without such an interception extension mechanism, you are correct I
> > think that you would need to create a new vhost type (potentially
> > using a new store type of your own implementation)
> >
> > -- Rob
> >
> > On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com>
> > wrote:
> >
> > > Hello,
> > >
> > > We're trying to implement the claim-check pattern on top of our
> > > broker
> > > + dispatch-router topology.
> > > So we have a blob store and the idea is for each message above 10KB
> > > to store the payload in it and to only keep an ID in the message.
> > > The tricky part is the delete of the payload in the blob store,
> > > especially since some of our use cases are using multicast...
> > > When looking at the Broker-J source code I found this
> > > MessageDeleteListener which would be really helpful for this use case:
> > >
> > > https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2a
> > > e7
> > > a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/sto
> > > re
> > > /MessageStore.java#L84
> > >
> > > My first question would be, is it ok to use it?
> > > Second, so far the only way I found to register a new listener is to
> > > define my own VirtualHost, is there an easier way ?
> > >
> > > Thanks,
> > > Olivier
> > > *******************************
> > > This e-mail contains information for the intended recipient only. It
> > > may contain proprietary material or confidential information. If you
> > > are not the intended recipient you are not authorized to distribute,
> > > copy or use this e-mail or any attachment to it. Murex cannot
> > > guarantee that it is virus free and accepts no responsibility for
> > > any loss or damage arising from its use. If you have received this
> > > e-mail in error please notify immediately the sender and delete the
> > > original email received, any attachments and all copies from your
> system.
> > >
> > *******************************
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorized to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for any
> > loss or damage arising from its use. If you have received this e-mail
> > in error please notify immediately the sender and delete the original
> > email received, any attachments and all copies from your system.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> > additional commands, e-mail: users-help@qpid.apache.org
> >
> >
> *******************************
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

RE: [Broker-J] Claim-check pattern and MessageDeleteListener

[VERMEULEN Olivier <Ol...@murex.com>]

The cast works fine in my case but the problem is that everything is NULL, including the meta-data.
Indeed the message is cleaned before the listeners are called...
https://github.com/apache/qpid-broker-j/blob/7193e26f6fad83f5ce81fd9ef855edb0fd5594ea/broker-plugins/jdbc-store/src/main/java/org/apache/qpid/server/store/jdbc/AbstractJDBCMessageStore.java#L1670

Olivier

-----Original Message-----
From: Rob Godfrey <ro...@gmail.com>
Sent: lundi 9 mars 2020 09:58
To: users@qpid.apache.org
Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener

On Mon, 9 Mar 2020 at 09:36, VERMEULEN Olivier <Ol...@murex.com>
wrote:

> Hello Rob,
>
> "Are you looking to have Broker-J trigger the deletion on the blob at
> the point at which it believes it no longer needs the message?"
> Yes that's exactly what I'm trying to do.
> Note that I managed to do it by creating my own vhost, just wanted to
> check if there was a less intrusive way of doing it...
> I agree that we shouldn't be able to modify the message but today I
> can't even read it easily.
> What I'm interested in are the headers but from a StoredMessage
> instance I have to go through internal classes and package-protected
> classes to get
> them:
> ((InternalMessageMetaData) storedMessage.getMetaData()).getHeader();
> getHeader being package-protected here...
>

Note that this will only work if the message is an "Internal" message (i.e.
the casting here would fail if the message were an AMQP 0-9, AMQP 0-10 or AMQP 1.0 message sent over the wire).  The store API is basically built to be as unaware of the form of the message as possible, and the api here is basically designed to allow the store to clean up a message which was potentially stored in multiple queues, but which has now been removed from all of them.  As such it's not really designed to make it easy for you to access the headers (or content - since the "content" will include structure for 1.0 messages).


> Did I miss something?
>

No - I think to do this safely we'd have to expose some new API here.

-- Rob


>
> Thanks,
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: vendredi 6 mars 2020 17:54
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
>
> Hi Olivier
>
> I'm not entirely sure what your exact idea here is - are you looking
> to have Broker-J trigger the deletion on the blob at the point at
> which it believes it no longer needs the message?  Or maybe to update
> the blob store to make clear that the broker no longer needs to retain
> a copy of the message?
>
> Clearly the API is not designed for allowing extensions here - and I
> think if we were to try to make it an extension / interception point
> then we would have to be careful in limiting what could be done here
> (e.g. offering only some read-only view of the message).
>
> Without such an interception extension mechanism, you are correct I
> think that you would need to create a new vhost type (potentially
> using a new store type of your own implementation)
>
> -- Rob
>
> On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Hello,
> >
> > We're trying to implement the claim-check pattern on top of our
> > broker
> > + dispatch-router topology.
> > So we have a blob store and the idea is for each message above 10KB
> > to store the payload in it and to only keep an ID in the message.
> > The tricky part is the delete of the payload in the blob store,
> > especially since some of our use cases are using multicast...
> > When looking at the Broker-J source code I found this
> > MessageDeleteListener which would be really helpful for this use case:
> >
> > https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2a
> > e7
> > a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/sto
> > re
> > /MessageStore.java#L84
> >
> > My first question would be, is it ok to use it?
> > Second, so far the only way I found to register a new listener is to
> > define my own VirtualHost, is there an easier way ?
> >
> > Thanks,
> > Olivier
> > *******************************
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorized to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for
> > any loss or damage arising from its use. If you have received this
> > e-mail in error please notify immediately the sender and delete the
> > original email received, any attachments and all copies from your system.
> >
> *******************************
> This e-mail contains information for the intended recipient only. It
> may contain proprietary material or confidential information. If you
> are not the intended recipient you are not authorized to distribute,
> copy or use this e-mail or any attachment to it. Murex cannot
> guarantee that it is virus free and accepts no responsibility for any
> loss or damage arising from its use. If you have received this e-mail
> in error please notify immediately the sender and delete the original
> email received, any attachments and all copies from your system.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> additional commands, e-mail: users-help@qpid.apache.org
>
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: [Broker-J] Claim-check pattern and MessageDeleteListener

[Rob Godfrey <ro...@gmail.com>]

On Mon, 9 Mar 2020 at 09:36, VERMEULEN Olivier <Ol...@murex.com>
wrote:

> Hello Rob,
>
> "Are you looking to have Broker-J trigger the deletion on the blob at the
> point at which it believes it no longer needs the message?"
> Yes that's exactly what I'm trying to do.
> Note that I managed to do it by creating my own vhost, just wanted to
> check if there was a less intrusive way of doing it...
> I agree that we shouldn't be able to modify the message but today I can't
> even read it easily.
> What I'm interested in are the headers but from a StoredMessage instance I
> have to go through internal classes and package-protected classes to get
> them:
> ((InternalMessageMetaData) storedMessage.getMetaData()).getHeader();
> getHeader being package-protected here...
>

Note that this will only work if the message is an "Internal" message (i.e.
the casting here would fail if the message were an AMQP 0-9, AMQP 0-10 or
AMQP 1.0 message sent over the wire).  The store API is basically built to
be as unaware of the form of the message as possible, and the api here is
basically designed to allow the store to clean up a message which was
potentially stored in multiple queues, but which has now been removed from
all of them.  As such it's not really designed to make it easy for you to
access the headers (or content - since the "content" will include structure
for 1.0 messages).


> Did I miss something?
>

No - I think to do this safely we'd have to expose some new API here.

-- Rob


>
> Thanks,
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: vendredi 6 mars 2020 17:54
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener
>
> Hi Olivier
>
> I'm not entirely sure what your exact idea here is - are you looking to
> have Broker-J trigger the deletion on the blob at the point at which it
> believes it no longer needs the message?  Or maybe to update the blob store
> to make clear that the broker no longer needs to retain a copy of the
> message?
>
> Clearly the API is not designed for allowing extensions here - and I think
> if we were to try to make it an extension / interception point then we
> would have to be careful in limiting what could be done here (e.g. offering
> only some read-only view of the message).
>
> Without such an interception extension mechanism, you are correct I think
> that you would need to create a new vhost type (potentially using a new
> store type of your own implementation)
>
> -- Rob
>
> On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Hello,
> >
> > We're trying to implement the claim-check pattern on top of our broker
> > + dispatch-router topology.
> > So we have a blob store and the idea is for each message above 10KB to
> > store the payload in it and to only keep an ID in the message.
> > The tricky part is the delete of the payload in the blob store,
> > especially since some of our use cases are using multicast...
> > When looking at the Broker-J source code I found this
> > MessageDeleteListener which would be really helpful for this use case:
> >
> > https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2ae7
> > a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/store
> > /MessageStore.java#L84
> >
> > My first question would be, is it ok to use it?
> > Second, so far the only way I found to register a new listener is to
> > define my own VirtualHost, is there an easier way ?
> >
> > Thanks,
> > Olivier
> > *******************************
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorized to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for any
> > loss or damage arising from its use. If you have received this e-mail
> > in error please notify immediately the sender and delete the original
> > email received, any attachments and all copies from your system.
> >
> *******************************
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

RE: [Broker-J] Claim-check pattern and MessageDeleteListener

[VERMEULEN Olivier <Ol...@murex.com>]

Hello Rob,

"Are you looking to have Broker-J trigger the deletion on the blob at the point at which it believes it no longer needs the message?"
Yes that's exactly what I'm trying to do.
Note that I managed to do it by creating my own vhost, just wanted to check if there was a less intrusive way of doing it...
I agree that we shouldn't be able to modify the message but today I can't even read it easily.
What I'm interested in are the headers but from a StoredMessage instance I have to go through internal classes and package-protected classes to get them:
((InternalMessageMetaData) storedMessage.getMetaData()).getHeader();
getHeader being package-protected here...
Did I miss something?

Thanks,
Olivier

-----Original Message-----
From: Rob Godfrey <ro...@gmail.com>
Sent: vendredi 6 mars 2020 17:54
To: users@qpid.apache.org
Subject: Re: [Broker-J] Claim-check pattern and MessageDeleteListener

Hi Olivier

I'm not entirely sure what your exact idea here is - are you looking to have Broker-J trigger the deletion on the blob at the point at which it believes it no longer needs the message?  Or maybe to update the blob store to make clear that the broker no longer needs to retain a copy of the message?

Clearly the API is not designed for allowing extensions here - and I think if we were to try to make it an extension / interception point then we would have to be careful in limiting what could be done here (e.g. offering only some read-only view of the message).

Without such an interception extension mechanism, you are correct I think that you would need to create a new vhost type (potentially using a new store type of your own implementation)

-- Rob

On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier <Ol...@murex.com>
wrote:

> Hello,
>
> We're trying to implement the claim-check pattern on top of our broker
> + dispatch-router topology.
> So we have a blob store and the idea is for each message above 10KB to
> store the payload in it and to only keep an ID in the message.
> The tricky part is the delete of the payload in the blob store,
> especially since some of our use cases are using multicast...
> When looking at the Broker-J source code I found this
> MessageDeleteListener which would be really helpful for this use case:
>
> https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2ae7
> a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/store
> /MessageStore.java#L84
>
> My first question would be, is it ok to use it?
> Second, so far the only way I found to register a new listener is to
> define my own VirtualHost, is there an easier way ?
>
> Thanks,
> Olivier
> *******************************
> This e-mail contains information for the intended recipient only. It
> may contain proprietary material or confidential information. If you
> are not the intended recipient you are not authorized to distribute,
> copy or use this e-mail or any attachment to it. Murex cannot
> guarantee that it is virus free and accepts no responsibility for any
> loss or damage arising from its use. If you have received this e-mail
> in error please notify immediately the sender and delete the original
> email received, any attachments and all copies from your system.
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: [Broker-J] Claim-check pattern and MessageDeleteListener

[Rob Godfrey <ro...@gmail.com>]

Hi Olivier

I'm not entirely sure what your exact idea here is - are you looking to
have Broker-J trigger the deletion on the blob at the point at which it
believes it no longer needs the message?  Or maybe to update the blob store
to make clear that the broker no longer needs to retain a copy of the
message?

Clearly the API is not designed for allowing extensions here - and I think
if we were to try to make it an extension / interception point then we
would have to be careful in limiting what could be done here (e.g. offering
only some read-only view of the message).

Without such an interception extension mechanism, you are correct I think
that you would need to create a new vhost type (potentially using a new
store type of your own implementation)

-- Rob

On Fri, 6 Mar 2020 at 16:37, VERMEULEN Olivier <Ol...@murex.com>
wrote:

> Hello,
>
> We're trying to implement the claim-check pattern on top of our broker +
> dispatch-router topology.
> So we have a blob store and the idea is for each message above 10KB to
> store the payload in it and to only keep an ID in the message.
> The tricky part is the delete of the payload in the blob store, especially
> since some of our use cases are using multicast...
> When looking at the Broker-J source code I found this
> MessageDeleteListener which would be really helpful for this use case:
>
> https://github.com/apache/qpid-broker-j/blob/c018e1ac9d21e9f5eb38d2ae7a26a31e63c07fdf/broker-core/src/main/java/org/apache/qpid/server/store/MessageStore.java#L84
>
> My first question would be, is it ok to use it?
> Second, so far the only way I found to register a new listener is to
> define my own VirtualHost, is there an easier way ?
>
> Thanks,
> Olivier
> *******************************
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>