You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by VERMEULEN Olivier <Ol...@murex.com> on 2018/09/14 13:29:55 UTC
[Broker-J] JDBC message store performance
Hello,
We ran a performance test with a bunch of brokers and an Oracle database to store the messages.
We noticed that the database was a bit overloaded with commits.
Looking at the logs we saw that sending a message was triggering 1 commit for 3 operations (QPID_QUEUE_ENTRIES, QPID_MESSAGE_METADATA, QPID_MESSAGE_CONTENT) which is what we were expecting but receiving a message was triggering 2 commits (1 for QPID_QUEUE_ENTRIES and 1 for QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
I debugged a bit the code and saw that in AbstractVirtualHost.executeTransaction the delete on QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a "post commit" operation, explaining why we have 2 commits.
Is it something expected? Do you think we could reduce this to 1 commit when receiving a message?
Thanks,
Olivier
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
RE: [Broker-J] JDBC message store performance
Posted by VERMEULEN Olivier <Ol...@murex.com>.
Yes you can go ahead.
Thanks a lot for the fix.
Olivier
-----Original Message-----
From: Rob Godfrey <ro...@gmail.com>
Sent: lundi 29 octobre 2018 16:16
To: users@qpid.apache.org
Subject: Re: [Broker-J] JDBC message store performance
Hi Olivier,
I'm not terribly surprised... I've not tested extensively with JDBC, but my experience with BDB and batch sizes was similar. If you are happy with the original patch I'll apply that on trunk from QPID-8242.
-- Rob
On Mon, 29 Oct 2018 at 15:41, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Hello Rob,
>
> I modified your patch to set a minimum batch size of 10 but didn't see
> any improvement in the overall throughput...
>
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: mardi 2 octobre 2018 18:08
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] JDBC message store performance
>
> On Tue, 2 Oct 2018 at 17:47, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Hello Rob,
> >
> > We tested your fix on an Oracle database and it works fine, we even
> > noticed a 7% improvement of the overall throughput!
> >
>
> Great!
>
>
> > But the average size of the batches is quite small: between 3 or 4.
> > Do you think it would possible (and a good idea) to define a minimum
> > batch size and/or a batch period?
> >
> >
> Sure - the only potential "issue" is having more orphan message
> content rows in the database if you have an abrupt shutdown of the
> broker; on restart these should be cleaned up anyway by the recovery
> process. I'm not sure if you would see much performance benefit (my
> guess is that a lot of the benefit you will see if from taking the
> message deletion off the critical path on message delivery) but we
> could experiment with a tiny change to the patch to make a minimum
> batch size of 10 or whatever and see if you get a further speedup.
>
> -- Rob
>
>
> > Olivier
> >
> > -----Original Message-----
> > From: Rob Godfrey <ro...@gmail.com>
> > Sent: mardi 25 septembre 2018 13:20
> > To: users@qpid.apache.org
> > Subject: Re: [Broker-J] JDBC message store performance
> >
> > And now with a fix to allow the system tests to pass consistently
> >
> > https://issues.apache.org/jira/secure/attachment/12941197/QPID-8242.
> > pa
> > tch
> >
> > -- Rob
> >
> > On Tue, 25 Sep 2018 at 12:24, Rob Godfrey <ro...@gmail.com>
> wrote:
> >
> > > D'oh - no idea what happened there.
> > >
> > > Re-attached with an actual diff this time (I hope :) )
> > >
> > > https://issues.apache.org/jira/secure/attachment/12941192/QPID-8242.
> > > pa
> > > tch
> > >
> > > -- Rob
> > >
> > > On Wed, 19 Sep 2018 at 01:23, Rob Godfrey
> > > <ro...@gmail.com>
> > wrote:
> > >
> > >> As an alternative approach, how about the (totally untested)
> > >> patch I've attached to QPID-8242 [1]
> > >>
> > >> Rather than trying to consolidate the commits to delete the queue
> > >> entry and the message content it instead attempts to delete the
> > >> message content asynchronously and batch up all message contents
> > >> awaiting deletion. From a performance point of view, on the
> > >> broker it takes the message content deletion off the connection
> > >> thread, and if the database is being the bottleneck it should
> > >> start batching up the
> > deletions of content.
> > >>
> > >> -- Rob
> > >>
> > >> [1]
> > >> https://issues.apache.org/jira/secure/attachment/12940318/QPID-82
> > >> 42
> > >> .p
> > >> atch
> > >>
> > >> On Mon, 17 Sep 2018 at 15:22, Rob Godfrey
> > >> <ro...@gmail.com>
> > >> wrote:
> > >>
> > >>> The reference counting is done in AbstractServerMessageImpl.java.
> > >>> In general instances of ServerMessage should not be passed
> > >>> around, rather a MessageReference (obtained by calling
> > >>> newReference(..) on the message object. Then if the message is
> > >>> no longer required in that context,
> > >>> release() can be called on the reference. Within the context of
> > >>> a queue this is normally encapsulated by the notion of the
> > >>> QueueEntry (which holds the relevant MessageReference). When
> > >>> delete() is called on the QueueEntry (delete being defined in
> > >>> the MessageInstance interface), this releases the reference,
> > >>> which decrements the reference count, and if the count has gone
> > >>> to zero the
> > message calls the store to delete itself.
> > >>>
> > >>> I think trying to change this logic would be a bit of a
> > >>> nightmare (understatement). I think a better alternative in
> > >>> terms of reducing the number of commits in the JDBC store might
> > >>> instead be for the
> > >>> remove() method on StoredJDBCMessage to (rather than immediately
> > >>> commit the message
> > >>> delete) schedule the message removal to be picked up by the next
> > >>> commit that the store is asked to perform. This would make the
> > >>> behaviour more like the BDB store (where we schedule the commit
> > >>> but don't actually force the sync to disk on message deletion).
> > >>>
> > >>> -- Rob
> > >>>
> > >>> On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
> > >>> Olivier.VERMEULEN@murex.com> wrote:
> > >>>
> > >>>> Ok then I missed something.
> > >>>> Whan/Where is the reference-counting you were talking about in
> > >>>> your first mail happening?
> > >>>>
> > >>>> Olivier
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: Rob Godfrey <ro...@gmail.com>
> > >>>> Sent: lundi 17 septembre 2018 14:05
> > >>>> To: users@qpid.apache.org
> > >>>> Subject: Re: [Broker-J] JDBC message store performance
> > >>>>
> > >>>> Hi Olivier,
> > >>>>
> > >>>> the approach you are attempting will not work for the reasons I
> > >>>> described previously. If the message has (for instance) been
> > >>>> placed in two durable subscription queues (because there are
> > >>>> two durable subscriptions) then as soon as the message is
> > >>>> consumed from the first queue it would be removed from the
> > >>>> store, leading to problems when the second consumer tries to read the message.
> > >>>>
> > >>>> -- Rob
> > >>>>
> > >>>> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
> > >>>> Olivier.VERMEULEN@murex.com>
> > >>>> wrote:
> > >>>>
> > >>>> > Hello Rob,
> > >>>> >
> > >>>> > Thanks for the answer.
> > >>>> > I started looking at the code to see if there is something I
> > >>>> > can do about these 2 commits.
> > >>>> > But before going any further I'd like your input on the
> > >>>> > below, to see if what I'm trying to do could work or if I'm
> > >>>> > missing something
> > >>>> (which
> > >>>> > I'm surely are)
> > >>>> >
> > >>>> >
> > >>>> https://github.com/apache/qpid-broker-j/compare/master...overme
> > >>>> ul
> > >>>> en
> > >>>> :fe
> > >>>> > ature/jdbc-message-store-commits
> > >>>> >
> > >>>> > Regards,
> > >>>> > Olivier
> > >>>> >
> > >>>> > -----Original Message-----
> > >>>> > From: Rob Godfrey <ro...@gmail.com>
> > >>>> > Sent: vendredi 14 septembre 2018 16:06
> > >>>> > To: users@qpid.apache.org
> > >>>> > Cc: AYOUBI Ali <Al...@murex.com>
> > >>>> > Subject: Re: [Broker-J] JDBC message store performance
> > >>>> >
> > >>>> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
> > >>>> > Olivier.VERMEULEN@murex.com>
> > >>>> > wrote:
> > >>>> >
> > >>>> > > Hello,
> > >>>> > >
> > >>>> > > We ran a performance test with a bunch of brokers and an
> > >>>> > > Oracle database to store the messages.
> > >>>> > > We noticed that the database was a bit overloaded with commits.
> > >>>> > > Looking at the logs we saw that sending a message was
> > >>>> > > triggering 1 commit for 3 operations (QPID_QUEUE_ENTRIES,
> > >>>> > > QPID_MESSAGE_METADATA,
> > >>>> > > QPID_MESSAGE_CONTENT) which is what we were expecting but
> > >>>> > > receiving a message was triggering 2 commits (1 for
> > >>>> > > QPID_QUEUE_ENTRIES and 1 for QPID_MESSAGE_METADATA and
> > QPID_MESSAGE_CONTENT).
> > >>>> > > I debugged a bit the code and saw that in
> > >>>> > > AbstractVirtualHost.executeTransaction the delete on
> > >>>> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined
> > >>>> > > as a "post commit" operation, explaining why we have 2 commits.
> > >>>> > > Is it something expected? Do you think we could reduce this
> > >>>> > > to
> > >>>> > > 1 commit when receiving a message?
> > >>>> > >
> > >>>> >
> > >>>> > It's not unexpected - basically the issue is that the broker
> > >>>> > needs to cope with the possibility that the same message is
> > >>>> > being stored on multiple queues. The first commit is
> > >>>> > deleting the referencing of the message from the given queue.
> > >>>> > The second commit is occuring after
> > >>>> the
> > >>>> > message has been definitively removed from the queue, and the
> > >>>> > store has determined that there are no more references, so it
> > >>>> > is ok to remove the message. This is driven by
> > >>>> > reference-counting of the message, and has historically been
> > >>>> > a place of many potential race conditions. I'm sure it is
> > >>>> > possible to optimise the code in some
> > >>>> way,
> > >>>> > but it may not be "easy". For the BDB store this doesn't
> > >>>> > matter as much as the actual synchronisation to disk of these
> > >>>> > operations is coalesced, obviously this is more of an issue
> > >>>> > for
> the JDBC store.
> > >>>> >
> > >>>> > -- Rob
> > >>>> >
> > >>>> >
> > >>>> > >
> > >>>> > > Thanks,
> > >>>> > > Olivier
> > >>>> > > *******************************
> > >>>> > >
> > >>>> > > This e-mail contains information for the intended recipient
> only.
> > >>>> It
> > >>>> > > may contain proprietary material or confidential information.
> > >>>> > > If
> > >>>> you
> > >>>> > > are not the intended recipient you are not authorised to
> > >>>> distribute,
> > >>>> > > copy or use this e-mail or any attachment to it. Murex
> > >>>> > > cannot guarantee that it is virus free and accepts no
> > >>>> > > responsibility for any loss or damage arising from its use.
> > >>>> > > If you have received this e-mail in error please notify
> > >>>> > > immediately the sender and delete the original email
> > >>>> > > received, any attachments and all copies from your
> > >>>> system.
> > >>>> > >
> > >>>> > *******************************
> > >>>> >
> > >>>> > This e-mail contains information for the intended recipient only.
> > >>>> > It may contain proprietary material or confidential information.
> > >>>> > If you are not the intended recipient you are not authorised
> > >>>> > to distribute, copy or use this e-mail or any attachment to it.
> > >>>> > Murex cannot guarantee that it is virus free and accepts no
> > >>>> > responsibility for any loss or damage arising from its use.
> > >>>> > If you have received this e-mail in error please notify
> > >>>> > immediately the sender and delete the original email
> > >>>> > received, any attachments
> > and all copies from your system.
> > >>>> >
> > >>>> *******************************
> > >>>>
> > >>>> This e-mail contains information for the intended recipient only.
> > >>>> It may contain proprietary material or confidential information.
> > >>>> If you are not the intended recipient you are not authorised to
> > >>>> distribute, copy or use this e-mail or any attachment to it.
> > >>>> Murex cannot guarantee that it is virus free and accepts no
> > >>>> responsibility for any loss or damage arising from its use. If
> > >>>> you have received this e-mail in error please notify
> > >>>> immediately the sender and delete the original email received,
> > >>>> any attachments and
> > all copies from your system.
> > >>>>
> > >>>
> > *******************************
> >
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorised to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for
> > any loss or damage arising from its use. If you have received this
> > e-mail in error please notify immediately the sender and delete the
> > original email received, any attachments and all copies from your system.
> >
> ******************************* This e-mail contains information for
> the intended recipient only. It may contain proprietary material or
> confidential information. If you are not the intended recipient you
> are not authorized to distribute, copy or use this e-mail or any attachment to it.
> Murex cannot guarantee that it is virus free and accepts no
> responsibility for any loss or damage arising from its use. If you
> have received this e-mail in error please notify immediately the
> sender and delete the original email received, any attachments and all copies from your system.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> additional commands, e-mail: users-help@qpid.apache.org
>
>
******************************* This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Broker-J] JDBC message store performance
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Olivier,
I'm not terribly surprised... I've not tested extensively with JDBC, but my
experience with BDB and batch sizes was similar. If you are happy with the
original patch I'll apply that on trunk from QPID-8242.
-- Rob
On Mon, 29 Oct 2018 at 15:41, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Hello Rob,
>
> I modified your patch to set a minimum batch size of 10 but didn't see any
> improvement in the overall throughput...
>
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: mardi 2 octobre 2018 18:08
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] JDBC message store performance
>
> On Tue, 2 Oct 2018 at 17:47, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Hello Rob,
> >
> > We tested your fix on an Oracle database and it works fine, we even
> > noticed a 7% improvement of the overall throughput!
> >
>
> Great!
>
>
> > But the average size of the batches is quite small: between 3 or 4.
> > Do you think it would possible (and a good idea) to define a minimum
> > batch size and/or a batch period?
> >
> >
> Sure - the only potential "issue" is having more orphan message content
> rows in the database if you have an abrupt shutdown of the broker; on
> restart these should be cleaned up anyway by the recovery process. I'm not
> sure if you would see much performance benefit (my guess is that a lot of
> the benefit you will see if from taking the message deletion off the
> critical path on message delivery) but we could experiment with a tiny
> change to the patch to make a minimum batch size of 10 or whatever and see
> if you get a further speedup.
>
> -- Rob
>
>
> > Olivier
> >
> > -----Original Message-----
> > From: Rob Godfrey <ro...@gmail.com>
> > Sent: mardi 25 septembre 2018 13:20
> > To: users@qpid.apache.org
> > Subject: Re: [Broker-J] JDBC message store performance
> >
> > And now with a fix to allow the system tests to pass consistently
> >
> > https://issues.apache.org/jira/secure/attachment/12941197/QPID-8242.pa
> > tch
> >
> > -- Rob
> >
> > On Tue, 25 Sep 2018 at 12:24, Rob Godfrey <ro...@gmail.com>
> wrote:
> >
> > > D'oh - no idea what happened there.
> > >
> > > Re-attached with an actual diff this time (I hope :) )
> > >
> > > https://issues.apache.org/jira/secure/attachment/12941192/QPID-8242.
> > > pa
> > > tch
> > >
> > > -- Rob
> > >
> > > On Wed, 19 Sep 2018 at 01:23, Rob Godfrey <ro...@gmail.com>
> > wrote:
> > >
> > >> As an alternative approach, how about the (totally untested) patch
> > >> I've attached to QPID-8242 [1]
> > >>
> > >> Rather than trying to consolidate the commits to delete the queue
> > >> entry and the message content it instead attempts to delete the
> > >> message content asynchronously and batch up all message contents
> > >> awaiting deletion. From a performance point of view, on the broker
> > >> it takes the message content deletion off the connection thread,
> > >> and if the database is being the bottleneck it should start
> > >> batching up the
> > deletions of content.
> > >>
> > >> -- Rob
> > >>
> > >> [1]
> > >> https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242
> > >> .p
> > >> atch
> > >>
> > >> On Mon, 17 Sep 2018 at 15:22, Rob Godfrey <ro...@gmail.com>
> > >> wrote:
> > >>
> > >>> The reference counting is done in AbstractServerMessageImpl.java.
> > >>> In general instances of ServerMessage should not be passed around,
> > >>> rather a MessageReference (obtained by calling newReference(..) on
> > >>> the message object. Then if the message is no longer required in
> > >>> that context,
> > >>> release() can be called on the reference. Within the context of a
> > >>> queue this is normally encapsulated by the notion of the
> > >>> QueueEntry (which holds the relevant MessageReference). When
> > >>> delete() is called on the QueueEntry (delete being defined in the
> > >>> MessageInstance interface), this releases the reference, which
> > >>> decrements the reference count, and if the count has gone to zero
> > >>> the
> > message calls the store to delete itself.
> > >>>
> > >>> I think trying to change this logic would be a bit of a nightmare
> > >>> (understatement). I think a better alternative in terms of
> > >>> reducing the number of commits in the JDBC store might instead be
> > >>> for the
> > >>> remove() method on StoredJDBCMessage to (rather than immediately
> > >>> commit the message
> > >>> delete) schedule the message removal to be picked up by the next
> > >>> commit that the store is asked to perform. This would make the
> > >>> behaviour more like the BDB store (where we schedule the commit
> > >>> but don't actually force the sync to disk on message deletion).
> > >>>
> > >>> -- Rob
> > >>>
> > >>> On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
> > >>> Olivier.VERMEULEN@murex.com> wrote:
> > >>>
> > >>>> Ok then I missed something.
> > >>>> Whan/Where is the reference-counting you were talking about in
> > >>>> your first mail happening?
> > >>>>
> > >>>> Olivier
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: Rob Godfrey <ro...@gmail.com>
> > >>>> Sent: lundi 17 septembre 2018 14:05
> > >>>> To: users@qpid.apache.org
> > >>>> Subject: Re: [Broker-J] JDBC message store performance
> > >>>>
> > >>>> Hi Olivier,
> > >>>>
> > >>>> the approach you are attempting will not work for the reasons I
> > >>>> described previously. If the message has (for instance) been
> > >>>> placed in two durable subscription queues (because there are two
> > >>>> durable subscriptions) then as soon as the message is consumed
> > >>>> from the first queue it would be removed from the store, leading
> > >>>> to problems when the second consumer tries to read the message.
> > >>>>
> > >>>> -- Rob
> > >>>>
> > >>>> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
> > >>>> Olivier.VERMEULEN@murex.com>
> > >>>> wrote:
> > >>>>
> > >>>> > Hello Rob,
> > >>>> >
> > >>>> > Thanks for the answer.
> > >>>> > I started looking at the code to see if there is something I
> > >>>> > can do about these 2 commits.
> > >>>> > But before going any further I'd like your input on the below,
> > >>>> > to see if what I'm trying to do could work or if I'm missing
> > >>>> > something
> > >>>> (which
> > >>>> > I'm surely are)
> > >>>> >
> > >>>> >
> > >>>> https://github.com/apache/qpid-broker-j/compare/master...overmeul
> > >>>> en
> > >>>> :fe
> > >>>> > ature/jdbc-message-store-commits
> > >>>> >
> > >>>> > Regards,
> > >>>> > Olivier
> > >>>> >
> > >>>> > -----Original Message-----
> > >>>> > From: Rob Godfrey <ro...@gmail.com>
> > >>>> > Sent: vendredi 14 septembre 2018 16:06
> > >>>> > To: users@qpid.apache.org
> > >>>> > Cc: AYOUBI Ali <Al...@murex.com>
> > >>>> > Subject: Re: [Broker-J] JDBC message store performance
> > >>>> >
> > >>>> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
> > >>>> > Olivier.VERMEULEN@murex.com>
> > >>>> > wrote:
> > >>>> >
> > >>>> > > Hello,
> > >>>> > >
> > >>>> > > We ran a performance test with a bunch of brokers and an
> > >>>> > > Oracle database to store the messages.
> > >>>> > > We noticed that the database was a bit overloaded with commits.
> > >>>> > > Looking at the logs we saw that sending a message was
> > >>>> > > triggering 1 commit for 3 operations (QPID_QUEUE_ENTRIES,
> > >>>> > > QPID_MESSAGE_METADATA,
> > >>>> > > QPID_MESSAGE_CONTENT) which is what we were expecting but
> > >>>> > > receiving a message was triggering 2 commits (1 for
> > >>>> > > QPID_QUEUE_ENTRIES and 1 for QPID_MESSAGE_METADATA and
> > QPID_MESSAGE_CONTENT).
> > >>>> > > I debugged a bit the code and saw that in
> > >>>> > > AbstractVirtualHost.executeTransaction the delete on
> > >>>> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as
> > >>>> > > a "post commit" operation, explaining why we have 2 commits.
> > >>>> > > Is it something expected? Do you think we could reduce this
> > >>>> > > to
> > >>>> > > 1 commit when receiving a message?
> > >>>> > >
> > >>>> >
> > >>>> > It's not unexpected - basically the issue is that the broker
> > >>>> > needs to cope with the possibility that the same message is
> > >>>> > being stored on multiple queues. The first commit is deleting
> > >>>> > the referencing of the message from the given queue. The
> > >>>> > second commit is occuring after
> > >>>> the
> > >>>> > message has been definitively removed from the queue, and the
> > >>>> > store has determined that there are no more references, so it
> > >>>> > is ok to remove the message. This is driven by
> > >>>> > reference-counting of the message, and has historically been a
> > >>>> > place of many potential race conditions. I'm sure it is
> > >>>> > possible to optimise the code in some
> > >>>> way,
> > >>>> > but it may not be "easy". For the BDB store this doesn't
> > >>>> > matter as much as the actual synchronisation to disk of these
> > >>>> > operations is coalesced, obviously this is more of an issue for
> the JDBC store.
> > >>>> >
> > >>>> > -- Rob
> > >>>> >
> > >>>> >
> > >>>> > >
> > >>>> > > Thanks,
> > >>>> > > Olivier
> > >>>> > > *******************************
> > >>>> > >
> > >>>> > > This e-mail contains information for the intended recipient
> only.
> > >>>> It
> > >>>> > > may contain proprietary material or confidential information.
> > >>>> > > If
> > >>>> you
> > >>>> > > are not the intended recipient you are not authorised to
> > >>>> distribute,
> > >>>> > > copy or use this e-mail or any attachment to it. Murex cannot
> > >>>> > > guarantee that it is virus free and accepts no responsibility
> > >>>> > > for any loss or damage arising from its use. If you have
> > >>>> > > received this e-mail in error please notify immediately the
> > >>>> > > sender and delete the original email received, any
> > >>>> > > attachments and all copies from your
> > >>>> system.
> > >>>> > >
> > >>>> > *******************************
> > >>>> >
> > >>>> > This e-mail contains information for the intended recipient only.
> > >>>> > It may contain proprietary material or confidential information.
> > >>>> > If you are not the intended recipient you are not authorised to
> > >>>> > distribute, copy or use this e-mail or any attachment to it.
> > >>>> > Murex cannot guarantee that it is virus free and accepts no
> > >>>> > responsibility for any loss or damage arising from its use. If
> > >>>> > you have received this e-mail in error please notify
> > >>>> > immediately the sender and delete the original email received,
> > >>>> > any attachments
> > and all copies from your system.
> > >>>> >
> > >>>> *******************************
> > >>>>
> > >>>> This e-mail contains information for the intended recipient only.
> > >>>> It may contain proprietary material or confidential information.
> > >>>> If you are not the intended recipient you are not authorised to
> > >>>> distribute, copy or use this e-mail or any attachment to it.
> > >>>> Murex cannot guarantee that it is virus free and accepts no
> > >>>> responsibility for any loss or damage arising from its use. If
> > >>>> you have received this e-mail in error please notify immediately
> > >>>> the sender and delete the original email received, any
> > >>>> attachments and
> > all copies from your system.
> > >>>>
> > >>>
> > *******************************
> >
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorised to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for any
> > loss or damage arising from its use. If you have received this e-mail
> > in error please notify immediately the sender and delete the original
> > email received, any attachments and all copies from your system.
> >
> ******************************* This e-mail contains information for the
> intended recipient only. It may contain proprietary material or
> confidential information. If you are not the intended recipient you are not
> authorized to distribute, copy or use this e-mail or any attachment to it.
> Murex cannot guarantee that it is virus free and accepts no responsibility
> for any loss or damage arising from its use. If you have received this
> e-mail in error please notify immediately the sender and delete the
> original email received, any attachments and all copies from your system.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
RE: [Broker-J] JDBC message store performance
Posted by VERMEULEN Olivier <Ol...@murex.com>.
Hello Rob,
I modified your patch to set a minimum batch size of 10 but didn't see any improvement in the overall throughput...
Olivier
-----Original Message-----
From: Rob Godfrey <ro...@gmail.com>
Sent: mardi 2 octobre 2018 18:08
To: users@qpid.apache.org
Subject: Re: [Broker-J] JDBC message store performance
On Tue, 2 Oct 2018 at 17:47, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Hello Rob,
>
> We tested your fix on an Oracle database and it works fine, we even
> noticed a 7% improvement of the overall throughput!
>
Great!
> But the average size of the batches is quite small: between 3 or 4.
> Do you think it would possible (and a good idea) to define a minimum
> batch size and/or a batch period?
>
>
Sure - the only potential "issue" is having more orphan message content rows in the database if you have an abrupt shutdown of the broker; on restart these should be cleaned up anyway by the recovery process. I'm not sure if you would see much performance benefit (my guess is that a lot of the benefit you will see if from taking the message deletion off the critical path on message delivery) but we could experiment with a tiny change to the patch to make a minimum batch size of 10 or whatever and see if you get a further speedup.
-- Rob
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: mardi 25 septembre 2018 13:20
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] JDBC message store performance
>
> And now with a fix to allow the system tests to pass consistently
>
> https://issues.apache.org/jira/secure/attachment/12941197/QPID-8242.pa
> tch
>
> -- Rob
>
> On Tue, 25 Sep 2018 at 12:24, Rob Godfrey <ro...@gmail.com> wrote:
>
> > D'oh - no idea what happened there.
> >
> > Re-attached with an actual diff this time (I hope :) )
> >
> > https://issues.apache.org/jira/secure/attachment/12941192/QPID-8242.
> > pa
> > tch
> >
> > -- Rob
> >
> > On Wed, 19 Sep 2018 at 01:23, Rob Godfrey <ro...@gmail.com>
> wrote:
> >
> >> As an alternative approach, how about the (totally untested) patch
> >> I've attached to QPID-8242 [1]
> >>
> >> Rather than trying to consolidate the commits to delete the queue
> >> entry and the message content it instead attempts to delete the
> >> message content asynchronously and batch up all message contents
> >> awaiting deletion. From a performance point of view, on the broker
> >> it takes the message content deletion off the connection thread,
> >> and if the database is being the bottleneck it should start
> >> batching up the
> deletions of content.
> >>
> >> -- Rob
> >>
> >> [1]
> >> https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242
> >> .p
> >> atch
> >>
> >> On Mon, 17 Sep 2018 at 15:22, Rob Godfrey <ro...@gmail.com>
> >> wrote:
> >>
> >>> The reference counting is done in AbstractServerMessageImpl.java.
> >>> In general instances of ServerMessage should not be passed around,
> >>> rather a MessageReference (obtained by calling newReference(..) on
> >>> the message object. Then if the message is no longer required in
> >>> that context,
> >>> release() can be called on the reference. Within the context of a
> >>> queue this is normally encapsulated by the notion of the
> >>> QueueEntry (which holds the relevant MessageReference). When
> >>> delete() is called on the QueueEntry (delete being defined in the
> >>> MessageInstance interface), this releases the reference, which
> >>> decrements the reference count, and if the count has gone to zero
> >>> the
> message calls the store to delete itself.
> >>>
> >>> I think trying to change this logic would be a bit of a nightmare
> >>> (understatement). I think a better alternative in terms of
> >>> reducing the number of commits in the JDBC store might instead be
> >>> for the
> >>> remove() method on StoredJDBCMessage to (rather than immediately
> >>> commit the message
> >>> delete) schedule the message removal to be picked up by the next
> >>> commit that the store is asked to perform. This would make the
> >>> behaviour more like the BDB store (where we schedule the commit
> >>> but don't actually force the sync to disk on message deletion).
> >>>
> >>> -- Rob
> >>>
> >>> On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
> >>> Olivier.VERMEULEN@murex.com> wrote:
> >>>
> >>>> Ok then I missed something.
> >>>> Whan/Where is the reference-counting you were talking about in
> >>>> your first mail happening?
> >>>>
> >>>> Olivier
> >>>>
> >>>> -----Original Message-----
> >>>> From: Rob Godfrey <ro...@gmail.com>
> >>>> Sent: lundi 17 septembre 2018 14:05
> >>>> To: users@qpid.apache.org
> >>>> Subject: Re: [Broker-J] JDBC message store performance
> >>>>
> >>>> Hi Olivier,
> >>>>
> >>>> the approach you are attempting will not work for the reasons I
> >>>> described previously. If the message has (for instance) been
> >>>> placed in two durable subscription queues (because there are two
> >>>> durable subscriptions) then as soon as the message is consumed
> >>>> from the first queue it would be removed from the store, leading
> >>>> to problems when the second consumer tries to read the message.
> >>>>
> >>>> -- Rob
> >>>>
> >>>> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
> >>>> Olivier.VERMEULEN@murex.com>
> >>>> wrote:
> >>>>
> >>>> > Hello Rob,
> >>>> >
> >>>> > Thanks for the answer.
> >>>> > I started looking at the code to see if there is something I
> >>>> > can do about these 2 commits.
> >>>> > But before going any further I'd like your input on the below,
> >>>> > to see if what I'm trying to do could work or if I'm missing
> >>>> > something
> >>>> (which
> >>>> > I'm surely are)
> >>>> >
> >>>> >
> >>>> https://github.com/apache/qpid-broker-j/compare/master...overmeul
> >>>> en
> >>>> :fe
> >>>> > ature/jdbc-message-store-commits
> >>>> >
> >>>> > Regards,
> >>>> > Olivier
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Rob Godfrey <ro...@gmail.com>
> >>>> > Sent: vendredi 14 septembre 2018 16:06
> >>>> > To: users@qpid.apache.org
> >>>> > Cc: AYOUBI Ali <Al...@murex.com>
> >>>> > Subject: Re: [Broker-J] JDBC message store performance
> >>>> >
> >>>> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
> >>>> > Olivier.VERMEULEN@murex.com>
> >>>> > wrote:
> >>>> >
> >>>> > > Hello,
> >>>> > >
> >>>> > > We ran a performance test with a bunch of brokers and an
> >>>> > > Oracle database to store the messages.
> >>>> > > We noticed that the database was a bit overloaded with commits.
> >>>> > > Looking at the logs we saw that sending a message was
> >>>> > > triggering 1 commit for 3 operations (QPID_QUEUE_ENTRIES,
> >>>> > > QPID_MESSAGE_METADATA,
> >>>> > > QPID_MESSAGE_CONTENT) which is what we were expecting but
> >>>> > > receiving a message was triggering 2 commits (1 for
> >>>> > > QPID_QUEUE_ENTRIES and 1 for QPID_MESSAGE_METADATA and
> QPID_MESSAGE_CONTENT).
> >>>> > > I debugged a bit the code and saw that in
> >>>> > > AbstractVirtualHost.executeTransaction the delete on
> >>>> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as
> >>>> > > a "post commit" operation, explaining why we have 2 commits.
> >>>> > > Is it something expected? Do you think we could reduce this
> >>>> > > to
> >>>> > > 1 commit when receiving a message?
> >>>> > >
> >>>> >
> >>>> > It's not unexpected - basically the issue is that the broker
> >>>> > needs to cope with the possibility that the same message is
> >>>> > being stored on multiple queues. The first commit is deleting
> >>>> > the referencing of the message from the given queue. The
> >>>> > second commit is occuring after
> >>>> the
> >>>> > message has been definitively removed from the queue, and the
> >>>> > store has determined that there are no more references, so it
> >>>> > is ok to remove the message. This is driven by
> >>>> > reference-counting of the message, and has historically been a
> >>>> > place of many potential race conditions. I'm sure it is
> >>>> > possible to optimise the code in some
> >>>> way,
> >>>> > but it may not be "easy". For the BDB store this doesn't
> >>>> > matter as much as the actual synchronisation to disk of these
> >>>> > operations is coalesced, obviously this is more of an issue for the JDBC store.
> >>>> >
> >>>> > -- Rob
> >>>> >
> >>>> >
> >>>> > >
> >>>> > > Thanks,
> >>>> > > Olivier
> >>>> > > *******************************
> >>>> > >
> >>>> > > This e-mail contains information for the intended recipient only.
> >>>> It
> >>>> > > may contain proprietary material or confidential information.
> >>>> > > If
> >>>> you
> >>>> > > are not the intended recipient you are not authorised to
> >>>> distribute,
> >>>> > > copy or use this e-mail or any attachment to it. Murex cannot
> >>>> > > guarantee that it is virus free and accepts no responsibility
> >>>> > > for any loss or damage arising from its use. If you have
> >>>> > > received this e-mail in error please notify immediately the
> >>>> > > sender and delete the original email received, any
> >>>> > > attachments and all copies from your
> >>>> system.
> >>>> > >
> >>>> > *******************************
> >>>> >
> >>>> > This e-mail contains information for the intended recipient only.
> >>>> > It may contain proprietary material or confidential information.
> >>>> > If you are not the intended recipient you are not authorised to
> >>>> > distribute, copy or use this e-mail or any attachment to it.
> >>>> > Murex cannot guarantee that it is virus free and accepts no
> >>>> > responsibility for any loss or damage arising from its use. If
> >>>> > you have received this e-mail in error please notify
> >>>> > immediately the sender and delete the original email received,
> >>>> > any attachments
> and all copies from your system.
> >>>> >
> >>>> *******************************
> >>>>
> >>>> This e-mail contains information for the intended recipient only.
> >>>> It may contain proprietary material or confidential information.
> >>>> If you are not the intended recipient you are not authorised to
> >>>> distribute, copy or use this e-mail or any attachment to it.
> >>>> Murex cannot guarantee that it is virus free and accepts no
> >>>> responsibility for any loss or damage arising from its use. If
> >>>> you have received this e-mail in error please notify immediately
> >>>> the sender and delete the original email received, any
> >>>> attachments and
> all copies from your system.
> >>>>
> >>>
> *******************************
>
> This e-mail contains information for the intended recipient only. It
> may contain proprietary material or confidential information. If you
> are not the intended recipient you are not authorised to distribute,
> copy or use this e-mail or any attachment to it. Murex cannot
> guarantee that it is virus free and accepts no responsibility for any
> loss or damage arising from its use. If you have received this e-mail
> in error please notify immediately the sender and delete the original
> email received, any attachments and all copies from your system.
>
******************************* This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [Broker-J] JDBC message store performance
Posted by Rob Godfrey <ro...@gmail.com>.
On Tue, 2 Oct 2018 at 17:47, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Hello Rob,
>
> We tested your fix on an Oracle database and it works fine, we even
> noticed a 7% improvement of the overall throughput!
>
Great!
> But the average size of the batches is quite small: between 3 or 4.
> Do you think it would possible (and a good idea) to define a minimum batch
> size and/or a batch period?
>
>
Sure - the only potential "issue" is having more orphan message content
rows in the database if you have an abrupt shutdown of the broker; on
restart these should be cleaned up anyway by the recovery process. I'm not
sure if you would see much performance benefit (my guess is that a lot of
the benefit you will see if from taking the message deletion off the
critical path on message delivery) but we could experiment with a tiny
change to the patch to make a minimum batch size of 10 or whatever and see
if you get a further speedup.
-- Rob
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: mardi 25 septembre 2018 13:20
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] JDBC message store performance
>
> And now with a fix to allow the system tests to pass consistently
>
> https://issues.apache.org/jira/secure/attachment/12941197/QPID-8242.patch
>
> -- Rob
>
> On Tue, 25 Sep 2018 at 12:24, Rob Godfrey <ro...@gmail.com> wrote:
>
> > D'oh - no idea what happened there.
> >
> > Re-attached with an actual diff this time (I hope :) )
> >
> > https://issues.apache.org/jira/secure/attachment/12941192/QPID-8242.pa
> > tch
> >
> > -- Rob
> >
> > On Wed, 19 Sep 2018 at 01:23, Rob Godfrey <ro...@gmail.com>
> wrote:
> >
> >> As an alternative approach, how about the (totally untested) patch
> >> I've attached to QPID-8242 [1]
> >>
> >> Rather than trying to consolidate the commits to delete the queue
> >> entry and the message content it instead attempts to delete the
> >> message content asynchronously and batch up all message contents
> >> awaiting deletion. From a performance point of view, on the broker
> >> it takes the message content deletion off the connection thread, and
> >> if the database is being the bottleneck it should start batching up the
> deletions of content.
> >>
> >> -- Rob
> >>
> >> [1]
> >> https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242.p
> >> atch
> >>
> >> On Mon, 17 Sep 2018 at 15:22, Rob Godfrey <ro...@gmail.com>
> >> wrote:
> >>
> >>> The reference counting is done in AbstractServerMessageImpl.java.
> >>> In general instances of ServerMessage should not be passed around,
> >>> rather a MessageReference (obtained by calling newReference(..) on
> >>> the message object. Then if the message is no longer required in
> >>> that context,
> >>> release() can be called on the reference. Within the context of a
> >>> queue this is normally encapsulated by the notion of the QueueEntry
> >>> (which holds the relevant MessageReference). When delete() is
> >>> called on the QueueEntry (delete being defined in the
> >>> MessageInstance interface), this releases the reference, which
> >>> decrements the reference count, and if the count has gone to zero the
> message calls the store to delete itself.
> >>>
> >>> I think trying to change this logic would be a bit of a nightmare
> >>> (understatement). I think a better alternative in terms of reducing
> >>> the number of commits in the JDBC store might instead be for the
> >>> remove() method on StoredJDBCMessage to (rather than immediately
> >>> commit the message
> >>> delete) schedule the message removal to be picked up by the next
> >>> commit that the store is asked to perform. This would make the
> >>> behaviour more like the BDB store (where we schedule the commit but
> >>> don't actually force the sync to disk on message deletion).
> >>>
> >>> -- Rob
> >>>
> >>> On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
> >>> Olivier.VERMEULEN@murex.com> wrote:
> >>>
> >>>> Ok then I missed something.
> >>>> Whan/Where is the reference-counting you were talking about in your
> >>>> first mail happening?
> >>>>
> >>>> Olivier
> >>>>
> >>>> -----Original Message-----
> >>>> From: Rob Godfrey <ro...@gmail.com>
> >>>> Sent: lundi 17 septembre 2018 14:05
> >>>> To: users@qpid.apache.org
> >>>> Subject: Re: [Broker-J] JDBC message store performance
> >>>>
> >>>> Hi Olivier,
> >>>>
> >>>> the approach you are attempting will not work for the reasons I
> >>>> described previously. If the message has (for instance) been
> >>>> placed in two durable subscription queues (because there are two
> >>>> durable subscriptions) then as soon as the message is consumed from
> >>>> the first queue it would be removed from the store, leading to
> >>>> problems when the second consumer tries to read the message.
> >>>>
> >>>> -- Rob
> >>>>
> >>>> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
> >>>> Olivier.VERMEULEN@murex.com>
> >>>> wrote:
> >>>>
> >>>> > Hello Rob,
> >>>> >
> >>>> > Thanks for the answer.
> >>>> > I started looking at the code to see if there is something I can
> >>>> > do about these 2 commits.
> >>>> > But before going any further I'd like your input on the below, to
> >>>> > see if what I'm trying to do could work or if I'm missing
> >>>> > something
> >>>> (which
> >>>> > I'm surely are)
> >>>> >
> >>>> >
> >>>> https://github.com/apache/qpid-broker-j/compare/master...overmeulen
> >>>> :fe
> >>>> > ature/jdbc-message-store-commits
> >>>> >
> >>>> > Regards,
> >>>> > Olivier
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Rob Godfrey <ro...@gmail.com>
> >>>> > Sent: vendredi 14 septembre 2018 16:06
> >>>> > To: users@qpid.apache.org
> >>>> > Cc: AYOUBI Ali <Al...@murex.com>
> >>>> > Subject: Re: [Broker-J] JDBC message store performance
> >>>> >
> >>>> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
> >>>> > Olivier.VERMEULEN@murex.com>
> >>>> > wrote:
> >>>> >
> >>>> > > Hello,
> >>>> > >
> >>>> > > We ran a performance test with a bunch of brokers and an Oracle
> >>>> > > database to store the messages.
> >>>> > > We noticed that the database was a bit overloaded with commits.
> >>>> > > Looking at the logs we saw that sending a message was
> >>>> > > triggering 1 commit for 3 operations (QPID_QUEUE_ENTRIES,
> >>>> > > QPID_MESSAGE_METADATA,
> >>>> > > QPID_MESSAGE_CONTENT) which is what we were expecting but
> >>>> > > receiving a message was triggering 2 commits (1 for
> >>>> > > QPID_QUEUE_ENTRIES and 1 for QPID_MESSAGE_METADATA and
> QPID_MESSAGE_CONTENT).
> >>>> > > I debugged a bit the code and saw that in
> >>>> > > AbstractVirtualHost.executeTransaction the delete on
> >>>> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a
> >>>> > > "post commit" operation, explaining why we have 2 commits.
> >>>> > > Is it something expected? Do you think we could reduce this to
> >>>> > > 1 commit when receiving a message?
> >>>> > >
> >>>> >
> >>>> > It's not unexpected - basically the issue is that the broker
> >>>> > needs to cope with the possibility that the same message is being
> >>>> > stored on multiple queues. The first commit is deleting the
> >>>> > referencing of the message from the given queue. The second
> >>>> > commit is occuring after
> >>>> the
> >>>> > message has been definitively removed from the queue, and the
> >>>> > store has determined that there are no more references, so it is
> >>>> > ok to remove the message. This is driven by reference-counting
> >>>> > of the message, and has historically been a place of many
> >>>> > potential race conditions. I'm sure it is possible to optimise
> >>>> > the code in some
> >>>> way,
> >>>> > but it may not be "easy". For the BDB store this doesn't matter
> >>>> > as much as the actual synchronisation to disk of these operations
> >>>> > is coalesced, obviously this is more of an issue for the JDBC store.
> >>>> >
> >>>> > -- Rob
> >>>> >
> >>>> >
> >>>> > >
> >>>> > > Thanks,
> >>>> > > Olivier
> >>>> > > *******************************
> >>>> > >
> >>>> > > This e-mail contains information for the intended recipient only.
> >>>> It
> >>>> > > may contain proprietary material or confidential information.
> >>>> > > If
> >>>> you
> >>>> > > are not the intended recipient you are not authorised to
> >>>> distribute,
> >>>> > > copy or use this e-mail or any attachment to it. Murex cannot
> >>>> > > guarantee that it is virus free and accepts no responsibility
> >>>> > > for any loss or damage arising from its use. If you have
> >>>> > > received this e-mail in error please notify immediately the
> >>>> > > sender and delete the original email received, any attachments
> >>>> > > and all copies from your
> >>>> system.
> >>>> > >
> >>>> > *******************************
> >>>> >
> >>>> > This e-mail contains information for the intended recipient only.
> >>>> > It may contain proprietary material or confidential information.
> >>>> > If you are not the intended recipient you are not authorised to
> >>>> > distribute, copy or use this e-mail or any attachment to it.
> >>>> > Murex cannot guarantee that it is virus free and accepts no
> >>>> > responsibility for any loss or damage arising from its use. If
> >>>> > you have received this e-mail in error please notify immediately
> >>>> > the sender and delete the original email received, any attachments
> and all copies from your system.
> >>>> >
> >>>> *******************************
> >>>>
> >>>> This e-mail contains information for the intended recipient only.
> >>>> It may contain proprietary material or confidential information. If
> >>>> you are not the intended recipient you are not authorised to
> >>>> distribute, copy or use this e-mail or any attachment to it. Murex
> >>>> cannot guarantee that it is virus free and accepts no
> >>>> responsibility for any loss or damage arising from its use. If you
> >>>> have received this e-mail in error please notify immediately the
> >>>> sender and delete the original email received, any attachments and
> all copies from your system.
> >>>>
> >>>
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
RE: [Broker-J] JDBC message store performance
Posted by VERMEULEN Olivier <Ol...@murex.com>.
Hello Rob,
We tested your fix on an Oracle database and it works fine, we even noticed a 7% improvement of the overall throughput!
But the average size of the batches is quite small: between 3 or 4.
Do you think it would possible (and a good idea) to define a minimum batch size and/or a batch period?
Olivier
-----Original Message-----
From: Rob Godfrey <ro...@gmail.com>
Sent: mardi 25 septembre 2018 13:20
To: users@qpid.apache.org
Subject: Re: [Broker-J] JDBC message store performance
And now with a fix to allow the system tests to pass consistently
https://issues.apache.org/jira/secure/attachment/12941197/QPID-8242.patch
-- Rob
On Tue, 25 Sep 2018 at 12:24, Rob Godfrey <ro...@gmail.com> wrote:
> D'oh - no idea what happened there.
>
> Re-attached with an actual diff this time (I hope :) )
>
> https://issues.apache.org/jira/secure/attachment/12941192/QPID-8242.pa
> tch
>
> -- Rob
>
> On Wed, 19 Sep 2018 at 01:23, Rob Godfrey <ro...@gmail.com> wrote:
>
>> As an alternative approach, how about the (totally untested) patch
>> I've attached to QPID-8242 [1]
>>
>> Rather than trying to consolidate the commits to delete the queue
>> entry and the message content it instead attempts to delete the
>> message content asynchronously and batch up all message contents
>> awaiting deletion. From a performance point of view, on the broker
>> it takes the message content deletion off the connection thread, and
>> if the database is being the bottleneck it should start batching up the deletions of content.
>>
>> -- Rob
>>
>> [1]
>> https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242.p
>> atch
>>
>> On Mon, 17 Sep 2018 at 15:22, Rob Godfrey <ro...@gmail.com>
>> wrote:
>>
>>> The reference counting is done in AbstractServerMessageImpl.java.
>>> In general instances of ServerMessage should not be passed around,
>>> rather a MessageReference (obtained by calling newReference(..) on
>>> the message object. Then if the message is no longer required in
>>> that context,
>>> release() can be called on the reference. Within the context of a
>>> queue this is normally encapsulated by the notion of the QueueEntry
>>> (which holds the relevant MessageReference). When delete() is
>>> called on the QueueEntry (delete being defined in the
>>> MessageInstance interface), this releases the reference, which
>>> decrements the reference count, and if the count has gone to zero the message calls the store to delete itself.
>>>
>>> I think trying to change this logic would be a bit of a nightmare
>>> (understatement). I think a better alternative in terms of reducing
>>> the number of commits in the JDBC store might instead be for the
>>> remove() method on StoredJDBCMessage to (rather than immediately
>>> commit the message
>>> delete) schedule the message removal to be picked up by the next
>>> commit that the store is asked to perform. This would make the
>>> behaviour more like the BDB store (where we schedule the commit but
>>> don't actually force the sync to disk on message deletion).
>>>
>>> -- Rob
>>>
>>> On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
>>> Olivier.VERMEULEN@murex.com> wrote:
>>>
>>>> Ok then I missed something.
>>>> Whan/Where is the reference-counting you were talking about in your
>>>> first mail happening?
>>>>
>>>> Olivier
>>>>
>>>> -----Original Message-----
>>>> From: Rob Godfrey <ro...@gmail.com>
>>>> Sent: lundi 17 septembre 2018 14:05
>>>> To: users@qpid.apache.org
>>>> Subject: Re: [Broker-J] JDBC message store performance
>>>>
>>>> Hi Olivier,
>>>>
>>>> the approach you are attempting will not work for the reasons I
>>>> described previously. If the message has (for instance) been
>>>> placed in two durable subscription queues (because there are two
>>>> durable subscriptions) then as soon as the message is consumed from
>>>> the first queue it would be removed from the store, leading to
>>>> problems when the second consumer tries to read the message.
>>>>
>>>> -- Rob
>>>>
>>>> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
>>>> Olivier.VERMEULEN@murex.com>
>>>> wrote:
>>>>
>>>> > Hello Rob,
>>>> >
>>>> > Thanks for the answer.
>>>> > I started looking at the code to see if there is something I can
>>>> > do about these 2 commits.
>>>> > But before going any further I'd like your input on the below, to
>>>> > see if what I'm trying to do could work or if I'm missing
>>>> > something
>>>> (which
>>>> > I'm surely are)
>>>> >
>>>> >
>>>> https://github.com/apache/qpid-broker-j/compare/master...overmeulen
>>>> :fe
>>>> > ature/jdbc-message-store-commits
>>>> >
>>>> > Regards,
>>>> > Olivier
>>>> >
>>>> > -----Original Message-----
>>>> > From: Rob Godfrey <ro...@gmail.com>
>>>> > Sent: vendredi 14 septembre 2018 16:06
>>>> > To: users@qpid.apache.org
>>>> > Cc: AYOUBI Ali <Al...@murex.com>
>>>> > Subject: Re: [Broker-J] JDBC message store performance
>>>> >
>>>> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
>>>> > Olivier.VERMEULEN@murex.com>
>>>> > wrote:
>>>> >
>>>> > > Hello,
>>>> > >
>>>> > > We ran a performance test with a bunch of brokers and an Oracle
>>>> > > database to store the messages.
>>>> > > We noticed that the database was a bit overloaded with commits.
>>>> > > Looking at the logs we saw that sending a message was
>>>> > > triggering 1 commit for 3 operations (QPID_QUEUE_ENTRIES,
>>>> > > QPID_MESSAGE_METADATA,
>>>> > > QPID_MESSAGE_CONTENT) which is what we were expecting but
>>>> > > receiving a message was triggering 2 commits (1 for
>>>> > > QPID_QUEUE_ENTRIES and 1 for QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
>>>> > > I debugged a bit the code and saw that in
>>>> > > AbstractVirtualHost.executeTransaction the delete on
>>>> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a
>>>> > > "post commit" operation, explaining why we have 2 commits.
>>>> > > Is it something expected? Do you think we could reduce this to
>>>> > > 1 commit when receiving a message?
>>>> > >
>>>> >
>>>> > It's not unexpected - basically the issue is that the broker
>>>> > needs to cope with the possibility that the same message is being
>>>> > stored on multiple queues. The first commit is deleting the
>>>> > referencing of the message from the given queue. The second
>>>> > commit is occuring after
>>>> the
>>>> > message has been definitively removed from the queue, and the
>>>> > store has determined that there are no more references, so it is
>>>> > ok to remove the message. This is driven by reference-counting
>>>> > of the message, and has historically been a place of many
>>>> > potential race conditions. I'm sure it is possible to optimise
>>>> > the code in some
>>>> way,
>>>> > but it may not be "easy". For the BDB store this doesn't matter
>>>> > as much as the actual synchronisation to disk of these operations
>>>> > is coalesced, obviously this is more of an issue for the JDBC store.
>>>> >
>>>> > -- Rob
>>>> >
>>>> >
>>>> > >
>>>> > > Thanks,
>>>> > > Olivier
>>>> > > *******************************
>>>> > >
>>>> > > This e-mail contains information for the intended recipient only.
>>>> It
>>>> > > may contain proprietary material or confidential information.
>>>> > > If
>>>> you
>>>> > > are not the intended recipient you are not authorised to
>>>> distribute,
>>>> > > copy or use this e-mail or any attachment to it. Murex cannot
>>>> > > guarantee that it is virus free and accepts no responsibility
>>>> > > for any loss or damage arising from its use. If you have
>>>> > > received this e-mail in error please notify immediately the
>>>> > > sender and delete the original email received, any attachments
>>>> > > and all copies from your
>>>> system.
>>>> > >
>>>> > *******************************
>>>> >
>>>> > This e-mail contains information for the intended recipient only.
>>>> > It may contain proprietary material or confidential information.
>>>> > If you are not the intended recipient you are not authorised to
>>>> > distribute, copy or use this e-mail or any attachment to it.
>>>> > Murex cannot guarantee that it is virus free and accepts no
>>>> > responsibility for any loss or damage arising from its use. If
>>>> > you have received this e-mail in error please notify immediately
>>>> > the sender and delete the original email received, any attachments and all copies from your system.
>>>> >
>>>> *******************************
>>>>
>>>> This e-mail contains information for the intended recipient only.
>>>> It may contain proprietary material or confidential information. If
>>>> you are not the intended recipient you are not authorised to
>>>> distribute, copy or use this e-mail or any attachment to it. Murex
>>>> cannot guarantee that it is virus free and accepts no
>>>> responsibility for any loss or damage arising from its use. If you
>>>> have received this e-mail in error please notify immediately the
>>>> sender and delete the original email received, any attachments and all copies from your system.
>>>>
>>>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Broker-J] JDBC message store performance
Posted by Rob Godfrey <ro...@gmail.com>.
And now with a fix to allow the system tests to pass consistently
https://issues.apache.org/jira/secure/attachment/12941197/QPID-8242.patch
-- Rob
On Tue, 25 Sep 2018 at 12:24, Rob Godfrey <ro...@gmail.com> wrote:
> D'oh - no idea what happened there.
>
> Re-attached with an actual diff this time (I hope :) )
>
> https://issues.apache.org/jira/secure/attachment/12941192/QPID-8242.patch
>
> -- Rob
>
> On Wed, 19 Sep 2018 at 01:23, Rob Godfrey <ro...@gmail.com> wrote:
>
>> As an alternative approach, how about the (totally untested) patch I've
>> attached to QPID-8242 [1]
>>
>> Rather than trying to consolidate the commits to delete the queue entry
>> and the message content it instead attempts to delete the message content
>> asynchronously and batch up all message contents awaiting deletion. From a
>> performance point of view, on the broker it takes the message content
>> deletion off the connection thread, and if the database is being the
>> bottleneck it should start batching up the deletions of content.
>>
>> -- Rob
>>
>> [1]
>> https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242.patch
>>
>> On Mon, 17 Sep 2018 at 15:22, Rob Godfrey <ro...@gmail.com>
>> wrote:
>>
>>> The reference counting is done in AbstractServerMessageImpl.java. In
>>> general instances of ServerMessage should not be passed around, rather a
>>> MessageReference (obtained by calling newReference(..) on the message
>>> object. Then if the message is no longer required in that context,
>>> release() can be called on the reference. Within the context of a queue
>>> this is normally encapsulated by the notion of the QueueEntry (which holds
>>> the relevant MessageReference). When delete() is called on the QueueEntry
>>> (delete being defined in the MessageInstance interface), this releases the
>>> reference, which decrements the reference count, and if the count has gone
>>> to zero the message calls the store to delete itself.
>>>
>>> I think trying to change this logic would be a bit of a nightmare
>>> (understatement). I think a better alternative in terms of reducing the
>>> number of commits in the JDBC store might instead be for the remove()
>>> method on StoredJDBCMessage to (rather than immediately commit the message
>>> delete) schedule the message removal to be picked up by the next commit
>>> that the store is asked to perform. This would make the behaviour more
>>> like the BDB store (where we schedule the commit but don't actually force
>>> the sync to disk on message deletion).
>>>
>>> -- Rob
>>>
>>> On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
>>> Olivier.VERMEULEN@murex.com> wrote:
>>>
>>>> Ok then I missed something.
>>>> Whan/Where is the reference-counting you were talking about in your
>>>> first mail happening?
>>>>
>>>> Olivier
>>>>
>>>> -----Original Message-----
>>>> From: Rob Godfrey <ro...@gmail.com>
>>>> Sent: lundi 17 septembre 2018 14:05
>>>> To: users@qpid.apache.org
>>>> Subject: Re: [Broker-J] JDBC message store performance
>>>>
>>>> Hi Olivier,
>>>>
>>>> the approach you are attempting will not work for the reasons I
>>>> described previously. If the message has (for instance) been placed in two
>>>> durable subscription queues (because there are two durable subscriptions)
>>>> then as soon as the message is consumed from the first queue it would be
>>>> removed from the store, leading to problems when the second consumer tries
>>>> to read the message.
>>>>
>>>> -- Rob
>>>>
>>>> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
>>>> Olivier.VERMEULEN@murex.com>
>>>> wrote:
>>>>
>>>> > Hello Rob,
>>>> >
>>>> > Thanks for the answer.
>>>> > I started looking at the code to see if there is something I can do
>>>> > about these 2 commits.
>>>> > But before going any further I'd like your input on the below, to see
>>>> > if what I'm trying to do could work or if I'm missing something
>>>> (which
>>>> > I'm surely are)
>>>> >
>>>> >
>>>> https://github.com/apache/qpid-broker-j/compare/master...overmeulen:fe
>>>> > ature/jdbc-message-store-commits
>>>> >
>>>> > Regards,
>>>> > Olivier
>>>> >
>>>> > -----Original Message-----
>>>> > From: Rob Godfrey <ro...@gmail.com>
>>>> > Sent: vendredi 14 septembre 2018 16:06
>>>> > To: users@qpid.apache.org
>>>> > Cc: AYOUBI Ali <Al...@murex.com>
>>>> > Subject: Re: [Broker-J] JDBC message store performance
>>>> >
>>>> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
>>>> > Olivier.VERMEULEN@murex.com>
>>>> > wrote:
>>>> >
>>>> > > Hello,
>>>> > >
>>>> > > We ran a performance test with a bunch of brokers and an Oracle
>>>> > > database to store the messages.
>>>> > > We noticed that the database was a bit overloaded with commits.
>>>> > > Looking at the logs we saw that sending a message was triggering 1
>>>> > > commit for 3 operations (QPID_QUEUE_ENTRIES, QPID_MESSAGE_METADATA,
>>>> > > QPID_MESSAGE_CONTENT) which is what we were expecting but receiving
>>>> > > a message was triggering 2 commits (1 for QPID_QUEUE_ENTRIES and 1
>>>> > > for QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
>>>> > > I debugged a bit the code and saw that in
>>>> > > AbstractVirtualHost.executeTransaction the delete on
>>>> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a
>>>> > > "post commit" operation, explaining why we have 2 commits.
>>>> > > Is it something expected? Do you think we could reduce this to 1
>>>> > > commit when receiving a message?
>>>> > >
>>>> >
>>>> > It's not unexpected - basically the issue is that the broker needs to
>>>> > cope with the possibility that the same message is being stored on
>>>> > multiple queues. The first commit is deleting the referencing of the
>>>> > message from the given queue. The second commit is occuring after
>>>> the
>>>> > message has been definitively removed from the queue, and the store
>>>> > has determined that there are no more references, so it is ok to
>>>> > remove the message. This is driven by reference-counting of the
>>>> > message, and has historically been a place of many potential race
>>>> > conditions. I'm sure it is possible to optimise the code in some
>>>> way,
>>>> > but it may not be "easy". For the BDB store this doesn't matter as
>>>> > much as the actual synchronisation to disk of these operations is
>>>> > coalesced, obviously this is more of an issue for the JDBC store.
>>>> >
>>>> > -- Rob
>>>> >
>>>> >
>>>> > >
>>>> > > Thanks,
>>>> > > Olivier
>>>> > > *******************************
>>>> > >
>>>> > > This e-mail contains information for the intended recipient only.
>>>> It
>>>> > > may contain proprietary material or confidential information. If
>>>> you
>>>> > > are not the intended recipient you are not authorised to
>>>> distribute,
>>>> > > copy or use this e-mail or any attachment to it. Murex cannot
>>>> > > guarantee that it is virus free and accepts no responsibility for
>>>> > > any loss or damage arising from its use. If you have received this
>>>> > > e-mail in error please notify immediately the sender and delete the
>>>> > > original email received, any attachments and all copies from your
>>>> system.
>>>> > >
>>>> > *******************************
>>>> >
>>>> > This e-mail contains information for the intended recipient only. It
>>>> > may contain proprietary material or confidential information. If you
>>>> > are not the intended recipient you are not authorised to distribute,
>>>> > copy or use this e-mail or any attachment to it. Murex cannot
>>>> > guarantee that it is virus free and accepts no responsibility for any
>>>> > loss or damage arising from its use. If you have received this e-mail
>>>> > in error please notify immediately the sender and delete the original
>>>> > email received, any attachments and all copies from your system.
>>>> >
>>>> *******************************
>>>>
>>>> This e-mail contains information for the intended recipient only. It
>>>> may contain proprietary material or confidential information. If you are
>>>> not the intended recipient you are not authorised to distribute, copy or
>>>> use this e-mail or any attachment to it. Murex cannot guarantee that it is
>>>> virus free and accepts no responsibility for any loss or damage arising
>>>> from its use. If you have received this e-mail in error please notify
>>>> immediately the sender and delete the original email received, any
>>>> attachments and all copies from your system.
>>>>
>>>
Re: [Broker-J] JDBC message store performance
Posted by Rob Godfrey <ro...@gmail.com>.
D'oh - no idea what happened there.
Re-attached with an actual diff this time (I hope :) )
https://issues.apache.org/jira/secure/attachment/12941192/QPID-8242.patch
-- Rob
On Wed, 19 Sep 2018 at 01:23, Rob Godfrey <ro...@gmail.com> wrote:
> As an alternative approach, how about the (totally untested) patch I've
> attached to QPID-8242 [1]
>
> Rather than trying to consolidate the commits to delete the queue entry
> and the message content it instead attempts to delete the message content
> asynchronously and batch up all message contents awaiting deletion. From a
> performance point of view, on the broker it takes the message content
> deletion off the connection thread, and if the database is being the
> bottleneck it should start batching up the deletions of content.
>
> -- Rob
>
> [1]
> https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242.patch
>
> On Mon, 17 Sep 2018 at 15:22, Rob Godfrey <ro...@gmail.com> wrote:
>
>> The reference counting is done in AbstractServerMessageImpl.java. In
>> general instances of ServerMessage should not be passed around, rather a
>> MessageReference (obtained by calling newReference(..) on the message
>> object. Then if the message is no longer required in that context,
>> release() can be called on the reference. Within the context of a queue
>> this is normally encapsulated by the notion of the QueueEntry (which holds
>> the relevant MessageReference). When delete() is called on the QueueEntry
>> (delete being defined in the MessageInstance interface), this releases the
>> reference, which decrements the reference count, and if the count has gone
>> to zero the message calls the store to delete itself.
>>
>> I think trying to change this logic would be a bit of a nightmare
>> (understatement). I think a better alternative in terms of reducing the
>> number of commits in the JDBC store might instead be for the remove()
>> method on StoredJDBCMessage to (rather than immediately commit the message
>> delete) schedule the message removal to be picked up by the next commit
>> that the store is asked to perform. This would make the behaviour more
>> like the BDB store (where we schedule the commit but don't actually force
>> the sync to disk on message deletion).
>>
>> -- Rob
>>
>> On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
>> Olivier.VERMEULEN@murex.com> wrote:
>>
>>> Ok then I missed something.
>>> Whan/Where is the reference-counting you were talking about in your
>>> first mail happening?
>>>
>>> Olivier
>>>
>>> -----Original Message-----
>>> From: Rob Godfrey <ro...@gmail.com>
>>> Sent: lundi 17 septembre 2018 14:05
>>> To: users@qpid.apache.org
>>> Subject: Re: [Broker-J] JDBC message store performance
>>>
>>> Hi Olivier,
>>>
>>> the approach you are attempting will not work for the reasons I
>>> described previously. If the message has (for instance) been placed in two
>>> durable subscription queues (because there are two durable subscriptions)
>>> then as soon as the message is consumed from the first queue it would be
>>> removed from the store, leading to problems when the second consumer tries
>>> to read the message.
>>>
>>> -- Rob
>>>
>>> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
>>> Olivier.VERMEULEN@murex.com>
>>> wrote:
>>>
>>> > Hello Rob,
>>> >
>>> > Thanks for the answer.
>>> > I started looking at the code to see if there is something I can do
>>> > about these 2 commits.
>>> > But before going any further I'd like your input on the below, to see
>>> > if what I'm trying to do could work or if I'm missing something (which
>>> > I'm surely are)
>>> >
>>> > https://github.com/apache/qpid-broker-j/compare/master...overmeulen:fe
>>> > ature/jdbc-message-store-commits
>>> >
>>> > Regards,
>>> > Olivier
>>> >
>>> > -----Original Message-----
>>> > From: Rob Godfrey <ro...@gmail.com>
>>> > Sent: vendredi 14 septembre 2018 16:06
>>> > To: users@qpid.apache.org
>>> > Cc: AYOUBI Ali <Al...@murex.com>
>>> > Subject: Re: [Broker-J] JDBC message store performance
>>> >
>>> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
>>> > Olivier.VERMEULEN@murex.com>
>>> > wrote:
>>> >
>>> > > Hello,
>>> > >
>>> > > We ran a performance test with a bunch of brokers and an Oracle
>>> > > database to store the messages.
>>> > > We noticed that the database was a bit overloaded with commits.
>>> > > Looking at the logs we saw that sending a message was triggering 1
>>> > > commit for 3 operations (QPID_QUEUE_ENTRIES, QPID_MESSAGE_METADATA,
>>> > > QPID_MESSAGE_CONTENT) which is what we were expecting but receiving
>>> > > a message was triggering 2 commits (1 for QPID_QUEUE_ENTRIES and 1
>>> > > for QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
>>> > > I debugged a bit the code and saw that in
>>> > > AbstractVirtualHost.executeTransaction the delete on
>>> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a
>>> > > "post commit" operation, explaining why we have 2 commits.
>>> > > Is it something expected? Do you think we could reduce this to 1
>>> > > commit when receiving a message?
>>> > >
>>> >
>>> > It's not unexpected - basically the issue is that the broker needs to
>>> > cope with the possibility that the same message is being stored on
>>> > multiple queues. The first commit is deleting the referencing of the
>>> > message from the given queue. The second commit is occuring after the
>>> > message has been definitively removed from the queue, and the store
>>> > has determined that there are no more references, so it is ok to
>>> > remove the message. This is driven by reference-counting of the
>>> > message, and has historically been a place of many potential race
>>> > conditions. I'm sure it is possible to optimise the code in some way,
>>> > but it may not be "easy". For the BDB store this doesn't matter as
>>> > much as the actual synchronisation to disk of these operations is
>>> > coalesced, obviously this is more of an issue for the JDBC store.
>>> >
>>> > -- Rob
>>> >
>>> >
>>> > >
>>> > > Thanks,
>>> > > Olivier
>>> > > *******************************
>>> > >
>>> > > This e-mail contains information for the intended recipient only. It
>>> > > may contain proprietary material or confidential information. If you
>>> > > are not the intended recipient you are not authorised to distribute,
>>> > > copy or use this e-mail or any attachment to it. Murex cannot
>>> > > guarantee that it is virus free and accepts no responsibility for
>>> > > any loss or damage arising from its use. If you have received this
>>> > > e-mail in error please notify immediately the sender and delete the
>>> > > original email received, any attachments and all copies from your
>>> system.
>>> > >
>>> > *******************************
>>> >
>>> > This e-mail contains information for the intended recipient only. It
>>> > may contain proprietary material or confidential information. If you
>>> > are not the intended recipient you are not authorised to distribute,
>>> > copy or use this e-mail or any attachment to it. Murex cannot
>>> > guarantee that it is virus free and accepts no responsibility for any
>>> > loss or damage arising from its use. If you have received this e-mail
>>> > in error please notify immediately the sender and delete the original
>>> > email received, any attachments and all copies from your system.
>>> >
>>> *******************************
>>>
>>> This e-mail contains information for the intended recipient only. It may
>>> contain proprietary material or confidential information. If you are not
>>> the intended recipient you are not authorised to distribute, copy or use
>>> this e-mail or any attachment to it. Murex cannot guarantee that it is
>>> virus free and accepts no responsibility for any loss or damage arising
>>> from its use. If you have received this e-mail in error please notify
>>> immediately the sender and delete the original email received, any
>>> attachments and all copies from your system.
>>>
>>
RE: [Broker-J] JDBC message store performance
Posted by VERMEULEN Olivier <Ol...@murex.com>.
Hello Rob,
I think there is something wrong with your patch, it only contains the git-diff manual...
Olivier
-----Original Message-----
From: Rob Godfrey <ro...@gmail.com>
Sent: mercredi 19 septembre 2018 09:31
To: users@qpid.apache.org
Subject: Re: [Broker-J] JDBC message store performance
I'm on vacation at the moment, but if I get a chance I'll at least try to test with Derby.
-- Rob
On Wed, 19 Sep 2018 at 09:14, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Thanks a lot Rob, I'll try to test that next week.
>
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: mercredi 19 septembre 2018 01:23
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] JDBC message store performance
>
> As an alternative approach, how about the (totally untested) patch
> I've attached to QPID-8242 [1]
>
> Rather than trying to consolidate the commits to delete the queue
> entry and the message content it instead attempts to delete the
> message content asynchronously and batch up all message contents
> awaiting deletion. From a performance point of view, on the broker it
> takes the message content deletion off the connection thread, and if
> the database is being the bottleneck it should start batching up the deletions of content.
>
> -- Rob
>
> [1]
> https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242.pa
> tch
>
> On Mon, 17 Sep 2018 at 15:22, Rob Godfrey <ro...@gmail.com> wrote:
>
> > The reference counting is done in AbstractServerMessageImpl.java.
> > In general instances of ServerMessage should not be passed around,
> > rather a MessageReference (obtained by calling newReference(..) on
> > the message object. Then if the message is no longer required in
> > that context,
> > release() can be called on the reference. Within the context of a
> > queue this is normally encapsulated by the notion of the QueueEntry
> > (which holds the relevant MessageReference). When delete() is
> > called on the QueueEntry (delete being defined in the
> > MessageInstance interface), this releases the reference, which
> > decrements the reference count, and if the count has gone to zero
> > the message calls the
> store to delete itself.
> >
> > I think trying to change this logic would be a bit of a nightmare
> > (understatement). I think a better alternative in terms of reducing
> > the number of commits in the JDBC store might instead be for the
> > remove() method on StoredJDBCMessage to (rather than immediately
> > commit the message
> > delete) schedule the message removal to be picked up by the next
> > commit that the store is asked to perform. This would make the
> > behaviour more like the BDB store (where we schedule the commit but
> > don't actually force the sync to disk on message deletion).
> >
> > -- Rob
> >
> > On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com> wrote:
> >
> >> Ok then I missed something.
> >> Whan/Where is the reference-counting you were talking about in your
> >> first mail happening?
> >>
> >> Olivier
> >>
> >> -----Original Message-----
> >> From: Rob Godfrey <ro...@gmail.com>
> >> Sent: lundi 17 septembre 2018 14:05
> >> To: users@qpid.apache.org
> >> Subject: Re: [Broker-J] JDBC message store performance
> >>
> >> Hi Olivier,
> >>
> >> the approach you are attempting will not work for the reasons I
> >> described previously. If the message has (for instance) been
> >> placed in two durable subscription queues (because there are two
> >> durable
> >> subscriptions) then as soon as the message is consumed from the
> >> first queue it would be removed from the store, leading to problems
> >> when the second consumer tries to read the message.
> >>
> >> -- Rob
> >>
> >> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
> >> Olivier.VERMEULEN@murex.com>
> >> wrote:
> >>
> >> > Hello Rob,
> >> >
> >> > Thanks for the answer.
> >> > I started looking at the code to see if there is something I can
> >> > do about these 2 commits.
> >> > But before going any further I'd like your input on the below, to
> >> > see if what I'm trying to do could work or if I'm missing
> >> > something (which I'm surely are)
> >> >
> >> > https://github.com/apache/qpid-broker-j/compare/master...overmeul
> >> > en
> >> > :fe
> >> > ature/jdbc-message-store-commits
> >> >
> >> > Regards,
> >> > Olivier
> >> >
> >> > -----Original Message-----
> >> > From: Rob Godfrey <ro...@gmail.com>
> >> > Sent: vendredi 14 septembre 2018 16:06
> >> > To: users@qpid.apache.org
> >> > Cc: AYOUBI Ali <Al...@murex.com>
> >> > Subject: Re: [Broker-J] JDBC message store performance
> >> >
> >> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
> >> > Olivier.VERMEULEN@murex.com>
> >> > wrote:
> >> >
> >> > > Hello,
> >> > >
> >> > > We ran a performance test with a bunch of brokers and an Oracle
> >> > > database to store the messages.
> >> > > We noticed that the database was a bit overloaded with commits.
> >> > > Looking at the logs we saw that sending a message was
> >> > > triggering
> >> > > 1 commit for 3 operations (QPID_QUEUE_ENTRIES,
> >> > > QPID_MESSAGE_METADATA,
> >> > > QPID_MESSAGE_CONTENT) which is what we were expecting but
> >> > > receiving a message was triggering 2 commits (1 for
> >> > > QPID_QUEUE_ENTRIES and 1 for QPID_MESSAGE_METADATA and
> QPID_MESSAGE_CONTENT).
> >> > > I debugged a bit the code and saw that in
> >> > > AbstractVirtualHost.executeTransaction the delete on
> >> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a
> >> > > "post commit" operation, explaining why we have 2 commits.
> >> > > Is it something expected? Do you think we could reduce this to
> >> > > 1 commit when receiving a message?
> >> > >
> >> >
> >> > It's not unexpected - basically the issue is that the broker
> >> > needs to cope with the possibility that the same message is being
> >> > stored on multiple queues. The first commit is deleting the
> >> > referencing of the message from the given queue. The second
> >> > commit is occuring after the message has been definitively
> >> > removed from the queue, and the store has determined that there
> >> > are no more references, so it is ok to remove the message. This
> >> > is driven by reference-counting of the message, and has
> >> > historically been a place of many potential race conditions. I'm
> >> > sure it is possible to optimise the code in some way, but it may
> >> > not be "easy". For the BDB store this doesn't matter as much as
> >> > the actual synchronisation to disk of these operations is
> >> > coalesced, obviously this is more of an issue for the
> JDBC store.
> >> >
> >> > -- Rob
> >> >
> >> >
> >> > >
> >> > > Thanks,
> >> > > Olivier
> >> > > *******************************
> >> > >
> >> > > This e-mail contains information for the intended recipient only.
> >> > > It may contain proprietary material or confidential information.
> >> > > If you are not the intended recipient you are not authorised to
> >> > > distribute, copy or use this e-mail or any attachment to it.
> >> > > Murex cannot guarantee that it is virus free and accepts no
> >> > > responsibility for any loss or damage arising from its use. If
> >> > > you have received this e-mail in error please notify
> >> > > immediately the sender and delete the original email received,
> >> > > any attachments and all copies from your
> >> system.
> >> > >
> >> > *******************************
> >> >
> >> > This e-mail contains information for the intended recipient only.
> >> > It may contain proprietary material or confidential information.
> >> > If you are not the intended recipient you are not authorised to
> >> > distribute, copy or use this e-mail or any attachment to it.
> >> > Murex cannot guarantee that it is virus free and accepts no
> >> > responsibility for any loss or damage arising from its use. If
> >> > you have received this e-mail in error please notify immediately
> >> > the sender and delete the original email received, any
> >> > attachments and
> all copies from your system.
> >> >
> >> *******************************
> >>
> >> This e-mail contains information for the intended recipient only.
> >> It may contain proprietary material or confidential information. If
> >> you are not the intended recipient you are not authorised to
> >> distribute, copy or use this e-mail or any attachment to it. Murex
> >> cannot guarantee that it is virus free and accepts no
> >> responsibility for any loss or damage arising from its use. If you
> >> have received this e-mail in error please notify immediately the
> >> sender and delete the original email received, any attachments and all copies from your system.
> >>
> >
> *******************************
>
> This e-mail contains information for the intended recipient only. It
> may contain proprietary material or confidential information. If you
> are not the intended recipient you are not authorised to distribute,
> copy or use this e-mail or any attachment to it. Murex cannot
> guarantee that it is virus free and accepts no responsibility for any
> loss or damage arising from its use. If you have received this e-mail
> in error please notify immediately the sender and delete the original
> email received, any attachments and all copies from your system.
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Broker-J] JDBC message store performance
Posted by Rob Godfrey <ro...@gmail.com>.
I'm on vacation at the moment, but if I get a chance I'll at least try to
test with Derby.
-- Rob
On Wed, 19 Sep 2018 at 09:14, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Thanks a lot Rob, I'll try to test that next week.
>
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: mercredi 19 septembre 2018 01:23
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] JDBC message store performance
>
> As an alternative approach, how about the (totally untested) patch I've
> attached to QPID-8242 [1]
>
> Rather than trying to consolidate the commits to delete the queue entry
> and the message content it instead attempts to delete the message content
> asynchronously and batch up all message contents awaiting deletion. From a
> performance point of view, on the broker it takes the message content
> deletion off the connection thread, and if the database is being the
> bottleneck it should start batching up the deletions of content.
>
> -- Rob
>
> [1]
> https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242.patch
>
> On Mon, 17 Sep 2018 at 15:22, Rob Godfrey <ro...@gmail.com> wrote:
>
> > The reference counting is done in AbstractServerMessageImpl.java. In
> > general instances of ServerMessage should not be passed around, rather
> > a MessageReference (obtained by calling newReference(..) on the
> > message object. Then if the message is no longer required in that
> > context,
> > release() can be called on the reference. Within the context of a
> > queue this is normally encapsulated by the notion of the QueueEntry
> > (which holds the relevant MessageReference). When delete() is called
> > on the QueueEntry (delete being defined in the MessageInstance
> > interface), this releases the reference, which decrements the
> > reference count, and if the count has gone to zero the message calls the
> store to delete itself.
> >
> > I think trying to change this logic would be a bit of a nightmare
> > (understatement). I think a better alternative in terms of reducing
> > the number of commits in the JDBC store might instead be for the
> > remove() method on StoredJDBCMessage to (rather than immediately
> > commit the message
> > delete) schedule the message removal to be picked up by the next
> > commit that the store is asked to perform. This would make the
> > behaviour more like the BDB store (where we schedule the commit but
> > don't actually force the sync to disk on message deletion).
> >
> > -- Rob
> >
> > On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com> wrote:
> >
> >> Ok then I missed something.
> >> Whan/Where is the reference-counting you were talking about in your
> >> first mail happening?
> >>
> >> Olivier
> >>
> >> -----Original Message-----
> >> From: Rob Godfrey <ro...@gmail.com>
> >> Sent: lundi 17 septembre 2018 14:05
> >> To: users@qpid.apache.org
> >> Subject: Re: [Broker-J] JDBC message store performance
> >>
> >> Hi Olivier,
> >>
> >> the approach you are attempting will not work for the reasons I
> >> described previously. If the message has (for instance) been placed
> >> in two durable subscription queues (because there are two durable
> >> subscriptions) then as soon as the message is consumed from the first
> >> queue it would be removed from the store, leading to problems when
> >> the second consumer tries to read the message.
> >>
> >> -- Rob
> >>
> >> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
> >> Olivier.VERMEULEN@murex.com>
> >> wrote:
> >>
> >> > Hello Rob,
> >> >
> >> > Thanks for the answer.
> >> > I started looking at the code to see if there is something I can do
> >> > about these 2 commits.
> >> > But before going any further I'd like your input on the below, to
> >> > see if what I'm trying to do could work or if I'm missing something
> >> > (which I'm surely are)
> >> >
> >> > https://github.com/apache/qpid-broker-j/compare/master...overmeulen
> >> > :fe
> >> > ature/jdbc-message-store-commits
> >> >
> >> > Regards,
> >> > Olivier
> >> >
> >> > -----Original Message-----
> >> > From: Rob Godfrey <ro...@gmail.com>
> >> > Sent: vendredi 14 septembre 2018 16:06
> >> > To: users@qpid.apache.org
> >> > Cc: AYOUBI Ali <Al...@murex.com>
> >> > Subject: Re: [Broker-J] JDBC message store performance
> >> >
> >> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
> >> > Olivier.VERMEULEN@murex.com>
> >> > wrote:
> >> >
> >> > > Hello,
> >> > >
> >> > > We ran a performance test with a bunch of brokers and an Oracle
> >> > > database to store the messages.
> >> > > We noticed that the database was a bit overloaded with commits.
> >> > > Looking at the logs we saw that sending a message was triggering
> >> > > 1 commit for 3 operations (QPID_QUEUE_ENTRIES,
> >> > > QPID_MESSAGE_METADATA,
> >> > > QPID_MESSAGE_CONTENT) which is what we were expecting but
> >> > > receiving a message was triggering 2 commits (1 for
> >> > > QPID_QUEUE_ENTRIES and 1 for QPID_MESSAGE_METADATA and
> QPID_MESSAGE_CONTENT).
> >> > > I debugged a bit the code and saw that in
> >> > > AbstractVirtualHost.executeTransaction the delete on
> >> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a
> >> > > "post commit" operation, explaining why we have 2 commits.
> >> > > Is it something expected? Do you think we could reduce this to 1
> >> > > commit when receiving a message?
> >> > >
> >> >
> >> > It's not unexpected - basically the issue is that the broker needs
> >> > to cope with the possibility that the same message is being stored
> >> > on multiple queues. The first commit is deleting the referencing
> >> > of the message from the given queue. The second commit is occuring
> >> > after the message has been definitively removed from the queue, and
> >> > the store has determined that there are no more references, so it
> >> > is ok to remove the message. This is driven by reference-counting
> >> > of the message, and has historically been a place of many potential
> >> > race conditions. I'm sure it is possible to optimise the code in
> >> > some way, but it may not be "easy". For the BDB store this doesn't
> >> > matter as much as the actual synchronisation to disk of these
> >> > operations is coalesced, obviously this is more of an issue for the
> JDBC store.
> >> >
> >> > -- Rob
> >> >
> >> >
> >> > >
> >> > > Thanks,
> >> > > Olivier
> >> > > *******************************
> >> > >
> >> > > This e-mail contains information for the intended recipient only.
> >> > > It may contain proprietary material or confidential information.
> >> > > If you are not the intended recipient you are not authorised to
> >> > > distribute, copy or use this e-mail or any attachment to it.
> >> > > Murex cannot guarantee that it is virus free and accepts no
> >> > > responsibility for any loss or damage arising from its use. If
> >> > > you have received this e-mail in error please notify immediately
> >> > > the sender and delete the original email received, any
> >> > > attachments and all copies from your
> >> system.
> >> > >
> >> > *******************************
> >> >
> >> > This e-mail contains information for the intended recipient only.
> >> > It may contain proprietary material or confidential information. If
> >> > you are not the intended recipient you are not authorised to
> >> > distribute, copy or use this e-mail or any attachment to it. Murex
> >> > cannot guarantee that it is virus free and accepts no
> >> > responsibility for any loss or damage arising from its use. If you
> >> > have received this e-mail in error please notify immediately the
> >> > sender and delete the original email received, any attachments and
> all copies from your system.
> >> >
> >> *******************************
> >>
> >> This e-mail contains information for the intended recipient only. It
> >> may contain proprietary material or confidential information. If you
> >> are not the intended recipient you are not authorised to distribute,
> >> copy or use this e-mail or any attachment to it. Murex cannot
> >> guarantee that it is virus free and accepts no responsibility for any
> >> loss or damage arising from its use. If you have received this e-mail
> >> in error please notify immediately the sender and delete the original
> >> email received, any attachments and all copies from your system.
> >>
> >
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
RE: [Broker-J] JDBC message store performance
Posted by VERMEULEN Olivier <Ol...@murex.com>.
Thanks a lot Rob, I'll try to test that next week.
Olivier
-----Original Message-----
From: Rob Godfrey <ro...@gmail.com>
Sent: mercredi 19 septembre 2018 01:23
To: users@qpid.apache.org
Subject: Re: [Broker-J] JDBC message store performance
As an alternative approach, how about the (totally untested) patch I've attached to QPID-8242 [1]
Rather than trying to consolidate the commits to delete the queue entry and the message content it instead attempts to delete the message content asynchronously and batch up all message contents awaiting deletion. From a performance point of view, on the broker it takes the message content deletion off the connection thread, and if the database is being the bottleneck it should start batching up the deletions of content.
-- Rob
[1]
https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242.patch
On Mon, 17 Sep 2018 at 15:22, Rob Godfrey <ro...@gmail.com> wrote:
> The reference counting is done in AbstractServerMessageImpl.java. In
> general instances of ServerMessage should not be passed around, rather
> a MessageReference (obtained by calling newReference(..) on the
> message object. Then if the message is no longer required in that
> context,
> release() can be called on the reference. Within the context of a
> queue this is normally encapsulated by the notion of the QueueEntry
> (which holds the relevant MessageReference). When delete() is called
> on the QueueEntry (delete being defined in the MessageInstance
> interface), this releases the reference, which decrements the
> reference count, and if the count has gone to zero the message calls the store to delete itself.
>
> I think trying to change this logic would be a bit of a nightmare
> (understatement). I think a better alternative in terms of reducing
> the number of commits in the JDBC store might instead be for the
> remove() method on StoredJDBCMessage to (rather than immediately
> commit the message
> delete) schedule the message removal to be picked up by the next
> commit that the store is asked to perform. This would make the
> behaviour more like the BDB store (where we schedule the commit but
> don't actually force the sync to disk on message deletion).
>
> -- Rob
>
> On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com> wrote:
>
>> Ok then I missed something.
>> Whan/Where is the reference-counting you were talking about in your
>> first mail happening?
>>
>> Olivier
>>
>> -----Original Message-----
>> From: Rob Godfrey <ro...@gmail.com>
>> Sent: lundi 17 septembre 2018 14:05
>> To: users@qpid.apache.org
>> Subject: Re: [Broker-J] JDBC message store performance
>>
>> Hi Olivier,
>>
>> the approach you are attempting will not work for the reasons I
>> described previously. If the message has (for instance) been placed
>> in two durable subscription queues (because there are two durable
>> subscriptions) then as soon as the message is consumed from the first
>> queue it would be removed from the store, leading to problems when
>> the second consumer tries to read the message.
>>
>> -- Rob
>>
>> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
>> Olivier.VERMEULEN@murex.com>
>> wrote:
>>
>> > Hello Rob,
>> >
>> > Thanks for the answer.
>> > I started looking at the code to see if there is something I can do
>> > about these 2 commits.
>> > But before going any further I'd like your input on the below, to
>> > see if what I'm trying to do could work or if I'm missing something
>> > (which I'm surely are)
>> >
>> > https://github.com/apache/qpid-broker-j/compare/master...overmeulen
>> > :fe
>> > ature/jdbc-message-store-commits
>> >
>> > Regards,
>> > Olivier
>> >
>> > -----Original Message-----
>> > From: Rob Godfrey <ro...@gmail.com>
>> > Sent: vendredi 14 septembre 2018 16:06
>> > To: users@qpid.apache.org
>> > Cc: AYOUBI Ali <Al...@murex.com>
>> > Subject: Re: [Broker-J] JDBC message store performance
>> >
>> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
>> > Olivier.VERMEULEN@murex.com>
>> > wrote:
>> >
>> > > Hello,
>> > >
>> > > We ran a performance test with a bunch of brokers and an Oracle
>> > > database to store the messages.
>> > > We noticed that the database was a bit overloaded with commits.
>> > > Looking at the logs we saw that sending a message was triggering
>> > > 1 commit for 3 operations (QPID_QUEUE_ENTRIES,
>> > > QPID_MESSAGE_METADATA,
>> > > QPID_MESSAGE_CONTENT) which is what we were expecting but
>> > > receiving a message was triggering 2 commits (1 for
>> > > QPID_QUEUE_ENTRIES and 1 for QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
>> > > I debugged a bit the code and saw that in
>> > > AbstractVirtualHost.executeTransaction the delete on
>> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a
>> > > "post commit" operation, explaining why we have 2 commits.
>> > > Is it something expected? Do you think we could reduce this to 1
>> > > commit when receiving a message?
>> > >
>> >
>> > It's not unexpected - basically the issue is that the broker needs
>> > to cope with the possibility that the same message is being stored
>> > on multiple queues. The first commit is deleting the referencing
>> > of the message from the given queue. The second commit is occuring
>> > after the message has been definitively removed from the queue, and
>> > the store has determined that there are no more references, so it
>> > is ok to remove the message. This is driven by reference-counting
>> > of the message, and has historically been a place of many potential
>> > race conditions. I'm sure it is possible to optimise the code in
>> > some way, but it may not be "easy". For the BDB store this doesn't
>> > matter as much as the actual synchronisation to disk of these
>> > operations is coalesced, obviously this is more of an issue for the JDBC store.
>> >
>> > -- Rob
>> >
>> >
>> > >
>> > > Thanks,
>> > > Olivier
>> > > *******************************
>> > >
>> > > This e-mail contains information for the intended recipient only.
>> > > It may contain proprietary material or confidential information.
>> > > If you are not the intended recipient you are not authorised to
>> > > distribute, copy or use this e-mail or any attachment to it.
>> > > Murex cannot guarantee that it is virus free and accepts no
>> > > responsibility for any loss or damage arising from its use. If
>> > > you have received this e-mail in error please notify immediately
>> > > the sender and delete the original email received, any
>> > > attachments and all copies from your
>> system.
>> > >
>> > *******************************
>> >
>> > This e-mail contains information for the intended recipient only.
>> > It may contain proprietary material or confidential information. If
>> > you are not the intended recipient you are not authorised to
>> > distribute, copy or use this e-mail or any attachment to it. Murex
>> > cannot guarantee that it is virus free and accepts no
>> > responsibility for any loss or damage arising from its use. If you
>> > have received this e-mail in error please notify immediately the
>> > sender and delete the original email received, any attachments and all copies from your system.
>> >
>> *******************************
>>
>> This e-mail contains information for the intended recipient only. It
>> may contain proprietary material or confidential information. If you
>> are not the intended recipient you are not authorised to distribute,
>> copy or use this e-mail or any attachment to it. Murex cannot
>> guarantee that it is virus free and accepts no responsibility for any
>> loss or damage arising from its use. If you have received this e-mail
>> in error please notify immediately the sender and delete the original
>> email received, any attachments and all copies from your system.
>>
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Broker-J] JDBC message store performance
Posted by Rob Godfrey <ro...@gmail.com>.
As an alternative approach, how about the (totally untested) patch I've
attached to QPID-8242 [1]
Rather than trying to consolidate the commits to delete the queue entry and
the message content it instead attempts to delete the message content
asynchronously and batch up all message contents awaiting deletion. From a
performance point of view, on the broker it takes the message content
deletion off the connection thread, and if the database is being the
bottleneck it should start batching up the deletions of content.
-- Rob
[1]
https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242.patch
On Mon, 17 Sep 2018 at 15:22, Rob Godfrey <ro...@gmail.com> wrote:
> The reference counting is done in AbstractServerMessageImpl.java. In
> general instances of ServerMessage should not be passed around, rather a
> MessageReference (obtained by calling newReference(..) on the message
> object. Then if the message is no longer required in that context,
> release() can be called on the reference. Within the context of a queue
> this is normally encapsulated by the notion of the QueueEntry (which holds
> the relevant MessageReference). When delete() is called on the QueueEntry
> (delete being defined in the MessageInstance interface), this releases the
> reference, which decrements the reference count, and if the count has gone
> to zero the message calls the store to delete itself.
>
> I think trying to change this logic would be a bit of a nightmare
> (understatement). I think a better alternative in terms of reducing the
> number of commits in the JDBC store might instead be for the remove()
> method on StoredJDBCMessage to (rather than immediately commit the message
> delete) schedule the message removal to be picked up by the next commit
> that the store is asked to perform. This would make the behaviour more
> like the BDB store (where we schedule the commit but don't actually force
> the sync to disk on message deletion).
>
> -- Rob
>
> On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com> wrote:
>
>> Ok then I missed something.
>> Whan/Where is the reference-counting you were talking about in your first
>> mail happening?
>>
>> Olivier
>>
>> -----Original Message-----
>> From: Rob Godfrey <ro...@gmail.com>
>> Sent: lundi 17 septembre 2018 14:05
>> To: users@qpid.apache.org
>> Subject: Re: [Broker-J] JDBC message store performance
>>
>> Hi Olivier,
>>
>> the approach you are attempting will not work for the reasons I described
>> previously. If the message has (for instance) been placed in two durable
>> subscription queues (because there are two durable subscriptions) then as
>> soon as the message is consumed from the first queue it would be removed
>> from the store, leading to problems when the second consumer tries to read
>> the message.
>>
>> -- Rob
>>
>> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
>> Olivier.VERMEULEN@murex.com>
>> wrote:
>>
>> > Hello Rob,
>> >
>> > Thanks for the answer.
>> > I started looking at the code to see if there is something I can do
>> > about these 2 commits.
>> > But before going any further I'd like your input on the below, to see
>> > if what I'm trying to do could work or if I'm missing something (which
>> > I'm surely are)
>> >
>> > https://github.com/apache/qpid-broker-j/compare/master...overmeulen:fe
>> > ature/jdbc-message-store-commits
>> >
>> > Regards,
>> > Olivier
>> >
>> > -----Original Message-----
>> > From: Rob Godfrey <ro...@gmail.com>
>> > Sent: vendredi 14 septembre 2018 16:06
>> > To: users@qpid.apache.org
>> > Cc: AYOUBI Ali <Al...@murex.com>
>> > Subject: Re: [Broker-J] JDBC message store performance
>> >
>> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
>> > Olivier.VERMEULEN@murex.com>
>> > wrote:
>> >
>> > > Hello,
>> > >
>> > > We ran a performance test with a bunch of brokers and an Oracle
>> > > database to store the messages.
>> > > We noticed that the database was a bit overloaded with commits.
>> > > Looking at the logs we saw that sending a message was triggering 1
>> > > commit for 3 operations (QPID_QUEUE_ENTRIES, QPID_MESSAGE_METADATA,
>> > > QPID_MESSAGE_CONTENT) which is what we were expecting but receiving
>> > > a message was triggering 2 commits (1 for QPID_QUEUE_ENTRIES and 1
>> > > for QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
>> > > I debugged a bit the code and saw that in
>> > > AbstractVirtualHost.executeTransaction the delete on
>> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a
>> > > "post commit" operation, explaining why we have 2 commits.
>> > > Is it something expected? Do you think we could reduce this to 1
>> > > commit when receiving a message?
>> > >
>> >
>> > It's not unexpected - basically the issue is that the broker needs to
>> > cope with the possibility that the same message is being stored on
>> > multiple queues. The first commit is deleting the referencing of the
>> > message from the given queue. The second commit is occuring after the
>> > message has been definitively removed from the queue, and the store
>> > has determined that there are no more references, so it is ok to
>> > remove the message. This is driven by reference-counting of the
>> > message, and has historically been a place of many potential race
>> > conditions. I'm sure it is possible to optimise the code in some way,
>> > but it may not be "easy". For the BDB store this doesn't matter as
>> > much as the actual synchronisation to disk of these operations is
>> > coalesced, obviously this is more of an issue for the JDBC store.
>> >
>> > -- Rob
>> >
>> >
>> > >
>> > > Thanks,
>> > > Olivier
>> > > *******************************
>> > >
>> > > This e-mail contains information for the intended recipient only. It
>> > > may contain proprietary material or confidential information. If you
>> > > are not the intended recipient you are not authorised to distribute,
>> > > copy or use this e-mail or any attachment to it. Murex cannot
>> > > guarantee that it is virus free and accepts no responsibility for
>> > > any loss or damage arising from its use. If you have received this
>> > > e-mail in error please notify immediately the sender and delete the
>> > > original email received, any attachments and all copies from your
>> system.
>> > >
>> > *******************************
>> >
>> > This e-mail contains information for the intended recipient only. It
>> > may contain proprietary material or confidential information. If you
>> > are not the intended recipient you are not authorised to distribute,
>> > copy or use this e-mail or any attachment to it. Murex cannot
>> > guarantee that it is virus free and accepts no responsibility for any
>> > loss or damage arising from its use. If you have received this e-mail
>> > in error please notify immediately the sender and delete the original
>> > email received, any attachments and all copies from your system.
>> >
>> *******************************
>>
>> This e-mail contains information for the intended recipient only. It may
>> contain proprietary material or confidential information. If you are not
>> the intended recipient you are not authorised to distribute, copy or use
>> this e-mail or any attachment to it. Murex cannot guarantee that it is
>> virus free and accepts no responsibility for any loss or damage arising
>> from its use. If you have received this e-mail in error please notify
>> immediately the sender and delete the original email received, any
>> attachments and all copies from your system.
>>
>
Re: [Broker-J] JDBC message store performance
Posted by Rob Godfrey <ro...@gmail.com>.
The reference counting is done in AbstractServerMessageImpl.java. In
general instances of ServerMessage should not be passed around, rather a
MessageReference (obtained by calling newReference(..) on the message
object. Then if the message is no longer required in that context,
release() can be called on the reference. Within the context of a queue
this is normally encapsulated by the notion of the QueueEntry (which holds
the relevant MessageReference). When delete() is called on the QueueEntry
(delete being defined in the MessageInstance interface), this releases the
reference, which decrements the reference count, and if the count has gone
to zero the message calls the store to delete itself.
I think trying to change this logic would be a bit of a nightmare
(understatement). I think a better alternative in terms of reducing the
number of commits in the JDBC store might instead be for the remove()
method on StoredJDBCMessage to (rather than immediately commit the message
delete) schedule the message removal to be picked up by the next commit
that the store is asked to perform. This would make the behaviour more
like the BDB store (where we schedule the commit but don't actually force
the sync to disk on message deletion).
-- Rob
On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Ok then I missed something.
> Whan/Where is the reference-counting you were talking about in your first
> mail happening?
>
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: lundi 17 septembre 2018 14:05
> To: users@qpid.apache.org
> Subject: Re: [Broker-J] JDBC message store performance
>
> Hi Olivier,
>
> the approach you are attempting will not work for the reasons I described
> previously. If the message has (for instance) been placed in two durable
> subscription queues (because there are two durable subscriptions) then as
> soon as the message is consumed from the first queue it would be removed
> from the store, leading to problems when the second consumer tries to read
> the message.
>
> -- Rob
>
> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Hello Rob,
> >
> > Thanks for the answer.
> > I started looking at the code to see if there is something I can do
> > about these 2 commits.
> > But before going any further I'd like your input on the below, to see
> > if what I'm trying to do could work or if I'm missing something (which
> > I'm surely are)
> >
> > https://github.com/apache/qpid-broker-j/compare/master...overmeulen:fe
> > ature/jdbc-message-store-commits
> >
> > Regards,
> > Olivier
> >
> > -----Original Message-----
> > From: Rob Godfrey <ro...@gmail.com>
> > Sent: vendredi 14 septembre 2018 16:06
> > To: users@qpid.apache.org
> > Cc: AYOUBI Ali <Al...@murex.com>
> > Subject: Re: [Broker-J] JDBC message store performance
> >
> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com>
> > wrote:
> >
> > > Hello,
> > >
> > > We ran a performance test with a bunch of brokers and an Oracle
> > > database to store the messages.
> > > We noticed that the database was a bit overloaded with commits.
> > > Looking at the logs we saw that sending a message was triggering 1
> > > commit for 3 operations (QPID_QUEUE_ENTRIES, QPID_MESSAGE_METADATA,
> > > QPID_MESSAGE_CONTENT) which is what we were expecting but receiving
> > > a message was triggering 2 commits (1 for QPID_QUEUE_ENTRIES and 1
> > > for QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
> > > I debugged a bit the code and saw that in
> > > AbstractVirtualHost.executeTransaction the delete on
> > > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a
> > > "post commit" operation, explaining why we have 2 commits.
> > > Is it something expected? Do you think we could reduce this to 1
> > > commit when receiving a message?
> > >
> >
> > It's not unexpected - basically the issue is that the broker needs to
> > cope with the possibility that the same message is being stored on
> > multiple queues. The first commit is deleting the referencing of the
> > message from the given queue. The second commit is occuring after the
> > message has been definitively removed from the queue, and the store
> > has determined that there are no more references, so it is ok to
> > remove the message. This is driven by reference-counting of the
> > message, and has historically been a place of many potential race
> > conditions. I'm sure it is possible to optimise the code in some way,
> > but it may not be "easy". For the BDB store this doesn't matter as
> > much as the actual synchronisation to disk of these operations is
> > coalesced, obviously this is more of an issue for the JDBC store.
> >
> > -- Rob
> >
> >
> > >
> > > Thanks,
> > > Olivier
> > > *******************************
> > >
> > > This e-mail contains information for the intended recipient only. It
> > > may contain proprietary material or confidential information. If you
> > > are not the intended recipient you are not authorised to distribute,
> > > copy or use this e-mail or any attachment to it. Murex cannot
> > > guarantee that it is virus free and accepts no responsibility for
> > > any loss or damage arising from its use. If you have received this
> > > e-mail in error please notify immediately the sender and delete the
> > > original email received, any attachments and all copies from your
> system.
> > >
> > *******************************
> >
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorised to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for any
> > loss or damage arising from its use. If you have received this e-mail
> > in error please notify immediately the sender and delete the original
> > email received, any attachments and all copies from your system.
> >
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
RE: [Broker-J] JDBC message store performance
Posted by VERMEULEN Olivier <Ol...@murex.com>.
Ok then I missed something.
Whan/Where is the reference-counting you were talking about in your first mail happening?
Olivier
-----Original Message-----
From: Rob Godfrey <ro...@gmail.com>
Sent: lundi 17 septembre 2018 14:05
To: users@qpid.apache.org
Subject: Re: [Broker-J] JDBC message store performance
Hi Olivier,
the approach you are attempting will not work for the reasons I described previously. If the message has (for instance) been placed in two durable subscription queues (because there are two durable subscriptions) then as soon as the message is consumed from the first queue it would be removed from the store, leading to problems when the second consumer tries to read the message.
-- Rob
On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Hello Rob,
>
> Thanks for the answer.
> I started looking at the code to see if there is something I can do
> about these 2 commits.
> But before going any further I'd like your input on the below, to see
> if what I'm trying to do could work or if I'm missing something (which
> I'm surely are)
>
> https://github.com/apache/qpid-broker-j/compare/master...overmeulen:fe
> ature/jdbc-message-store-commits
>
> Regards,
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: vendredi 14 septembre 2018 16:06
> To: users@qpid.apache.org
> Cc: AYOUBI Ali <Al...@murex.com>
> Subject: Re: [Broker-J] JDBC message store performance
>
> On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Hello,
> >
> > We ran a performance test with a bunch of brokers and an Oracle
> > database to store the messages.
> > We noticed that the database was a bit overloaded with commits.
> > Looking at the logs we saw that sending a message was triggering 1
> > commit for 3 operations (QPID_QUEUE_ENTRIES, QPID_MESSAGE_METADATA,
> > QPID_MESSAGE_CONTENT) which is what we were expecting but receiving
> > a message was triggering 2 commits (1 for QPID_QUEUE_ENTRIES and 1
> > for QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
> > I debugged a bit the code and saw that in
> > AbstractVirtualHost.executeTransaction the delete on
> > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a
> > "post commit" operation, explaining why we have 2 commits.
> > Is it something expected? Do you think we could reduce this to 1
> > commit when receiving a message?
> >
>
> It's not unexpected - basically the issue is that the broker needs to
> cope with the possibility that the same message is being stored on
> multiple queues. The first commit is deleting the referencing of the
> message from the given queue. The second commit is occuring after the
> message has been definitively removed from the queue, and the store
> has determined that there are no more references, so it is ok to
> remove the message. This is driven by reference-counting of the
> message, and has historically been a place of many potential race
> conditions. I'm sure it is possible to optimise the code in some way,
> but it may not be "easy". For the BDB store this doesn't matter as
> much as the actual synchronisation to disk of these operations is
> coalesced, obviously this is more of an issue for the JDBC store.
>
> -- Rob
>
>
> >
> > Thanks,
> > Olivier
> > *******************************
> >
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorised to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for
> > any loss or damage arising from its use. If you have received this
> > e-mail in error please notify immediately the sender and delete the
> > original email received, any attachments and all copies from your system.
> >
> *******************************
>
> This e-mail contains information for the intended recipient only. It
> may contain proprietary material or confidential information. If you
> are not the intended recipient you are not authorised to distribute,
> copy or use this e-mail or any attachment to it. Murex cannot
> guarantee that it is virus free and accepts no responsibility for any
> loss or damage arising from its use. If you have received this e-mail
> in error please notify immediately the sender and delete the original
> email received, any attachments and all copies from your system.
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Broker-J] JDBC message store performance
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Olivier,
the approach you are attempting will not work for the reasons I described
previously. If the message has (for instance) been placed in two durable
subscription queues (because there are two durable subscriptions) then as
soon as the message is consumed from the first queue it would be removed
from the store, leading to problems when the second consumer tries to read
the message.
-- Rob
On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Hello Rob,
>
> Thanks for the answer.
> I started looking at the code to see if there is something I can do about
> these 2 commits.
> But before going any further I'd like your input on the below, to see if
> what I'm trying to do could work or if I'm missing something (which I'm
> surely are)
>
> https://github.com/apache/qpid-broker-j/compare/master...overmeulen:feature/jdbc-message-store-commits
>
> Regards,
> Olivier
>
> -----Original Message-----
> From: Rob Godfrey <ro...@gmail.com>
> Sent: vendredi 14 septembre 2018 16:06
> To: users@qpid.apache.org
> Cc: AYOUBI Ali <Al...@murex.com>
> Subject: Re: [Broker-J] JDBC message store performance
>
> On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <
> Olivier.VERMEULEN@murex.com>
> wrote:
>
> > Hello,
> >
> > We ran a performance test with a bunch of brokers and an Oracle
> > database to store the messages.
> > We noticed that the database was a bit overloaded with commits.
> > Looking at the logs we saw that sending a message was triggering 1
> > commit for 3 operations (QPID_QUEUE_ENTRIES, QPID_MESSAGE_METADATA,
> > QPID_MESSAGE_CONTENT) which is what we were expecting but receiving a
> > message was triggering 2 commits (1 for QPID_QUEUE_ENTRIES and 1 for
> > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
> > I debugged a bit the code and saw that in
> > AbstractVirtualHost.executeTransaction the delete on
> > QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a "post
> > commit" operation, explaining why we have 2 commits.
> > Is it something expected? Do you think we could reduce this to 1
> > commit when receiving a message?
> >
>
> It's not unexpected - basically the issue is that the broker needs to cope
> with the possibility that the same message is being stored on multiple
> queues. The first commit is deleting the referencing of the message from
> the given queue. The second commit is occuring after the message has been
> definitively removed from the queue, and the store has determined that
> there are no more references, so it is ok to remove the message. This is
> driven by reference-counting of the message, and has historically been a
> place of many potential race conditions. I'm sure it is possible to
> optimise the code in some way, but it may not be "easy". For the BDB store
> this doesn't matter as much as the actual synchronisation to disk of these
> operations is coalesced, obviously this is more of an issue for the JDBC
> store.
>
> -- Rob
>
>
> >
> > Thanks,
> > Olivier
> > *******************************
> >
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorised to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for any
> > loss or damage arising from its use. If you have received this e-mail
> > in error please notify immediately the sender and delete the original
> > email received, any attachments and all copies from your system.
> >
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
RE: [Broker-J] JDBC message store performance
Posted by VERMEULEN Olivier <Ol...@murex.com>.
Hello Rob,
Thanks for the answer.
I started looking at the code to see if there is something I can do about these 2 commits.
But before going any further I'd like your input on the below, to see if what I'm trying to do could work or if I'm missing something (which I'm surely are)
https://github.com/apache/qpid-broker-j/compare/master...overmeulen:feature/jdbc-message-store-commits
Regards,
Olivier
-----Original Message-----
From: Rob Godfrey <ro...@gmail.com>
Sent: vendredi 14 septembre 2018 16:06
To: users@qpid.apache.org
Cc: AYOUBI Ali <Al...@murex.com>
Subject: Re: [Broker-J] JDBC message store performance
On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Hello,
>
> We ran a performance test with a bunch of brokers and an Oracle
> database to store the messages.
> We noticed that the database was a bit overloaded with commits.
> Looking at the logs we saw that sending a message was triggering 1
> commit for 3 operations (QPID_QUEUE_ENTRIES, QPID_MESSAGE_METADATA,
> QPID_MESSAGE_CONTENT) which is what we were expecting but receiving a
> message was triggering 2 commits (1 for QPID_QUEUE_ENTRIES and 1 for
> QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
> I debugged a bit the code and saw that in
> AbstractVirtualHost.executeTransaction the delete on
> QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT was defined as a "post
> commit" operation, explaining why we have 2 commits.
> Is it something expected? Do you think we could reduce this to 1
> commit when receiving a message?
>
It's not unexpected - basically the issue is that the broker needs to cope with the possibility that the same message is being stored on multiple queues. The first commit is deleting the referencing of the message from the given queue. The second commit is occuring after the message has been definitively removed from the queue, and the store has determined that there are no more references, so it is ok to remove the message. This is driven by reference-counting of the message, and has historically been a place of many potential race conditions. I'm sure it is possible to optimise the code in some way, but it may not be "easy". For the BDB store this doesn't matter as much as the actual synchronisation to disk of these operations is coalesced, obviously this is more of an issue for the JDBC store.
-- Rob
>
> Thanks,
> Olivier
> *******************************
>
> This e-mail contains information for the intended recipient only. It
> may contain proprietary material or confidential information. If you
> are not the intended recipient you are not authorised to distribute,
> copy or use this e-mail or any attachment to it. Murex cannot
> guarantee that it is virus free and accepts no responsibility for any
> loss or damage arising from its use. If you have received this e-mail
> in error please notify immediately the sender and delete the original
> email received, any attachments and all copies from your system.
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Broker-J] JDBC message store performance
Posted by Rob Godfrey <ro...@gmail.com>.
On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier <Ol...@murex.com>
wrote:
> Hello,
>
> We ran a performance test with a bunch of brokers and an Oracle database
> to store the messages.
> We noticed that the database was a bit overloaded with commits.
> Looking at the logs we saw that sending a message was triggering 1 commit
> for 3 operations (QPID_QUEUE_ENTRIES, QPID_MESSAGE_METADATA,
> QPID_MESSAGE_CONTENT) which is what we were expecting but receiving a
> message was triggering 2 commits (1 for QPID_QUEUE_ENTRIES and 1 for
> QPID_MESSAGE_METADATA and QPID_MESSAGE_CONTENT).
> I debugged a bit the code and saw that in
> AbstractVirtualHost.executeTransaction the delete on QPID_MESSAGE_METADATA
> and QPID_MESSAGE_CONTENT was defined as a "post commit" operation,
> explaining why we have 2 commits.
> Is it something expected? Do you think we could reduce this to 1 commit
> when receiving a message?
>
It's not unexpected - basically the issue is that the broker needs to cope
with the possibility that the same message is being stored on multiple
queues. The first commit is deleting the referencing of the message from
the given queue. The second commit is occuring after the message has been
definitively removed from the queue, and the store has determined that
there are no more references, so it is ok to remove the message. This is
driven by reference-counting of the message, and has historically been a
place of many potential race conditions. I'm sure it is possible to
optimise the code in some way, but it may not be "easy". For the BDB store
this doesn't matter as much as the actual synchronisation to disk of these
operations is coalesced, obviously this is more of an issue for the JDBC
store.
-- Rob
>
> Thanks,
> Olivier
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>