Java M2 - status of three outstanding Jiras

[Robert Greig <ro...@gmail.com>]

Hi,

We were aiming to get the three critical M2 jiras done today, so I
thought I should ask what the status is - here were the earlier
summaries I had:

QPID-531 : Martin has a one line fix for this which will be committed very soon

There haven't been any commits for this yet. Martin is this on track
for today? Did testing show up bugs?

QPID-572 : Believed to be fixed

There has been some further discussion about this following a commit
by Martin. Do we think we have a fix or is more work required?

QPID-573 : Fixed by Rafi on trunk. Can you merge to M2 and M2.1?

As far as I know just pending the merge.

Thanks,

RG

Re: Java M2 - status of three outstanding Jiras

[Rafael Schloming <ra...@redhat.com>]

Rafael Schloming wrote:
> Robert Greig wrote:
>> Hi,
>>
>> We were aiming to get the three critical M2 jiras done today, so I
>> thought I should ask what the status is - here were the earlier
>> summaries I had:
>>
>> QPID-531 : Martin has a one line fix for this which will be committed 
>> very soon
>>
>> There haven't been any commits for this yet. Martin is this on track
>> for today? Did testing show up bugs?
>>
>> QPID-572 : Believed to be fixed
>>
>> There has been some further discussion about this following a commit
>> by Martin. Do we think we have a fix or is more work required?
> 
> This is a tricky issue. There are actually two bugs here. The correct 
> fix for QPID-572 (the original patch submitted to the JIRA) actually 
> causes another race condition to be more likely. I've described the 
> other race condition in detail in QPID-601.
> 
> The modified patch for QPID-572 simply circumvents the effect of the 
> correct fix when selectors are used. This makes QPID-601 less likely to 
> occur, but the basic race condition described in QPID-601 is still 
> there. Unfortunately this also reintroduces the ordering bug if 
> selectors are used, and also unfortunately, AFAICT a proper fix for 
> QPID-601 is non trivial.
> 
> So basically if we want to release M2 in a timely manner we need to 
> choose between selectors with correct message ordering and a higher 
> chance of messages being stranded, or selectors with incorrect message 
> ordering and a lower chance of messages being stranded.

Just to be clear, the message ordering problem should be fixed either 
way if you don't use selectors.

--Rafael

Re: Java M2 - status of three outstanding Jiras

[Martin Ritchie <ri...@apache.org>]

I have committed the changes to M2.1 along with an updated AckTest.
NOTE: The AckTest has only been updated to cover NO_ACK.

I am on holiday until next Thursday so if someone could commit the
changes to the other branches(M2,Trunk) I would be grateful.

Regards

On 14/09/2007, Robert Greig <ro...@gmail.com> wrote:
> On 14/09/2007, Rafael Schloming <ra...@redhat.com> wrote:
>
> > This is a tricky issue. There are actually two bugs here. The correct
> > fix for QPID-572 (the original patch submitted to the JIRA) actually
> > causes another race condition to be more likely. I've described the
> > other race condition in detail in QPID-601.
>
> Ah ok. It is useful to split this into the real issues.
>
> > So basically if we want to release M2 in a timely manner we need to
> > choose between selectors with correct message ordering and a higher
> > chance of messages being stranded, or selectors with incorrect message
> > ordering and a lower chance of messages being stranded.
>
> OK my own view is that stranded messages are worse than out of order
> messages therefore I would say we go for selectors with incorrect
> message ordering and a lower chance of messages being stranded.
>
> But we should note in the release notes that when using selectors
> ordering guarantees are "relaxed".
>
> What do others think?
>
> > > QPID-573 : Fixed by Rafi on trunk. Can you merge to M2 and M2.1?
> > >
> > > As far as I know just pending the merge.
> >
> > The fix should be merged to both M2 and M2.1 now.
>
> Excellent. Thanks.
>
> So, Martin - QPID-531 - is that ready to commit?
>
> RG
>


-- 
Martin Ritchie

Re: Java M2 - status of three outstanding Jiras

[Carl Trieloff <cc...@redhat.com>]

>>
>> OK my own view is that stranded messages are worse than out of order
>> messages therefore I would say we go for selectors with incorrect
>> message ordering and a lower chance of messages being stranded.
>>
>> But we should note in the release notes that when using selectors
>> ordering guarantees are "relaxed".
>>
>> What do others think?
>
> I agree that's the best option if there is no simple fix for QPID-601.
>
+1

Carl.

Re: Java M2 - status of three outstanding Jiras

[Robert Godfrey <ro...@gmail.com>]

Martin - please don't tell me yuou have Qpid running on your phone :-)

PS You're supposed to be on holiday

-- Rob

On 17/09/2007, Martin Ritchie <ri...@apache.org> wrote:
>
> Rafi. do you have a test case that can reproduce the problem in 602?
> I've been trying to do so by populating the queue and then connect a
> client wils sending more messages. So far it always gets the messages
> in the right order. I haven't had much time to get a failing test so i
> can test my solution so he you have such a test that would really
> help.
>
> Martin,
> The north end of Skye. Near the mobile mast.
>
> On 15/09/2007, Rafael Schloming <ra...@redhat.com> wrote:
> > Robert Greig wrote:
> > > On 14/09/2007, Rafael Schloming <ra...@redhat.com> wrote:
> > >
> > >> This is a tricky issue. There are actually two bugs here. The correct
> > >> fix for QPID-572 (the original patch submitted to the JIRA) actually
> > >> causes another race condition to be more likely. I've described the
> > >> other race condition in detail in QPID-601.
> > >
> > > Ah ok. It is useful to split this into the real issues.
> > >
> > >> So basically if we want to release M2 in a timely manner we need to
> > >> choose between selectors with correct message ordering and a higher
> > >> chance of messages being stranded, or selectors with incorrect
> message
> > >> ordering and a lower chance of messages being stranded.
> > >
> > > OK my own view is that stranded messages are worse than out of order
> > > messages therefore I would say we go for selectors with incorrect
> > > message ordering and a lower chance of messages being stranded.
> > >
> > > But we should note in the release notes that when using selectors
> > > ordering guarantees are "relaxed".
> > >
> > > What do others think?
> >
> > I agree that's the best option if there is no simple fix for QPID-601.
> >
> > --Rafael
> >
> >
>
>
> --
> Martin Ritchie
>

Re: Java M2 - status of three outstanding Jiras

[Rafael Schloming <ra...@redhat.com>]

Martin Ritchie wrote:
> On 17/09/2007, Rafael Schloming <ra...@redhat.com> wrote:
>> Martin Ritchie wrote:
>>> Rafi. do you have a test case that can reproduce the problem in 602?
>>> I've been trying to do so by populating the queue and then connect a
>>> client wils sending more messages. So far it always gets the messages
>>> in the right order. I haven't had much time to get a failing test so i
>>> can test my solution so he you have such a test that would really
>>> help.
>> I assume you mean QPID-601, not QPID-602.
> 
> Indeed I did, I don't have Qpid running on my phone as Robert
> suggested so had to remember the JIRA number
> 
>> I had thought the TopicSessionTest failure was an example of the race
>> condition, however your explanation in another post also accounts for
>> that failure.
> Indeed, I haven't seen TST fail since I put that fix in, but I do see
> the window you are talking about.
> 
>> I haven't tested this myself, but in theory a long enough Thread.sleep
>> at the end of ConcurrentSelectorDeliveryManager.populatePreDeliveryQueue
>> should make the race condition easy to reproduce because at that point
>> the Subscription has not been added to _subscriptions, so any messages
>> delivered at that point will be stranded in the _messages queue.
> 
> Ah, I think I had to much of my fix in place. I was hoping to see
> re-ordering though as I was doing the add to the SubscriberSet before
> the populate. I'd really like a test case that fails, for the right
> reasons, without adding sleeps to the broker. I had hopped a large
> back log of complex selects to check might slow it down enough to let
> some new msgs slip through. Will take another look at it. I certainly
> have an approach that doesn't break things and only requires the _lock
> for a brief time. So shouldn't impact performance to much.
> 
> I'd like to get the test sorted and commit it to M2.1 for you to cast
> your eye over it. Seems easier to commit it rather than have you apply
> at as a patch, but can do either

A patch is just as easy for me, but either way works.

--Rafael

Re: Java M2 - status of three outstanding Jiras

[Martin Ritchie <ri...@apache.org>]

On 17/09/2007, Rafael Schloming <ra...@redhat.com> wrote:
> Martin Ritchie wrote:
> > Rafi. do you have a test case that can reproduce the problem in 602?
> > I've been trying to do so by populating the queue and then connect a
> > client wils sending more messages. So far it always gets the messages
> > in the right order. I haven't had much time to get a failing test so i
> > can test my solution so he you have such a test that would really
> > help.
>
> I assume you mean QPID-601, not QPID-602.

Indeed I did, I don't have Qpid running on my phone as Robert
suggested so had to remember the JIRA number

> I had thought the TopicSessionTest failure was an example of the race
> condition, however your explanation in another post also accounts for
> that failure.
Indeed, I haven't seen TST fail since I put that fix in, but I do see
the window you are talking about.

> I haven't tested this myself, but in theory a long enough Thread.sleep
> at the end of ConcurrentSelectorDeliveryManager.populatePreDeliveryQueue
> should make the race condition easy to reproduce because at that point
> the Subscription has not been added to _subscriptions, so any messages
> delivered at that point will be stranded in the _messages queue.

Ah, I think I had to much of my fix in place. I was hoping to see
re-ordering though as I was doing the add to the SubscriberSet before
the populate. I'd really like a test case that fails, for the right
reasons, without adding sleeps to the broker. I had hopped a large
back log of complex selects to check might slow it down enough to let
some new msgs slip through. Will take another look at it. I certainly
have an approach that doesn't break things and only requires the _lock
for a brief time. So shouldn't impact performance to much.

I'd like to get the test sorted and commit it to M2.1 for you to cast
your eye over it. Seems easier to commit it rather than have you apply
at as a patch, but can do either

That is unless Rob catches me doing work ;)

> --Rafael

-- 
Martin Ritchie

Re: Java M2 - status of three outstanding Jiras

[Rafael Schloming <ra...@redhat.com>]

Martin Ritchie wrote:
> Rafi. do you have a test case that can reproduce the problem in 602?
> I've been trying to do so by populating the queue and then connect a
> client wils sending more messages. So far it always gets the messages
> in the right order. I haven't had much time to get a failing test so i
> can test my solution so he you have such a test that would really
> help.

I assume you mean QPID-601, not QPID-602.

I had thought the TopicSessionTest failure was an example of the race 
condition, however your explanation in another post also accounts for 
that failure.

I haven't tested this myself, but in theory a long enough Thread.sleep 
at the end of ConcurrentSelectorDeliveryManager.populatePreDeliveryQueue 
should make the race condition easy to reproduce because at that point 
the Subscription has not been added to _subscriptions, so any messages 
delivered at that point will be stranded in the _messages queue.

--Rafael

Re: Java M2 - status of three outstanding Jiras

[Martin Ritchie <ri...@apache.org>]

Rafi. do you have a test case that can reproduce the problem in 602?
I've been trying to do so by populating the queue and then connect a
client wils sending more messages. So far it always gets the messages
in the right order. I haven't had much time to get a failing test so i
can test my solution so he you have such a test that would really
help.

Martin,
The north end of Skye. Near the mobile mast.

On 15/09/2007, Rafael Schloming <ra...@redhat.com> wrote:
> Robert Greig wrote:
> > On 14/09/2007, Rafael Schloming <ra...@redhat.com> wrote:
> >
> >> This is a tricky issue. There are actually two bugs here. The correct
> >> fix for QPID-572 (the original patch submitted to the JIRA) actually
> >> causes another race condition to be more likely. I've described the
> >> other race condition in detail in QPID-601.
> >
> > Ah ok. It is useful to split this into the real issues.
> >
> >> So basically if we want to release M2 in a timely manner we need to
> >> choose between selectors with correct message ordering and a higher
> >> chance of messages being stranded, or selectors with incorrect message
> >> ordering and a lower chance of messages being stranded.
> >
> > OK my own view is that stranded messages are worse than out of order
> > messages therefore I would say we go for selectors with incorrect
> > message ordering and a lower chance of messages being stranded.
> >
> > But we should note in the release notes that when using selectors
> > ordering guarantees are "relaxed".
> >
> > What do others think?
>
> I agree that's the best option if there is no simple fix for QPID-601.
>
> --Rafael
>
>


-- 
Martin Ritchie

Re: Java M2 - status of three outstanding Jiras

[Martin Ritchie <ri...@apache.org>]

On 15/09/2007, Rafael Schloming <ra...@redhat.com> wrote:
> Robert Greig wrote:
> > On 14/09/2007, Rafael Schloming <ra...@redhat.com> wrote:
> >
> >> This is a tricky issue. There are actually two bugs here. The correct
> >> fix for QPID-572 (the original patch submitted to the JIRA) actually
> >> causes another race condition to be more likely. I've described the
> >> other race condition in detail in QPID-601.
> >
> > Ah ok. It is useful to split this into the real issues.
> >
> >> So basically if we want to release M2 in a timely manner we need to
> >> choose between selectors with correct message ordering and a higher
> >> chance of messages being stranded, or selectors with incorrect message
> >> ordering and a lower chance of messages being stranded.
> >
> > OK my own view is that stranded messages are worse than out of order
> > messages therefore I would say we go for selectors with incorrect
> > message ordering and a lower chance of messages being stranded.
> >
> > But we should note in the release notes that when using selectors
> > ordering guarantees are "relaxed".
> >
> > What do others think?
>
> I agree that's the best option if there is no simple fix for QPID-601.
>
> --Rafael

I've been testing a fix as overnight, but I don't have time to
complete it as I've got to go on holiday.

I've been held up this morning as AMQSessionTest is taking over 120s
to complete. I don't remember this but it is doing this on a clean
M2.1!

Four of the tests in AMQST are taking 30s each.. sounds like they are
waiting for a timeout to complete which isn't failing the test.

I don't have time to look but will investigate later.

-- 
Martin Ritchie

Re: Java M2 - status of three outstanding Jiras

[Rafael Schloming <ra...@redhat.com>]

Robert Greig wrote:
> On 14/09/2007, Rafael Schloming <ra...@redhat.com> wrote:
> 
>> This is a tricky issue. There are actually two bugs here. The correct
>> fix for QPID-572 (the original patch submitted to the JIRA) actually
>> causes another race condition to be more likely. I've described the
>> other race condition in detail in QPID-601.
> 
> Ah ok. It is useful to split this into the real issues.
> 
>> So basically if we want to release M2 in a timely manner we need to
>> choose between selectors with correct message ordering and a higher
>> chance of messages being stranded, or selectors with incorrect message
>> ordering and a lower chance of messages being stranded.
> 
> OK my own view is that stranded messages are worse than out of order
> messages therefore I would say we go for selectors with incorrect
> message ordering and a lower chance of messages being stranded.
> 
> But we should note in the release notes that when using selectors
> ordering guarantees are "relaxed".
> 
> What do others think?

I agree that's the best option if there is no simple fix for QPID-601.

--Rafael

Re: Java M2 - status of three outstanding Jiras

[Robert Greig <ro...@gmail.com>]

On 14/09/2007, Rafael Schloming <ra...@redhat.com> wrote:

> This is a tricky issue. There are actually two bugs here. The correct
> fix for QPID-572 (the original patch submitted to the JIRA) actually
> causes another race condition to be more likely. I've described the
> other race condition in detail in QPID-601.

Ah ok. It is useful to split this into the real issues.

> So basically if we want to release M2 in a timely manner we need to
> choose between selectors with correct message ordering and a higher
> chance of messages being stranded, or selectors with incorrect message
> ordering and a lower chance of messages being stranded.

OK my own view is that stranded messages are worse than out of order
messages therefore I would say we go for selectors with incorrect
message ordering and a lower chance of messages being stranded.

But we should note in the release notes that when using selectors
ordering guarantees are "relaxed".

What do others think?

> > QPID-573 : Fixed by Rafi on trunk. Can you merge to M2 and M2.1?
> >
> > As far as I know just pending the merge.
>
> The fix should be merged to both M2 and M2.1 now.

Excellent. Thanks.

So, Martin - QPID-531 - is that ready to commit?

RG

Re: Java M2 - status of three outstanding Jiras

[Rafael Schloming <ra...@redhat.com>]

Robert Greig wrote:
> Hi,
> 
> We were aiming to get the three critical M2 jiras done today, so I
> thought I should ask what the status is - here were the earlier
> summaries I had:
> 
> QPID-531 : Martin has a one line fix for this which will be committed very soon
> 
> There haven't been any commits for this yet. Martin is this on track
> for today? Did testing show up bugs?
> 
> QPID-572 : Believed to be fixed
> 
> There has been some further discussion about this following a commit
> by Martin. Do we think we have a fix or is more work required?

This is a tricky issue. There are actually two bugs here. The correct 
fix for QPID-572 (the original patch submitted to the JIRA) actually 
causes another race condition to be more likely. I've described the 
other race condition in detail in QPID-601.

The modified patch for QPID-572 simply circumvents the effect of the 
correct fix when selectors are used. This makes QPID-601 less likely to 
occur, but the basic race condition described in QPID-601 is still 
there. Unfortunately this also reintroduces the ordering bug if 
selectors are used, and also unfortunately, AFAICT a proper fix for 
QPID-601 is non trivial.

So basically if we want to release M2 in a timely manner we need to 
choose between selectors with correct message ordering and a higher 
chance of messages being stranded, or selectors with incorrect message 
ordering and a lower chance of messages being stranded.

> QPID-573 : Fixed by Rafi on trunk. Can you merge to M2 and M2.1?
> 
> As far as I know just pending the merge.

The fix should be merged to both M2 and M2.1 now.

--Rafael