You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Roger Hoover <ro...@gmail.com> on 2009/04/29 00:15:25 UTC
Queue message order
Hi,
Are queued messages supposed to be delivered in order? I have a simple
STOMP Perl script below that shows messages delivered out of order. I've
run it against AMQ 4.1.1 and 5.2.0 in with both persistent and
non-persistent messages. Note from the message ids and timestamps that the
newer message is delivered first.
Is there a way to force message delivery order? I don't think selectors
allow for ordering, right? Our application doesn't particularly depend on
message order but this behavior is causing my unit tests to fail every third
time they are run. :(
This also seems problematic for STOMP clients using "client" ack mode.
Since newer messages are delivered first, ack'ing the newer message will
implicitly ack the older message which has not even been delivered yet.
Any help is appreciated. Thanks,
Roger
$ perl out-of-order.pl
OUT OF ORDER!!!
Frame 1: $VAR1 = bless( {
'body' => 'test2',
'headers' => {
'priority' => '0',
'timestamp' => '1240956094252',
'destination' => '/queue/testOutOfOrder',
'message-id' =>
'ID:thedude.local-58168-1240956067250-3:3897:-1:1:2',
'expires' => '0'
},
'command' => 'MESSAGE'
}, 'Net::Stomp::Frame' );
Frame 2: $VAR1 = bless( {
'body' => 'test1',
'headers' => {
'priority' => '0',
'timestamp' => '1240956094247',
'destination' => '/queue/testOutOfOrder',
'message-id' =>
'ID:thedude.local-58168-1240956067250-3:3897:-1:1:1',
'expires' => '0'
},
'command' => 'MESSAGE'
}, 'Net::Stomp::Frame' );
#!/usr/bin/env perl
use strict;
use Net::Stomp;
use Data::Dumper;
my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' => '61613'
} );
$stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
$stomp->subscribe(
{ 'destination' => '/queue/testOutOfOrder',
'ack' => 'client',
'activemq.prefetchSize' => 1
}
);
while ($stomp->can_read({'timeout' => 1})) {
my $frame = $stomp->receive_frame();
$stomp->ack({'frame' => $frame});
}
$stomp->disconnect();
while (1) {
doit();
}
sub doit {
my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
'61613' } );
$stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
$stomp->send({ 'destination' => '/queue/testOutOfOrder', 'body' =>
'test1', 'persistent' => 'true'} );
$stomp->send({ 'destination' => '/queue/testOutOfOrder', 'body' =>
'test2', 'persistent' => 'true'} );
$stomp->disconnect();
my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
'61613' } );
$stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
$stomp->subscribe(
{ 'destination' => '/queue/testOutOfOrder',
'ack' => 'client',
'activemq.prefetchSize' => 1
}
);
my $frame1 = $stomp->receive_frame();
$stomp->ack({'frame' => $frame1});
my $frame2 = $stomp->receive_frame();
$stomp->ack({'frame' => $frame2});
$stomp->disconnect();
if ($frame1->{'body'} ne 'test1') {
print "OUT OF ORDER!!!\n";
print "Frame 1: " . Dumper($frame1);
print "Frame 2: " . Dumper($frame2);
}
}
Re: Queue message order
Posted by Andreas Gies <an...@soa-knowledge.net>.
Hi Roger,
I see....thanks for answering.
Best regards
Andreas
On Apr 29, 2009, at 8:31 AM, Roger Hoover wrote:
> Hi Andreas,
>
> That's an insightful question and there is a good answer, albeit
> complicated.
>
> The test is trying to validate the exception handling logic in our
> STOMP
> client. To prevent the poison pill problem where the same bad
> message gets
> redelivered over and over in an infinite loop, the logic for a
> message that
> causes an exception is to 1) ACK the offending message to the
> broker, 2)
> send it to an error queue, 3) disconnect cleanly from the broker (to
> prevent
> a bug in AMQ 4 which does not release the thread if the connection
> is not
> closed cleanly), and then 4) rethrow the original exception.
>
> The unit test requires two distinct messages. The client purposely
> throws
> an exception trying to handle the first message and the second
> message is
> necessary to verify #1 above. If msg 1 was ACK'ed, msg 2 will be
> next in the
> queue. This is why two distinct messages are required to be
> delivered in
> order for the test.
>
> Cheers,
> Roger
>
> On Tue, Apr 28, 2009 at 10:42 PM, Andreas Gies <andreas@soa-knowledge.net
> >wrote:
>
>> Hi Roger,
>>
>> may be I missed something, but if the application doesn't need the
>> messages to be in order, why are you testing for it ?
>>
>> Just wondering
>> Andreas
>>
>>
>> On Apr 29, 2009, at 12:15 AM, Roger Hoover wrote:
>>
>> Hi,
>>>
>>> Are queued messages supposed to be delivered in order? I have a
>>> simple
>>> STOMP Perl script below that shows messages delivered out of
>>> order. I've
>>> run it against AMQ 4.1.1 and 5.2.0 in with both persistent and
>>> non-persistent messages. Note from the message ids and timestamps
>>> that
>>> the
>>> newer message is delivered first.
>>>
>>> Is there a way to force message delivery order? I don't think
>>> selectors
>>> allow for ordering, right? Our application doesn't particularly
>>> depend on
>>> message order but this behavior is causing my unit tests to fail
>>> every
>>> third
>>> time they are run. :(
>>>
>>> This also seems problematic for STOMP clients using "client" ack
>>> mode.
>>> Since newer messages are delivered first, ack'ing the newer
>>> message will
>>> implicitly ack the older message which has not even been delivered
>>> yet.
>>>
>>> Any help is appreciated. Thanks,
>>>
>>> Roger
>>>
>>> $ perl out-of-order.pl
>>> OUT OF ORDER!!!
>>> Frame 1: $VAR1 = bless( {
>>> 'body' => 'test2',
>>> 'headers' => {
>>> 'priority' => '0',
>>> 'timestamp' => '1240956094252',
>>> 'destination' => '/queue/
>>> testOutOfOrder',
>>> 'message-id' =>
>>> 'ID:thedude.local-58168-1240956067250-3:3897:-1:1:2',
>>> 'expires' => '0'
>>> },
>>> 'command' => 'MESSAGE'
>>> }, 'Net::Stomp::Frame' );
>>> Frame 2: $VAR1 = bless( {
>>> 'body' => 'test1',
>>> 'headers' => {
>>> 'priority' => '0',
>>> 'timestamp' => '1240956094247',
>>> 'destination' => '/queue/
>>> testOutOfOrder',
>>> 'message-id' =>
>>> 'ID:thedude.local-58168-1240956067250-3:3897:-1:1:1',
>>> 'expires' => '0'
>>> },
>>> 'command' => 'MESSAGE'
>>> }, 'Net::Stomp::Frame' );
>>>
>>> #!/usr/bin/env perl
>>>
>>> use strict;
>>> use Net::Stomp;
>>> use Data::Dumper;
>>>
>>> my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
>>> '61613'
>>> } );
>>> $stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
>>> $stomp->subscribe(
>>> { 'destination' => '/queue/testOutOfOrder',
>>> 'ack' => 'client',
>>> 'activemq.prefetchSize' => 1
>>> }
>>> );
>>> while ($stomp->can_read({'timeout' => 1})) {
>>> my $frame = $stomp->receive_frame();
>>> $stomp->ack({'frame' => $frame});
>>> }
>>> $stomp->disconnect();
>>>
>>> while (1) {
>>> doit();
>>> }
>>>
>>> sub doit {
>>> my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
>>> '61613' } );
>>> $stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
>>>
>>> $stomp->send({ 'destination' => '/queue/testOutOfOrder', 'body' =>
>>> 'test1', 'persistent' => 'true'} );
>>> $stomp->send({ 'destination' => '/queue/testOutOfOrder', 'body' =>
>>> 'test2', 'persistent' => 'true'} );
>>> $stomp->disconnect();
>>>
>>> my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
>>> '61613' } );
>>> $stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
>>> $stomp->subscribe(
>>> { 'destination' => '/queue/testOutOfOrder',
>>> 'ack' => 'client',
>>> 'activemq.prefetchSize' => 1
>>> }
>>> );
>>> my $frame1 = $stomp->receive_frame();
>>> $stomp->ack({'frame' => $frame1});
>>> my $frame2 = $stomp->receive_frame();
>>> $stomp->ack({'frame' => $frame2});
>>> $stomp->disconnect();
>>>
>>> if ($frame1->{'body'} ne 'test1') {
>>> print "OUT OF ORDER!!!\n";
>>> print "Frame 1: " . Dumper($frame1);
>>> print "Frame 2: " . Dumper($frame2);
>>> }
>>> }
>>>
>>
>>
Re: Queue message order
Posted by Roger Hoover <ro...@gmail.com>.
Hi Andreas,
That's an insightful question and there is a good answer, albeit
complicated.
The test is trying to validate the exception handling logic in our STOMP
client. To prevent the poison pill problem where the same bad message gets
redelivered over and over in an infinite loop, the logic for a message that
causes an exception is to 1) ACK the offending message to the broker, 2)
send it to an error queue, 3) disconnect cleanly from the broker (to prevent
a bug in AMQ 4 which does not release the thread if the connection is not
closed cleanly), and then 4) rethrow the original exception.
The unit test requires two distinct messages. The client purposely throws
an exception trying to handle the first message and the second message is
necessary to verify #1 above. If msg 1 was ACK'ed, msg 2 will be next in the
queue. This is why two distinct messages are required to be delivered in
order for the test.
Cheers,
Roger
On Tue, Apr 28, 2009 at 10:42 PM, Andreas Gies <an...@soa-knowledge.net>wrote:
> Hi Roger,
>
> may be I missed something, but if the application doesn't need the
> messages to be in order, why are you testing for it ?
>
> Just wondering
> Andreas
>
>
> On Apr 29, 2009, at 12:15 AM, Roger Hoover wrote:
>
> Hi,
>>
>> Are queued messages supposed to be delivered in order? I have a simple
>> STOMP Perl script below that shows messages delivered out of order. I've
>> run it against AMQ 4.1.1 and 5.2.0 in with both persistent and
>> non-persistent messages. Note from the message ids and timestamps that
>> the
>> newer message is delivered first.
>>
>> Is there a way to force message delivery order? I don't think selectors
>> allow for ordering, right? Our application doesn't particularly depend on
>> message order but this behavior is causing my unit tests to fail every
>> third
>> time they are run. :(
>>
>> This also seems problematic for STOMP clients using "client" ack mode.
>> Since newer messages are delivered first, ack'ing the newer message will
>> implicitly ack the older message which has not even been delivered yet.
>>
>> Any help is appreciated. Thanks,
>>
>> Roger
>>
>> $ perl out-of-order.pl
>> OUT OF ORDER!!!
>> Frame 1: $VAR1 = bless( {
>> 'body' => 'test2',
>> 'headers' => {
>> 'priority' => '0',
>> 'timestamp' => '1240956094252',
>> 'destination' => '/queue/testOutOfOrder',
>> 'message-id' =>
>> 'ID:thedude.local-58168-1240956067250-3:3897:-1:1:2',
>> 'expires' => '0'
>> },
>> 'command' => 'MESSAGE'
>> }, 'Net::Stomp::Frame' );
>> Frame 2: $VAR1 = bless( {
>> 'body' => 'test1',
>> 'headers' => {
>> 'priority' => '0',
>> 'timestamp' => '1240956094247',
>> 'destination' => '/queue/testOutOfOrder',
>> 'message-id' =>
>> 'ID:thedude.local-58168-1240956067250-3:3897:-1:1:1',
>> 'expires' => '0'
>> },
>> 'command' => 'MESSAGE'
>> }, 'Net::Stomp::Frame' );
>>
>> #!/usr/bin/env perl
>>
>> use strict;
>> use Net::Stomp;
>> use Data::Dumper;
>>
>> my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
>> '61613'
>> } );
>> $stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
>> $stomp->subscribe(
>> { 'destination' => '/queue/testOutOfOrder',
>> 'ack' => 'client',
>> 'activemq.prefetchSize' => 1
>> }
>> );
>> while ($stomp->can_read({'timeout' => 1})) {
>> my $frame = $stomp->receive_frame();
>> $stomp->ack({'frame' => $frame});
>> }
>> $stomp->disconnect();
>>
>> while (1) {
>> doit();
>> }
>>
>> sub doit {
>> my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
>> '61613' } );
>> $stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
>>
>> $stomp->send({ 'destination' => '/queue/testOutOfOrder', 'body' =>
>> 'test1', 'persistent' => 'true'} );
>> $stomp->send({ 'destination' => '/queue/testOutOfOrder', 'body' =>
>> 'test2', 'persistent' => 'true'} );
>> $stomp->disconnect();
>>
>> my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
>> '61613' } );
>> $stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
>> $stomp->subscribe(
>> { 'destination' => '/queue/testOutOfOrder',
>> 'ack' => 'client',
>> 'activemq.prefetchSize' => 1
>> }
>> );
>> my $frame1 = $stomp->receive_frame();
>> $stomp->ack({'frame' => $frame1});
>> my $frame2 = $stomp->receive_frame();
>> $stomp->ack({'frame' => $frame2});
>> $stomp->disconnect();
>>
>> if ($frame1->{'body'} ne 'test1') {
>> print "OUT OF ORDER!!!\n";
>> print "Frame 1: " . Dumper($frame1);
>> print "Frame 2: " . Dumper($frame2);
>> }
>> }
>>
>
>
Re: Queue message order
Posted by Andreas Gies <an...@soa-knowledge.net>.
Hi Roger,
may be I missed something, but if the application doesn't need the
messages to be in order, why are you testing for it ?
Just wondering
Andreas
On Apr 29, 2009, at 12:15 AM, Roger Hoover wrote:
> Hi,
>
> Are queued messages supposed to be delivered in order? I have a
> simple
> STOMP Perl script below that shows messages delivered out of order.
> I've
> run it against AMQ 4.1.1 and 5.2.0 in with both persistent and
> non-persistent messages. Note from the message ids and timestamps
> that the
> newer message is delivered first.
>
> Is there a way to force message delivery order? I don't think
> selectors
> allow for ordering, right? Our application doesn't particularly
> depend on
> message order but this behavior is causing my unit tests to fail
> every third
> time they are run. :(
>
> This also seems problematic for STOMP clients using "client" ack mode.
> Since newer messages are delivered first, ack'ing the newer message
> will
> implicitly ack the older message which has not even been delivered
> yet.
>
> Any help is appreciated. Thanks,
>
> Roger
>
> $ perl out-of-order.pl
> OUT OF ORDER!!!
> Frame 1: $VAR1 = bless( {
> 'body' => 'test2',
> 'headers' => {
> 'priority' => '0',
> 'timestamp' => '1240956094252',
> 'destination' => '/queue/
> testOutOfOrder',
> 'message-id' =>
> 'ID:thedude.local-58168-1240956067250-3:3897:-1:1:2',
> 'expires' => '0'
> },
> 'command' => 'MESSAGE'
> }, 'Net::Stomp::Frame' );
> Frame 2: $VAR1 = bless( {
> 'body' => 'test1',
> 'headers' => {
> 'priority' => '0',
> 'timestamp' => '1240956094247',
> 'destination' => '/queue/
> testOutOfOrder',
> 'message-id' =>
> 'ID:thedude.local-58168-1240956067250-3:3897:-1:1:1',
> 'expires' => '0'
> },
> 'command' => 'MESSAGE'
> }, 'Net::Stomp::Frame' );
>
> #!/usr/bin/env perl
>
> use strict;
> use Net::Stomp;
> use Data::Dumper;
>
> my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
> '61613'
> } );
> $stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
> $stomp->subscribe(
> { 'destination' => '/queue/testOutOfOrder',
> 'ack' => 'client',
> 'activemq.prefetchSize' => 1
> }
> );
> while ($stomp->can_read({'timeout' => 1})) {
> my $frame = $stomp->receive_frame();
> $stomp->ack({'frame' => $frame});
> }
> $stomp->disconnect();
>
> while (1) {
> doit();
> }
>
> sub doit {
> my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
> '61613' } );
> $stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
>
> $stomp->send({ 'destination' => '/queue/testOutOfOrder', 'body' =>
> 'test1', 'persistent' => 'true'} );
> $stomp->send({ 'destination' => '/queue/testOutOfOrder', 'body' =>
> 'test2', 'persistent' => 'true'} );
> $stomp->disconnect();
>
> my $stomp = Net::Stomp->new( { 'hostname' => 'localhost', 'port' =>
> '61613' } );
> $stomp->connect( { 'login' => 'hello', 'passcode' => 'there' } );
> $stomp->subscribe(
> { 'destination' => '/queue/testOutOfOrder',
> 'ack' => 'client',
> 'activemq.prefetchSize' => 1
> }
> );
> my $frame1 = $stomp->receive_frame();
> $stomp->ack({'frame' => $frame1});
> my $frame2 = $stomp->receive_frame();
> $stomp->ack({'frame' => $frame2});
> $stomp->disconnect();
>
> if ($frame1->{'body'} ne 'test1') {
> print "OUT OF ORDER!!!\n";
> print "Frame 1: " . Dumper($frame1);
> print "Frame 2: " . Dumper($frame2);
> }
> }
Re: Queue message order
Posted by Roger Hoover <ro...@gmail.com>.
Thanks, Bruce. I'll give this a try.
On Tue, Apr 28, 2009 at 5:00 PM, Bruce Snyder <br...@gmail.com>wrote:
> On Wed, Apr 29, 2009 at 12:15 AM, Roger Hoover <ro...@gmail.com>
> wrote:
> > Hi,
> >
> > Are queued messages supposed to be delivered in order? I have a simple
> > STOMP Perl script below that shows messages delivered out of order. I've
> > run it against AMQ 4.1.1 and 5.2.0 in with both persistent and
> > non-persistent messages. Note from the message ids and timestamps that
> the
> > newer message is delivered first.
> >
> > Is there a way to force message delivery order? I don't think selectors
> > allow for ordering, right? Our application doesn't particularly depend on
> > message order but this behavior is causing my unit tests to fail every
> third
> > time they are run. :(
> >
> > This also seems problematic for STOMP clients using "client" ack mode.
> > Since newer messages are delivered first, ack'ing the newer message will
> > implicitly ack the older message which has not even been delivered yet.
>
> Ordering of message dispatch doesn't take place by default but it's
> easy to achieve. You just need to configure it in the
> conf/activemq.xml file using the strict order dispatch policy AKA
> total ordering:
>
> http://activemq.apache.org/total-ordering.html
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
>
Re: Queue message order
Posted by Bruce Snyder <br...@gmail.com>.
On Wed, Apr 29, 2009 at 12:15 AM, Roger Hoover <ro...@gmail.com> wrote:
> Hi,
>
> Are queued messages supposed to be delivered in order? I have a simple
> STOMP Perl script below that shows messages delivered out of order. I've
> run it against AMQ 4.1.1 and 5.2.0 in with both persistent and
> non-persistent messages. Note from the message ids and timestamps that the
> newer message is delivered first.
>
> Is there a way to force message delivery order? I don't think selectors
> allow for ordering, right? Our application doesn't particularly depend on
> message order but this behavior is causing my unit tests to fail every third
> time they are run. :(
>
> This also seems problematic for STOMP clients using "client" ack mode.
> Since newer messages are delivered first, ack'ing the newer message will
> implicitly ack the older message which has not even been delivered yet.
Ordering of message dispatch doesn't take place by default but it's
easy to achieve. You just need to configure it in the
conf/activemq.xml file using the strict order dispatch policy AKA
total ordering:
http://activemq.apache.org/total-ordering.html
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder