You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Se...@troika.ru on 2011/03/30 15:20:49 UTC
I/O Errors
Hi there,
I'm using qpid 0.8 and sending only durable messages.
>From time to time the following exceptions can occur:
1. Journal "MyQueue": get_events() returned JERR_JCNTL_AIOCMPLWAIT;
wmgr_status: wmgr: pi=29 pc=44 po=0 aer=32 edac:TFFF
ps=[AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA] wrfc: state: Active fcntl[6]: pfid=6
ws=11268 wc=3256 rs=0 rc=0 ec=1282 ac=32
2. Journal "MyQueue": Unexpected I/O response (RHM_IORES_BUSY) on queue
MyQueue
3. error Unexpected exception: Attempted size underflow on dequeue(559):
size: max=104857600, current=119; count: unlimited; type=reject
>From the source code I know that
JERR_JCNTL_AIOCMPLWAIT means that "Timeout waiting for AIOs to complete"
RHM_IORES_BUSY means that "Another blocking operation is in progress"
but that all and I can't go any further.
Could you please help to determine what kind of I/O operations can cause
such errors and why they can occur?
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
Phone. +7 495 2580500 ext. 1246
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
Re: I/O Errors
Posted by Se...@troika.ru.
Hi there,
Sorry for so long answer.
We are planning to use the MRG Messaging with the latest updates, so the
problem may disappear.
For now I've raised the JIRA issue
https://issues.apache.org/jira/browse/QPID-3189 with minor priority
because such a behavior is hardly reproducable.
Best Regards,
Sergey Zhemzhitsky
Alan Conway <ac...@redhat.com>
01.04.2011 17:52
To
Sergey_Zhemzhitsky@troika.ru
cc
Kim van der Riet <ki...@redhat.com>, users@qpid.apache.org
Subject
Re: I/O Errors
On 04/01/2011 01:55 AM, Sergey_Zhemzhitsky@troika.ru wrote:
>
> Hi Alan,
>
> Thanks for information,
>
> From you message I understood that queue policy error can occur due to
previous
> errors with persistent store, for example, queue policy is incremented
and just
> after that the error occurs when trying to persist a message.
>
Yes that's my theory. Can you raise a JIRA for this, or perhaps 2? I think
there
are 2 issues - first, why the errors with the persistent store - Kim is
better
placed to answer that - and second errors from the store shouldn't cause
queue
policy underflow errors.
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
Re: I/O Errors
Posted by Alan Conway <ac...@redhat.com>.
On 04/01/2011 01:55 AM, Sergey_Zhemzhitsky@troika.ru wrote:
>
> Hi Alan,
>
> Thanks for information,
>
> From you message I understood that queue policy error can occur due to previous
> errors with persistent store, for example, queue policy is incremented and just
> after that the error occurs when trying to persist a message.
>
Yes that's my theory. Can you raise a JIRA for this, or perhaps 2? I think there
are 2 issues - first, why the errors with the persistent store - Kim is better
placed to answer that - and second errors from the store shouldn't cause queue
policy underflow errors.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: I/O Errors
Posted by Se...@troika.ru.
Hi Alan,
Thanks for information,
>From you message I understood that queue policy error can occur due to
previous errors with persistent store, for example, queue policy is
incremented and just after that the error occurs when trying to persist a
message.
Best Regards,
Sergey Zhemzhitsky
Alan Conway <ac...@redhat.com>
31.03.2011 19:06
Please respond to
users@qpid.apache.org
To
users@qpid.apache.org
cc
Kim van der Riet <ki...@redhat.com>
Subject
Re: I/O Errors
On 03/31/2011 09:33 AM, Kim van der Riet wrote:
> The first two errors are store errors; the third is a broker error
> related to policy handling (in QueuePolicy.cpp). However, these errors
> may be related.
>
> I would be interested in reproducing these errors, can you supply
> additional details, including the hardware (ie, how many processors, how
> much memory and disk space), OS and version you are using, also if
> possible a simple reproducer.
>
> None of these errors should show up during normal usage, and indicate a
> possible race condition or logic error.
>
I think the broker error is caused by the store error plus a bug in the
broker:
the queue policy count is incremented before the call to the store, when
the
store throws the count is not decremented so its incorrect since the
message
didn't actually get on the queue.
>
> On Wed, 2011-03-30 at 17:20 +0400, Sergey_Zhemzhitsky@troika.ru wrote:
>> Hi there,
>>
>> I'm using qpid 0.8 and sending only durable messages.
>>
>> From time to time the following exceptions can occur:
>>
>> 1. Journal "MyQueue": get_events() returned JERR_JCNTL_AIOCMPLWAIT;
>> wmgr_status: wmgr: pi=29 pc=44 po=0 aer=32 edac:TFFF
>> ps=[AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA] wrfc: state: Active fcntl[6]:
pfid=6
>> ws=11268 wc=3256 rs=0 rc=0 ec=1282 ac=32
>> 2. Journal "MyQueue": Unexpected I/O response (RHM_IORES_BUSY) on queue
>> MyQueue
>> 3. error Unexpected exception: Attempted size underflow on
dequeue(559):
>> size: max=104857600, current=119; count: unlimited; type=reject
>>
>> From the source code I know that
>>
>> JERR_JCNTL_AIOCMPLWAIT means that "Timeout waiting for AIOs to
complete"
>> RHM_IORES_BUSY means that "Another blocking operation is in progress"
>>
>> but that all and I can't go any further.
>>
>> Could you please help to determine what kind of I/O operations can
cause
>> such errors and why they can occur?
>>
>>
>> Best Regards,
>> Sergey Zhemzhitsky
>> Information Technology Division
>> Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
>> Phone. +7 495 2580500 ext. 1246
>> _______________________________________________________
>>
>> The information contained in this message may be privileged and conf
idential and protected from disclosure. If you are not the original
intended recipient, you are hereby notified that any review,
retransmission, dissemination, or other use of, or taking of any action in
reliance upon, this information is prohibited. If you have received this
communication in error, please notify the sender immediately by replying
to this message and delete it from your computer. Thank you for your
cooperation. Troika Dialog, Russia.
>> If you need assistance please contact our Contact Center (+7495) 258
0500 or go to www.troika.ru/eng/Contacts/system.wbp
>>
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
Re: I/O Errors
Posted by Alan Conway <ac...@redhat.com>.
On 03/31/2011 09:33 AM, Kim van der Riet wrote:
> The first two errors are store errors; the third is a broker error
> related to policy handling (in QueuePolicy.cpp). However, these errors
> may be related.
>
> I would be interested in reproducing these errors, can you supply
> additional details, including the hardware (ie, how many processors, how
> much memory and disk space), OS and version you are using, also if
> possible a simple reproducer.
>
> None of these errors should show up during normal usage, and indicate a
> possible race condition or logic error.
>
I think the broker error is caused by the store error plus a bug in the broker:
the queue policy count is incremented before the call to the store, when the
store throws the count is not decremented so its incorrect since the message
didn't actually get on the queue.
>
> On Wed, 2011-03-30 at 17:20 +0400, Sergey_Zhemzhitsky@troika.ru wrote:
>> Hi there,
>>
>> I'm using qpid 0.8 and sending only durable messages.
>>
>> From time to time the following exceptions can occur:
>>
>> 1. Journal "MyQueue": get_events() returned JERR_JCNTL_AIOCMPLWAIT;
>> wmgr_status: wmgr: pi=29 pc=44 po=0 aer=32 edac:TFFF
>> ps=[AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA] wrfc: state: Active fcntl[6]: pfid=6
>> ws=11268 wc=3256 rs=0 rc=0 ec=1282 ac=32
>> 2. Journal "MyQueue": Unexpected I/O response (RHM_IORES_BUSY) on queue
>> MyQueue
>> 3. error Unexpected exception: Attempted size underflow on dequeue(559):
>> size: max=104857600, current=119; count: unlimited; type=reject
>>
>> From the source code I know that
>>
>> JERR_JCNTL_AIOCMPLWAIT means that "Timeout waiting for AIOs to complete"
>> RHM_IORES_BUSY means that "Another blocking operation is in progress"
>>
>> but that all and I can't go any further.
>>
>> Could you please help to determine what kind of I/O operations can cause
>> such errors and why they can occur?
>>
>>
>> Best Regards,
>> Sergey Zhemzhitsky
>> Information Technology Division
>> Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
>> Phone. +7 495 2580500 ext. 1246
>> _______________________________________________________
>>
>> The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
>> If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
>>
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: I/O Errors
Posted by Kim van der Riet <ki...@redhat.com>.
The first two errors are store errors; the third is a broker error
related to policy handling (in QueuePolicy.cpp). However, these errors
may be related.
I would be interested in reproducing these errors, can you supply
additional details, including the hardware (ie, how many processors, how
much memory and disk space), OS and version you are using, also if
possible a simple reproducer.
None of these errors should show up during normal usage, and indicate a
possible race condition or logic error.
Thanks,
Kim
On Wed, 2011-03-30 at 17:20 +0400, Sergey_Zhemzhitsky@troika.ru wrote:
> Hi there,
>
> I'm using qpid 0.8 and sending only durable messages.
>
> From time to time the following exceptions can occur:
>
> 1. Journal "MyQueue": get_events() returned JERR_JCNTL_AIOCMPLWAIT;
> wmgr_status: wmgr: pi=29 pc=44 po=0 aer=32 edac:TFFF
> ps=[AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA] wrfc: state: Active fcntl[6]: pfid=6
> ws=11268 wc=3256 rs=0 rc=0 ec=1282 ac=32
> 2. Journal "MyQueue": Unexpected I/O response (RHM_IORES_BUSY) on queue
> MyQueue
> 3. error Unexpected exception: Attempted size underflow on dequeue(559):
> size: max=104857600, current=119; count: unlimited; type=reject
>
> From the source code I know that
>
> JERR_JCNTL_AIOCMPLWAIT means that "Timeout waiting for AIOs to complete"
> RHM_IORES_BUSY means that "Another blocking operation is in progress"
>
> but that all and I can't go any further.
>
> Could you please help to determine what kind of I/O operations can cause
> such errors and why they can occur?
>
>
> Best Regards,
> Sergey Zhemzhitsky
> Information Technology Division
> Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
> Phone. +7 495 2580500 ext. 1246
> _______________________________________________________
>
> The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
> If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org