You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by surya prakash <as...@gmail.com> on 2011/10/11 11:40:03 UTC
problem with qpid receiving
Hi ,
I am new to qpidd ,due to persistence queue concept i am using qpidd .
To get the concept i wrote a sample program in this when receiving messages
i killed my receiver application ,In this case data is losing very huge. can
you please solve this problem.
I am using qpidd::client API , the above case i tried with LocalQueue class
and MessageListner in both case is same.
Best Regards,
A.SURYA PRAKASH REDDY .
**
*
*
Re: problem with qpid receiving
Posted by Gordon Sim <gs...@redhat.com>.
On 10/13/2011 07:45 AM, surya prakash wrote:
> Is it possible to set max_count in qpid::client API.i am able to
> do this in qpidd::message API.
Yes, e.g.
FieldTable args;
args.setInt("qpid.max_count",10);
session.queueDeclare(arg::queue="my-queue", arg::arguments=args);
(but i would still recommend the messaging API ;-)
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: problem with qpid receiving
Posted by surya prakash <as...@gmail.com>.
Hi,
Is it possible to set max_count in qpid::client API.i am able to
do this in qpidd::message API.
Regards,
surya
Re: problem with qpid receiving
Posted by Gordon Sim <gs...@redhat.com>.
On 10/12/2011 12:36 PM, surya prakash wrote:
> I already tried sync() .but this is slow i am not able to achieve 5000TPS. i
> don't want to compromise on TPS. any way like this
How often were you calling sync()? Your configured queue depth is quite
large so you shouldn't need to do it that often.
However if you really don't want to do that then you can simply flush()
and then keep track of incomplete transfers and ensure that the number
of those does not get too high (the messaging API does this for you).
Alternatively you can build in some feedback from consumers, e.g. by
having them respond to a reply-to address every N messages. That way you
can track the queue depth even more directly.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: problem with qpid receiving
Posted by surya prakash <as...@gmail.com>.
I already tried sync() .but this is slow i am not able to achieve 5000TPS. i
don't want to compromise on TPS. any way like this
On Wed, Oct 12, 2011 at 4:58 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 10/12/2011 11:12 AM, surya prakash wrote:
>
>> Hi,
>>
>> I want to stop when i will get this message because i feel that the
>> consumer
>> is not running if send with out considering this means messages will be
>> loss.when i am sending it asynchronously it is not passable to know when
>> exactly this is happened.give any solution i want minimum 5000 TPS with
>> out
>> data lose
>>
>
> If you are using a recent broker (e.g. 0.10 or later), there is a mechanism
> to help flow control producers. To utilise this, I would suggest
> periodically syncing the session (or the list of in doubt message
> transfers). The sync will block until there is sufficient capacity on the
> queue, thus preventing the publisher exceeding the limit.
>
>
> ------------------------------**------------------------------**---------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.**apache.org<us...@qpid.apache.org>
>
>
--
**
*Thanks and Regards,*
*A.SURYA PRAKASH REDDY*
***Ph.No:+91 7676724614*
Re: problem with qpid receiving
Posted by Gordon Sim <gs...@redhat.com>.
On 10/12/2011 11:12 AM, surya prakash wrote:
> Hi,
>
> I want to stop when i will get this message because i feel that the consumer
> is not running if send with out considering this means messages will be
> loss.when i am sending it asynchronously it is not passable to know when
> exactly this is happened.give any solution i want minimum 5000 TPS with out
> data lose
If you are using a recent broker (e.g. 0.10 or later), there is a
mechanism to help flow control producers. To utilise this, I would
suggest periodically syncing the session (or the list of in doubt
message transfers). The sync will block until there is sufficient
capacity on the queue, thus preventing the publisher exceeding the limit.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: problem with qpid receiving
Posted by surya prakash <as...@gmail.com>.
Hi,
I want to stop when i will get this message because i feel that the consumer
is not running if send with out considering this means messages will be
loss.when i am sending it asynchronously it is not passable to know when
exactly this is happened.give any solution i want minimum 5000 TPS with out
data lose
Best Regards,
surya
On Wed, Oct 12, 2011 at 2:55 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 10/12/2011 09:52 AM, surya prakash wrote:
>
>> HI pavel,
>>
>> "resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on
>> TEST,
>> policy: size: max=4000000, current=3999984; count: unlimited; type=reject
>> (qpid/broker/QueuePolicy.cpp:**87)"
>>
>
> That means that the queue is reaching 4000000 of aggregate message content,
> which is set to be the limit.
>
> You can increase that if you expect the queue to grow deeper than this.
> However usually it means that your consumers are unable to keep up with your
> producers and you need to consider some way of addressing that.
>
>
> ------------------------------**------------------------------**---------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.**apache.org<us...@qpid.apache.org>
>
>
--
**
*Thanks and Regards,*
*A.SURYA PRAKASH REDDY*
***Ph.No:+91 7676724614*
Re: problem with qpid receiving
Posted by Gordon Sim <gs...@redhat.com>.
On 10/12/2011 09:52 AM, surya prakash wrote:
> HI pavel,
>
> "resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on TEST,
> policy: size: max=4000000, current=3999984; count: unlimited; type=reject
> (qpid/broker/QueuePolicy.cpp:87)"
That means that the queue is reaching 4000000 of aggregate message
content, which is set to be the limit.
You can increase that if you expect the queue to grow deeper than this.
However usually it means that your consumers are unable to keep up with
your producers and you need to consider some way of addressing that.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: problem with qpid receiving
Posted by surya prakash <as...@gmail.com>.
HI pavel,
"resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on TEST,
policy: size: max=4000000, current=3999984; count: unlimited; type=reject
(qpid/broker/QueuePolicy.cpp:87)"
this is the exception i am getting in this TEST queue name.
Regards,
surya
On Wed, Oct 12, 2011 at 2:18 PM, Pavel Moravec <pm...@redhat.com> wrote:
> Hi Surya,
> what parameters do you have set up for the queue that rejects the messages?
> (queue name should be written in logs, e.g.)
>
> Kind regards,
> Pavel
>
>
> ----- Original Message -----
> > From: "surya prakash" <as...@gmail.com>
> > To: users@qpid.apache.org
> > Sent: Wednesday, October 12, 2011 8:40:59 AM
> > Subject: Re: problem with qpid receiving
> >
> > i have one more problem when i am sending messages asynchronously i
> > am
> > getting "resource-limit-exceeded: resource-limit-exceeded: Policy
> > exceeded"
> > after going some messages which are getting rejected by the qpidd.
> > is there
> > any way to solve this instead of reject messages are in pending means
> > my
> > problem will be solved is there any way to make it as pending.
> >
> >
> > On Tue, Oct 11, 2011 at 6:31 PM, Gordon Sim <gs...@redhat.com> wrote:
> >
> > > On 10/11/2011 01:53 PM, surya prakash wrote:
> > >
> > >> do you mean need to use failOver class
> > >>
> > >
> > > No, I mean you need to somehow detect and discard duplicates. The
> > > best way
> > > to do that depends on the application and messaging pattern. At
> > > some point
> > > in the future the client libraries and/or broker will have more to
> > > offer
> > > here. For now though you need to do it yourself.
> > >
> > >
> > >
> ------------------------------**------------------------------**---------
> > > Apache Qpid - AMQP Messaging Implementation
> > > Project: http://qpid.apache.org
> > > Use/Interact:
> > > mailto:users-subscribe@qpid.**apache.org<
> users-subscribe@qpid.apache.org>
> > >
> > >
> >
> >
> > --
> >
> > **
> > *Thanks and Regards,*
> > *A.SURYA PRAKASH REDDY*
> > ***Ph.No:+91 7676724614*
> >
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
--
**
*Thanks and Regards,*
*A.SURYA PRAKASH REDDY*
***Ph.No:+91 7676724614*
Re: problem with qpid receiving
Posted by Pavel Moravec <pm...@redhat.com>.
Hi Surya,
what parameters do you have set up for the queue that rejects the messages? (queue name should be written in logs, e.g.)
Kind regards,
Pavel
----- Original Message -----
> From: "surya prakash" <as...@gmail.com>
> To: users@qpid.apache.org
> Sent: Wednesday, October 12, 2011 8:40:59 AM
> Subject: Re: problem with qpid receiving
>
> i have one more problem when i am sending messages asynchronously i
> am
> getting "resource-limit-exceeded: resource-limit-exceeded: Policy
> exceeded"
> after going some messages which are getting rejected by the qpidd.
> is there
> any way to solve this instead of reject messages are in pending means
> my
> problem will be solved is there any way to make it as pending.
>
>
> On Tue, Oct 11, 2011 at 6:31 PM, Gordon Sim <gs...@redhat.com> wrote:
>
> > On 10/11/2011 01:53 PM, surya prakash wrote:
> >
> >> do you mean need to use failOver class
> >>
> >
> > No, I mean you need to somehow detect and discard duplicates. The
> > best way
> > to do that depends on the application and messaging pattern. At
> > some point
> > in the future the client libraries and/or broker will have more to
> > offer
> > here. For now though you need to do it yourself.
> >
> >
> > ------------------------------**------------------------------**---------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact:
> > mailto:users-subscribe@qpid.**apache.org<us...@qpid.apache.org>
> >
> >
>
>
> --
>
> **
> *Thanks and Regards,*
> *A.SURYA PRAKASH REDDY*
> ***Ph.No:+91 7676724614*
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: problem with qpid receiving
Posted by surya prakash <as...@gmail.com>.
i have one more problem when i am sending messages asynchronously i am
getting "resource-limit-exceeded: resource-limit-exceeded: Policy exceeded"
after going some messages which are getting rejected by the qpidd. is there
any way to solve this instead of reject messages are in pending means my
problem will be solved is there any way to make it as pending.
On Tue, Oct 11, 2011 at 6:31 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 10/11/2011 01:53 PM, surya prakash wrote:
>
>> do you mean need to use failOver class
>>
>
> No, I mean you need to somehow detect and discard duplicates. The best way
> to do that depends on the application and messaging pattern. At some point
> in the future the client libraries and/or broker will have more to offer
> here. For now though you need to do it yourself.
>
>
> ------------------------------**------------------------------**---------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.**apache.org<us...@qpid.apache.org>
>
>
--
**
*Thanks and Regards,*
*A.SURYA PRAKASH REDDY*
***Ph.No:+91 7676724614*
Re: problem with qpid receiving
Posted by Gordon Sim <gs...@redhat.com>.
On 10/11/2011 01:53 PM, surya prakash wrote:
> do you mean need to use failOver class
No, I mean you need to somehow detect and discard duplicates. The best
way to do that depends on the application and messaging pattern. At some
point in the future the client libraries and/or broker will have more to
offer here. For now though you need to do it yourself.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: problem with qpid receiving
Posted by surya prakash <as...@gmail.com>.
do you mean need to use failOver class
On Tue, Oct 11, 2011 at 6:20 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 10/11/2011 01:45 PM, surya prakash wrote:
>
>> If i use ACCEPT_MODE_EXPLICIT then i am getting duplicate entry's that's
>> the reason i user ACCEPT_MODE_NONE
>>
>
> Correct. You can choose between unreliable or at-least-once (meaning
> duplicates are possible). To get exactly-once (which I think is what you
> want) you need to de-duplicate in your application.
>
> ------------------------------**------------------------------**---------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.**apache.org<us...@qpid.apache.org>
>
>
--
**
*Thanks and Regards,*
*A.SURYA PRAKASH REDDY*
***Ph.No:+91 7676724614*
Re: problem with qpid receiving
Posted by Gordon Sim <gs...@redhat.com>.
On 10/11/2011 01:45 PM, surya prakash wrote:
> If i use ACCEPT_MODE_EXPLICIT then i am getting duplicate entry's that's
> the reason i user ACCEPT_MODE_NONE
Correct. You can choose between unreliable or at-least-once (meaning
duplicates are possible). To get exactly-once (which I think is what you
want) you need to de-duplicate in your application.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: problem with qpid receiving
Posted by surya prakash <as...@gmail.com>.
If i use ACCEPT_MODE_EXPLICIT then i am getting duplicate entry's that's
the reason i user ACCEPT_MODE_NONE
On Tue, Oct 11, 2011 at 6:10 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 10/11/2011 01:36 PM, surya prakash wrote:
>
>> Hi,
>>
>> i used 2 programs one for sending simple data like "Test and some
>> unique id" and one for receiving.
>> when receiving i used
>> SubscriptionSettings
>> SubsSetting(FlowControl::**unlimited(),ACCEPT_MODE_NONE,**
>> ACQUIRE_MODE_PRE_ACQUIRED,1,**COMPLETE_ON_ACCEPT);
>>
>
> ACCEPT_MODE_NONE means messages are not acknowledged but are assumed
> delivered as soon as the broker sends them out. This is inherently
> unreliable and will result in any in-flight messages being lost in the event
> of a crash.
>
>
> ------------------------------**------------------------------**---------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.**apache.org<us...@qpid.apache.org>
>
>
--
**
*Thanks and Regards,*
*A.SURYA PRAKASH REDDY*
***Ph.No:+91 7676724614*
Re: problem with qpid receiving
Posted by Gordon Sim <gs...@redhat.com>.
On 10/11/2011 01:36 PM, surya prakash wrote:
> Hi,
>
> i used 2 programs one for sending simple data like "Test and some
> unique id" and one for receiving.
> when receiving i used
> SubscriptionSettings
> SubsSetting(FlowControl::unlimited(),ACCEPT_MODE_NONE,ACQUIRE_MODE_PRE_ACQUIRED,1,COMPLETE_ON_ACCEPT);
ACCEPT_MODE_NONE means messages are not acknowledged but are assumed
delivered as soon as the broker sends them out. This is inherently
unreliable and will result in any in-flight messages being lost in the
event of a crash.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: problem with qpid receiving
Posted by surya prakash <as...@gmail.com>.
Hi,
i used 2 programs one for sending simple data like "Test and some
unique id" and one for receiving.
when receiving i used
SubscriptionSettings
SubsSetting(FlowControl::unlimited(),ACCEPT_MODE_NONE,ACQUIRE_MODE_PRE_ACQUIRED,1,COMPLETE_ON_ACCEPT);
and i am using messageListner class for receiving messages.
On Tue, Oct 11, 2011 at 5:22 PM, Andy Goldstein <ag...@redhat.com> wrote:
> Could you please describe what your test program is doing in more detail -
> perhaps post some sample code? When you say you're losing messages, where is
> the loss occurring? If you've received and acknowledged messages, and then
> your app is killed/crashes, these messages will be lost - once a message has
> been acknowledged, qpid is no longer responsible for it.
>
> Also, the qpid::messaging API is the preferred one going forward, so I
> would recommend using that if at all possible.
>
> Andy
>
> On Oct 11, 2011, at 5:41 AM, surya prakash <as...@gmail.com>
> wrote:
>
> > Hi ,
> > I am new to qpidd ,due to persistence queue concept i am using qpidd
> .
> > To get the concept i wrote a sample program in this when receiving
> messages
> > i killed my receiver application ,In this case data is losing very huge.
> can
> > you please solve this problem.
> >
> > I am using qpidd::client API , the above case i tried with LocalQueue
> class
> > and MessageListner in both case is same.
> >
> > Best Regards,
> > A.SURYA PRAKASH REDDY .
> >
> >
> >
> >
> > **
> > *
> > *
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
--
**
*Thanks and Regards,*
*A.SURYA PRAKASH REDDY*
***Ph.No:+91 7676724614*
Re: problem with qpid receiving
Posted by Andy Goldstein <ag...@redhat.com>.
Could you please describe what your test program is doing in more detail - perhaps post some sample code? When you say you're losing messages, where is the loss occurring? If you've received and acknowledged messages, and then your app is killed/crashes, these messages will be lost - once a message has been acknowledged, qpid is no longer responsible for it.
Also, the qpid::messaging API is the preferred one going forward, so I would recommend using that if at all possible.
Andy
On Oct 11, 2011, at 5:41 AM, surya prakash <as...@gmail.com> wrote:
> Hi ,
> I am new to qpidd ,due to persistence queue concept i am using qpidd .
> To get the concept i wrote a sample program in this when receiving messages
> i killed my receiver application ,In this case data is losing very huge. can
> you please solve this problem.
>
> I am using qpidd::client API , the above case i tried with LocalQueue class
> and MessageListner in both case is same.
>
> Best Regards,
> A.SURYA PRAKASH REDDY .
>
>
>
>
> **
> *
> *
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org