You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by tide08 <sa...@yahoo.com> on 2010/03/18 08:40:45 UTC
ScheduledPollConsumer - poll() behavior?
Hi,
I am writing custom component and I need a polling consumer but I am not
sure, if the approach I am taking is correct. I need create
scheduledPollingConsumer which polls external system to receive messages.
Looking at other components SchedulePollingConsumers implement poll() method
which works with "known" number of messages i.e. Mail, File - each know what
number of messages to work with inside of poll()
//mail
int count = folder.getMessageCount();
But in my scenario, there is no way to know it until you receive "null"
message from external system. So as I understand in my scenario, if there
are tonnes of messages, this consumer will keep running indefinitely. How
would this affect threadpool?
How are ScheduledPollingConsumer supposed to behave? Consume as much in
polling interval? or consume restrictively so that the scheduled run ends
before the next schedule?
Please advice. Thanks!
--
View this message in context: http://old.nabble.com/ScheduledPollConsumer---poll%28%29-behavior--tp27942516p27942516.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: ScheduledPollConsumer - poll() behavior?
Posted by tide08 <sa...@yahoo.com>.
Ok, Thanks!
As I see most of the components use maxMessagesPerPoll to limit processing
of messages, after they have polled?
Ex: For mail component - int count = folder.getMessageCount(); might return
10,000 but it only process as per maxMessagesPerPoll - say 1000.
And rest 9000 will be available for precessing in next poll when it invokes
folder.getMessageCount(), right?
In my scenario, external component hands off the message than those are not
available back for 5mins. So I do not want to getMessage() unless I can
process it right away.
Would it make sense to limit getMessage() by maxMessagesPerPoll?
Thanks!
Claus Ibsen-2 wrote:
>
> On Thu, Mar 18, 2010 at 8:40 AM, tide08 <sa...@yahoo.com> wrote:
>>
>> Hi,
>>
>> I am writing custom component and I need a polling consumer but I am not
>> sure, if the approach I am taking is correct. I need create
>> scheduledPollingConsumer which polls external system to receive messages.
>>
>> Looking at other components SchedulePollingConsumers implement poll()
>> method
>> which works with "known" number of messages i.e. Mail, File - each know
>> what
>> number of messages to work with inside of poll()
>>
>> //mail
>> int count = folder.getMessageCount();
>>
>> But in my scenario, there is no way to know it until you receive "null"
>> message from external system. So as I understand in my scenario, if there
>> are tonnes of messages, this consumer will keep running indefinitely. How
>> would this affect threadpool?
>>
>> How are ScheduledPollingConsumer supposed to behave? Consume as much in
>> polling interval? or consume restrictively so that the scheduled run ends
>> before the next schedule?
>>
>
> There is only one task scheduled, so if your poll takes 5 seconds or
> 37 minutes to complete does not matter.
>
> You can add an option to limit the number of intakes as we have the
> maxMessagesPerPoll option on eg BatchConsumers.
> If you are worried to run _forever_ :)
>
>
>> Please advice. Thanks!
>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/ScheduledPollConsumer---poll%28%29-behavior--tp27942516p27942516.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/ScheduledPollConsumer---poll%28%29-behavior--tp27942516p27947271.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: ScheduledPollConsumer - poll() behavior?
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Mar 18, 2010 at 8:40 AM, tide08 <sa...@yahoo.com> wrote:
>
> Hi,
>
> I am writing custom component and I need a polling consumer but I am not
> sure, if the approach I am taking is correct. I need create
> scheduledPollingConsumer which polls external system to receive messages.
>
> Looking at other components SchedulePollingConsumers implement poll() method
> which works with "known" number of messages i.e. Mail, File - each know what
> number of messages to work with inside of poll()
>
> //mail
> int count = folder.getMessageCount();
>
> But in my scenario, there is no way to know it until you receive "null"
> message from external system. So as I understand in my scenario, if there
> are tonnes of messages, this consumer will keep running indefinitely. How
> would this affect threadpool?
>
> How are ScheduledPollingConsumer supposed to behave? Consume as much in
> polling interval? or consume restrictively so that the scheduled run ends
> before the next schedule?
>
There is only one task scheduled, so if your poll takes 5 seconds or
37 minutes to complete does not matter.
You can add an option to limit the number of intakes as we have the
maxMessagesPerPoll option on eg BatchConsumers.
If you are worried to run _forever_ :)
> Please advice. Thanks!
>
>
> --
> View this message in context: http://old.nabble.com/ScheduledPollConsumer---poll%28%29-behavior--tp27942516p27942516.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: ScheduledPollConsumer - poll() behavior?
Posted by Claus Ibsen <cl...@gmail.com>.
On Sat, Mar 20, 2010 at 8:43 PM, tide08 <sa...@yahoo.com> wrote:
>
> So, would it make sense that in my customConsumer that extends
> ScheduledPollConsumer, I override doStart() to schedule more tasks? Is this
> thread-safe to do?
>
You can do that.
Also nothing is thread safe out of the box, its something that YOU
have to ensure.
And it depends what the scheduled task is doing etc.
> @Override
> protected void doStart() throws Exception {
>
> if (log.isDebugEnabled()) {
> log.debug("Starting consumer: " + this);
> }
>
> ServiceHelper.startServices(getProcessor());
>
> for(int i=0; i < threadPoolSize; i++){
> if (isUseFixedDelay()) {
> executorService.scheduleWithFixedDelay(this,
> getInitialDelay(), getDelay(), getTimeUnit());
> } else {
> executorService.scheduleAtFixedRate(this, getInitialDelay(),
> getDelay(), getTimeUnit());
> }
> }
> }
>
>
>
>
> Claus Ibsen-2 wrote:
>>
>> On Fri, Mar 19, 2010 at 2:49 AM, tide08 <sa...@yahoo.com> wrote:
>>>
>>> Sorry, I am much confused at this point :(
>>>
>>> In regards to ScheduledPollConsumer, is poll() method invoked by
>>> concurrent
>>> consumers? or is it just one consumer? I just see one thread invoking
>>> poll(), what is threadpool for than?
>>>
>>
>> Yes 1 thread. The thread pool can schedule as its a
>> ScheduledExecutorService.
>>
>>
>>> Thanks!
>>>
>>>
>>>
>>> tide08 wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am writing custom component and I need a polling consumer but I am not
>>>> sure, if the approach I am taking is correct. I need create
>>>> scheduledPollingConsumer which polls external system to receive
>>>> messages.
>>>>
>>>> Looking at other components SchedulePollingConsumers implement poll()
>>>> method which works with "known" number of messages i.e. Mail, File -
>>>> each
>>>> know what number of messages to work with inside of poll()
>>>>
>>>> //mail
>>>> int count = folder.getMessageCount();
>>>>
>>>> But in my scenario, there is no way to know it until you receive "null"
>>>> message from external system. So as I understand in my scenario, if
>>>> there
>>>> are tonnes of messages, this consumer will keep running indefinitely.
>>>> How
>>>> would this affect threadpool?
>>>>
>>>> How are ScheduledPollingConsumer supposed to behave? Consume as much in
>>>> polling interval? or consume restrictively so that the scheduled run
>>>> ends
>>>> before the next schedule?
>>>>
>>>> Please advice. Thanks!
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/ScheduledPollConsumer---poll%28%29-behavior--tp27942516p27950874.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/ScheduledPollConsumer---poll%28%29-behavior--tp27942516p27968018.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: ScheduledPollConsumer - poll() behavior?
Posted by tide08 <sa...@yahoo.com>.
So, would it make sense that in my customConsumer that extends
ScheduledPollConsumer, I override doStart() to schedule more tasks? Is this
thread-safe to do?
@Override
protected void doStart() throws Exception {
if (log.isDebugEnabled()) {
log.debug("Starting consumer: " + this);
}
ServiceHelper.startServices(getProcessor());
for(int i=0; i < threadPoolSize; i++){
if (isUseFixedDelay()) {
executorService.scheduleWithFixedDelay(this,
getInitialDelay(), getDelay(), getTimeUnit());
} else {
executorService.scheduleAtFixedRate(this, getInitialDelay(),
getDelay(), getTimeUnit());
}
}
}
Claus Ibsen-2 wrote:
>
> On Fri, Mar 19, 2010 at 2:49 AM, tide08 <sa...@yahoo.com> wrote:
>>
>> Sorry, I am much confused at this point :(
>>
>> In regards to ScheduledPollConsumer, is poll() method invoked by
>> concurrent
>> consumers? or is it just one consumer? I just see one thread invoking
>> poll(), what is threadpool for than?
>>
>
> Yes 1 thread. The thread pool can schedule as its a
> ScheduledExecutorService.
>
>
>> Thanks!
>>
>>
>>
>> tide08 wrote:
>>>
>>> Hi,
>>>
>>> I am writing custom component and I need a polling consumer but I am not
>>> sure, if the approach I am taking is correct. I need create
>>> scheduledPollingConsumer which polls external system to receive
>>> messages.
>>>
>>> Looking at other components SchedulePollingConsumers implement poll()
>>> method which works with "known" number of messages i.e. Mail, File -
>>> each
>>> know what number of messages to work with inside of poll()
>>>
>>> //mail
>>> int count = folder.getMessageCount();
>>>
>>> But in my scenario, there is no way to know it until you receive "null"
>>> message from external system. So as I understand in my scenario, if
>>> there
>>> are tonnes of messages, this consumer will keep running indefinitely.
>>> How
>>> would this affect threadpool?
>>>
>>> How are ScheduledPollingConsumer supposed to behave? Consume as much in
>>> polling interval? or consume restrictively so that the scheduled run
>>> ends
>>> before the next schedule?
>>>
>>> Please advice. Thanks!
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/ScheduledPollConsumer---poll%28%29-behavior--tp27942516p27950874.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/ScheduledPollConsumer---poll%28%29-behavior--tp27942516p27968018.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: ScheduledPollConsumer - poll() behavior?
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Mar 19, 2010 at 2:49 AM, tide08 <sa...@yahoo.com> wrote:
>
> Sorry, I am much confused at this point :(
>
> In regards to ScheduledPollConsumer, is poll() method invoked by concurrent
> consumers? or is it just one consumer? I just see one thread invoking
> poll(), what is threadpool for than?
>
Yes 1 thread. The thread pool can schedule as its a ScheduledExecutorService.
> Thanks!
>
>
>
> tide08 wrote:
>>
>> Hi,
>>
>> I am writing custom component and I need a polling consumer but I am not
>> sure, if the approach I am taking is correct. I need create
>> scheduledPollingConsumer which polls external system to receive messages.
>>
>> Looking at other components SchedulePollingConsumers implement poll()
>> method which works with "known" number of messages i.e. Mail, File - each
>> know what number of messages to work with inside of poll()
>>
>> //mail
>> int count = folder.getMessageCount();
>>
>> But in my scenario, there is no way to know it until you receive "null"
>> message from external system. So as I understand in my scenario, if there
>> are tonnes of messages, this consumer will keep running indefinitely. How
>> would this affect threadpool?
>>
>> How are ScheduledPollingConsumer supposed to behave? Consume as much in
>> polling interval? or consume restrictively so that the scheduled run ends
>> before the next schedule?
>>
>> Please advice. Thanks!
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/ScheduledPollConsumer---poll%28%29-behavior--tp27942516p27950874.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: ScheduledPollConsumer - poll() behavior?
Posted by tide08 <sa...@yahoo.com>.
Sorry, I am much confused at this point :(
In regards to ScheduledPollConsumer, is poll() method invoked by concurrent
consumers? or is it just one consumer? I just see one thread invoking
poll(), what is threadpool for than?
Thanks!
tide08 wrote:
>
> Hi,
>
> I am writing custom component and I need a polling consumer but I am not
> sure, if the approach I am taking is correct. I need create
> scheduledPollingConsumer which polls external system to receive messages.
>
> Looking at other components SchedulePollingConsumers implement poll()
> method which works with "known" number of messages i.e. Mail, File - each
> know what number of messages to work with inside of poll()
>
> //mail
> int count = folder.getMessageCount();
>
> But in my scenario, there is no way to know it until you receive "null"
> message from external system. So as I understand in my scenario, if there
> are tonnes of messages, this consumer will keep running indefinitely. How
> would this affect threadpool?
>
> How are ScheduledPollingConsumer supposed to behave? Consume as much in
> polling interval? or consume restrictively so that the scheduled run ends
> before the next schedule?
>
> Please advice. Thanks!
>
>
>
--
View this message in context: http://old.nabble.com/ScheduledPollConsumer---poll%28%29-behavior--tp27942516p27950874.html
Sent from the Camel - Users mailing list archive at Nabble.com.