You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by cgiera <ch...@mic-cust.com> on 2013/05/23 11:01:24 UTC
MailComponent/MailConsumer misses Mails
Hello,
I'm facing strange problems when consuming mails from a cyrus mail server.
Some mails were not processed but appear as SEEN in the mailbox.
I've added the mail debug option to the route and I was pretty confused when
analyzing the log:
First poll is quite normal: 14 new mails, 14 get fetched and processed.
Second poll is very confusing to me:
14 new messages and only 11 get fetched.
Missing emails are 266, 267 and 268, but they still exist in the
mailbox(marked as SEEN).
Our camel application was (and still is) the only client which was connected
to this mailbox.
So I've to assume that the camel MailComponent has marked as seen(but not
processed the mail).
We are using camel 2.6.0 and cyrus-imapd-2.3.16-6.el6_2.5.x86_64.
The route looks like the following:
This behaviour is almost everytime reproducable when loading "a lot" of
messages to the mailbox(~25+).
Mailserver log shows that our camel application is the only connected
client.
Any help/advice/hint would be nice, I don't have a clue what I can do or
where I should start my next analyzes.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Christian Mueller wrote
> Can you send me the entire log file. I need more context than only two
> lines...
Above of the 2 lines you can find a big part of the log file, I thought this
will be enough, but sure I can
send you the 30 MB zip files :).
Christian Mueller wrote
> Do you have multiple routes polling the same INBOX?
No.
Christian Mueller wrote
> Did you stopped/started the routes (may we have an issue not correctly
> stopping the rouet)?
I have stopped the MailProducer routes when testing one time, not the
consumer route.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734383.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Müller <ch...@gmail.com>.
Can you send me the entire log file. I need more context than only two
lines...
Do you have multiple routes polling the same INBOX?
Did you stopped/started the routes (may we have an issue not correctly
stopping the rouet)?
Best,
Christian Müller
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Wed, Jun 19, 2013 at 12:58 PM, cgiera <ch...@mic-cust.com>wrote:
> Here is another part of the logfile where this problem occures:
>
> Here is something really strange:
> INFO | jvm 1 | 2013/06/19 11:30:37 | 2013-06-19 11:30:37,237 [Camel
> (camel) thread #2 - imap://<ipaddress>] DEBUG - Polling mailbox folder:
> imap://<ipaddress>:143, folder=INBOX
> INFO | jvm 1 | 2013/06/19 11:30:37 | 2013-06-19 11:30:37,300 [Camel
> (camel) thread #8 - imap://<ipaddress>] DEBUG - Polling mailbox folder:
> imap://<ipaddress>:143, folder=INBOX
>
> 2 Threads which are polling at the same time.
>
> kind regards,
> Christoph
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734367.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Here is another part of the logfile where this problem occures:
Here is something really strange:
INFO | jvm 1 | 2013/06/19 11:30:37 | 2013-06-19 11:30:37,237 [Camel
(camel) thread #2 - imap://<ipaddress>] DEBUG - Polling mailbox folder:
imap://<ipaddress>:143, folder=INBOX
INFO | jvm 1 | 2013/06/19 11:30:37 | 2013-06-19 11:30:37,300 [Camel
(camel) thread #8 - imap://<ipaddress>] DEBUG - Polling mailbox folder:
imap://<ipaddress>:143, folder=INBOX
2 Threads which are polling at the same time.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734367.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
now with java mail 1.4.5 and log entries(have added slf4j_log4j.jar to
ensure debug messages occure):
It's the same behavior and the same log entries,
missing mail logs:
INFO | jvm 1 | 2013/06/19 11:54:23 | * 2 RECENT
INFO | jvm 1 | 2013/06/19 11:54:23 | * SEARCH
INFO | jvm 1 | 2013/06/19 11:54:23 | A9329 OK Completed (0 msgs in
0.000 secs)
For comparing, here it works:
INFO | jvm 1 | 2013/06/19 11:54:44 | * 2 RECENT
INFO | jvm 1 | 2013/06/19 11:54:44 | * SEARCH 1550 1551
INFO | jvm 1 | 2013/06/19 11:54:44 | A9357 OK Completed (2 msgs in
0.000 secs)
INFO | jvm 1 | 2013/06/19 11:54:44 | 2013-06-19 11:54:44,944 [Camel
(camel) thread #2 - imap://<ipaddress>] DEBUG - Fetching 1 messages. Total
2 messages.
Although it is only fetching 1 of 2 messages here the second message was
processed with the next run.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734364.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Müller <ch...@gmail.com>.
Using
log4j.logger.org.apache.camel.component.mail=DEBUG
should be fine.
I would like to see the log entries at the time Camel mark/flag more
messages as seen as messages were processed.
And could you please check whether do you see the same strange Java Mail
log messages (2 RECENT but no message found) if you use mail.jar 1.4.5.
Because this is the version we are using in Camel 2.11.0...
Best,
Christian Müller
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Tue, Jun 18, 2013 at 5:27 PM, cgiera <ch...@mic-cust.com>wrote:
> Hello,
>
> I've updated our customers qa system to camel 2.11.0 and mail 1.5 now.
> We've added some test routes to send mails to the mail box we are reading
> from and had a look at it
> if mails are missing.
> After ~1 hour we saw that sent mails vs. received mails was not equal
> anymore and analyzed the log files.
> I've found the log entries where something is wrong:
>
> There are 2 recent mails, but they were not fetched:
> INFO | jvm 1 | 2013/06/18 16:32:21 | * 2 RECENT
> INFO | jvm 1 | 2013/06/18 16:32:21 | * SEARCH
> INFO | jvm 1 | 2013/06/18 16:32:21 | A2749 OK Completed (0 msgs in
> 0.000 secs)
>
> For comparing here are log entries where mails are consumed correctly:
>
> 2 Recent and 2 fetched
> INFO | jvm 1 | 2013/06/18 16:32:43 | * 2 RECENT
> INFO | jvm 1 | 2013/06/18 16:32:43 | * SEARCH 1069 1070
> INFO | jvm 1 | 2013/06/18 16:32:43 | A2777 OK Completed (2 msgs in
> 0.000 secs)
>
> Here is the setup of our routes:
> Consumerroute:
>
> imap://<ip-adress>?consumer.delay=3000&consumer.useFixedDelay=true&debugMode=true&fetchSize=1&password=...._&unseen=true&username=....
>
> 2 routes which send to the mailbox:
> timer://foo?fixedRate=true&period=27000
> timer://foo2?fixedRate=true&period=29000
>
>
> I've activated mailDebug option on the consumer route and additionally some
> log4j logging:
> log4j.logger.org.apache.camel.component.mail=TRACE
> log4j.logger.org.apache.camel.impl=TRACE
>
> Pls let me know if I you need other logs or how I should adjust log4j to
> get
> more logs out of the MailComponent.
>
> kind regards,
> Christoph
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734311.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Müller <ch...@gmail.com>.
Hello Christoph!
Thanks for sharing this with us. I hope upgrading to cyrus 2.4.x will solve
your issue.
Best,
Christian
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Fri, Aug 9, 2013 at 7:56 AM, cgiera <ch...@mic-cust.com> wrote:
> Hello,
>
> it looks like we have found the problem:
> http://www.mail-archive.com/cyrus-devel@lists.andrew.cmu.edu/msg01684.html
>
> We use exactly the cyrus version mentioned in the link, it is not 100% sure
> if this is the real problem, but we are setting up a server with cyrus
> 2.4.x
> and run our tests again.
>
> As a simple workaround the option closeFolder=false looks good, we haven't
> lost a single mail in our testings.
>
> Thanks for your time and help, maybe someone will need this information in
> the future.
>
> kind regards,
> Christoph
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5737024.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
it looks like we have found the problem:
http://www.mail-archive.com/cyrus-devel@lists.andrew.cmu.edu/msg01684.html
We use exactly the cyrus version mentioned in the link, it is not 100% sure
if this is the real problem, but we are setting up a server with cyrus 2.4.x
and run our tests again.
As a simple workaround the option closeFolder=false looks good, we haven't
lost a single mail in our testings.
Thanks for your time and help, maybe someone will need this information in
the future.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5737024.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Mueller <ch...@gmail.com>.
I just updated Camel 2.11.1-SNAPSHOT to javax.mail/mail from 1.4.5 to 1.4.7
without any problems. Could you please try Camel 2.11.0 with javax.mail/mail
1.4.7?
I don't think it's a Camel issue, but we have to investigate a bit more...
Best,
Christian
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5735315.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
I've updated our customers qa system to camel 2.11.0 and mail 1.5 now.
We've added some test routes to send mails to the mail box we are reading
from and had a look at it
if mails are missing.
After ~1 hour we saw that sent mails vs. received mails was not equal
anymore and analyzed the log files.
I've found the log entries where something is wrong:
There are 2 recent mails, but they were not fetched:
INFO | jvm 1 | 2013/06/18 16:32:21 | * 2 RECENT
INFO | jvm 1 | 2013/06/18 16:32:21 | * SEARCH
INFO | jvm 1 | 2013/06/18 16:32:21 | A2749 OK Completed (0 msgs in
0.000 secs)
For comparing here are log entries where mails are consumed correctly:
2 Recent and 2 fetched
INFO | jvm 1 | 2013/06/18 16:32:43 | * 2 RECENT
INFO | jvm 1 | 2013/06/18 16:32:43 | * SEARCH 1069 1070
INFO | jvm 1 | 2013/06/18 16:32:43 | A2777 OK Completed (2 msgs in
0.000 secs)
Here is the setup of our routes:
Consumerroute:
imap://<ip-adress>?consumer.delay=3000&consumer.useFixedDelay=true&debugMode=true&fetchSize=1&password=...._&unseen=true&username=....
2 routes which send to the mailbox:
timer://foo?fixedRate=true&period=27000
timer://foo2?fixedRate=true&period=29000
I've activated mailDebug option on the consumer route and additionally some
log4j logging:
log4j.logger.org.apache.camel.component.mail=TRACE
log4j.logger.org.apache.camel.impl=TRACE
Pls let me know if I you need other logs or how I should adjust log4j to get
more logs out of the MailComponent.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734311.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
thx for the reply.
I've updated to camel 2.11.0 and java mail 1.5 before I've begun the
testing.
So all the log's from the previous posts are already from the newest
version.
On my local system I've tested the workaround with fetchSize=1, here both
versions 2.6.0 and 2.11.0 are working for me.
So we decided to get this workaround to the customer system(2.6.0), but here
the workaround is not working.
I have not found out what is the difference between the customer system and
my testings, but I don't think it is fixed in 2.11.0.
I haven't found something new in this log files, all information is also
included in the logs I've posted in the previous posts.
However, if you still need more log entries, I can post them if you want.
kind regards,
Christoph
Christian Mueller wrote
> As I already mentioned, Camel 2.6.0 is really old (released in Jan. 2011)
> and not supported anymore by Apache Camel.
>
> The reason why I ask to upgrade you application is to check whether this
> issue still exists with the actual version or not.
>
> Camel-mail 2.11.0 use the following dependencies (I skipped the test
> dependencies):
> [INFO] org.apache.camel:camel-mail:bundle:2.11.0
> [INFO] +- org.apache.camel:camel-core:jar:2.11.0:compile
> [INFO] | \- org.slf4j:slf4j-api:jar:1.6.6:compile (version managed from
> 1.6.6)
> [INFO] +- javax.mail:mail:jar:1.4.5:compile
>
> Can you re-run the failing tests in DEBUG mode (enable DEBUG log level for
> Camel) and share the log with us?
> At the moment I'm wondering why we have fetchSize and maxMessagesPerPoll
> and what the difference should be...
>
> Best,
>
> Christian Müller
> -----------------
>
> Software Integration Specialist
>
> Apache Camel committer: https://camel.apache.org/team
> V.P. Apache Camel: https://www.apache.org/foundation/
> Apache Member: https://www.apache.org/foundation/members.html
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734253.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Müller <ch...@gmail.com>.
As I already mentioned, Camel 2.6.0 is really old (released in Jan. 2011)
and not supported anymore by Apache Camel.
The reason why I ask to upgrade you application is to check whether this
issue still exists with the actual version or not.
Camel-mail 2.11.0 use the following dependencies (I skipped the test
dependencies):
[INFO] org.apache.camel:camel-mail:bundle:2.11.0
[INFO] +- org.apache.camel:camel-core:jar:2.11.0:compile
[INFO] | \- org.slf4j:slf4j-api:jar:1.6.6:compile (version managed from
1.6.6)
[INFO] +- javax.mail:mail:jar:1.4.5:compile
Can you re-run the failing tests in DEBUG mode (enable DEBUG log level for
Camel) and share the log with us?
At the moment I'm wondering why we have fetchSize and maxMessagesPerPoll
and what the difference should be...
Best,
Christian Müller
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Fri, Jun 7, 2013 at 10:42 AM, cgiera <ch...@mic-cust.com>wrote:
> Hello,
>
> thx for the reply.
> But even if the consumer is not transaction secure, it should not happen
> that mails are marked as seen but don't get processed. I haven't found why
> this happens exactly, but I have described where it happens in my previous
> posts. I'm wondering that no one had this behavior before. Maybe you can
> give me a hint how to fix this problem so that I can probaply apply a patch
> for the community.
> I'm also happy with other suggestions like your environment causes the
> problem(but I don't think it is an environment problem :) ).
>
> kind regards,
> Christoph
>
>
> Claus Ibsen-2 wrote
> > Hi
> >
> > No the mail consumer is not TX per see. When it marks a mail as SEEN
> > it cannot do this in an atomic transaction.
> > Also there is a bit difference when using pop3 or imap.
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734001.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
thx for the reply.
But even if the consumer is not transaction secure, it should not happen
that mails are marked as seen but don't get processed. I haven't found why
this happens exactly, but I have described where it happens in my previous
posts. I'm wondering that no one had this behavior before. Maybe you can
give me a hint how to fix this problem so that I can probaply apply a patch
for the community.
I'm also happy with other suggestions like your environment causes the
problem(but I don't think it is an environment problem :) ).
kind regards,
Christoph
Claus Ibsen-2 wrote
> Hi
>
> No the mail consumer is not TX per see. When it marks a mail as SEEN
> it cannot do this in an atomic transaction.
> Also there is a bit difference when using pop3 or imap.
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734001.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
No the mail consumer is not TX per see. When it marks a mail as SEEN
it cannot do this in an atomic transaction.
Also there is a bit difference when using pop3 or imap.
On Thu, Jun 6, 2013 at 9:28 AM, cgiera <ch...@mic-cust.com> wrote:
> A general question:
> Is the camel mail consumer transaction secure?
>
> kind regards,
> Christoph
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733918.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
A general question:
Is the camel mail consumer transaction secure?
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733918.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Seems like my workaround is not working 100% correctly.
We added the fetchSize parameter to the necessary routes on our customer
system and it looked really fine, but when 2 mails arrived within the same
second only one message get processed but two were marked as seen :(.
However on the customer system camel 2.6.0 and java mail 1.4.4 is used, so
perhaps the behavior is better in camel 2.11.0.
kind regards,
Christoph
Christian Mueller wrote
> I will have a look at it if I find some time later this week...
>
> Best,
>
> Christian Müller
> -----------------
>
> Software Integration Specialist
>
> Apache Camel committer: https://camel.apache.org/team
> V.P. Apache Camel: https://www.apache.org/foundation/
> Apache Member: https://www.apache.org/foundation/members.html
>
> https://www.linkedin.com/pub/christian-mueller/11/551/642
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733539.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Müller <ch...@gmail.com>.
I will have a look at it if I find some time later this week...
Best,
Christian Müller
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Tue, May 28, 2013 at 9:54 AM, cgiera <ch...@mic-cust.com>wrote:
> Ok, i see fetchSize is also a parameter.
>
> The only way to don't lose mails in my testcase is to set fetchSize to 1.
> But that's not a solution, it's only a workaround.
>
> I'm sorry I couldn't find the problem by myself, I could only say it has
> something to do with the folder.isOpen check. After this all fetched mails
> are marked as seen, but only 1 message is processed.
>
>
> cgiera wrote
> > Even if I set the maxMessagesPerPoll option to 1, the fetchsize in this
> > case is -1.
> > So all messages will be looped in this case.
> > In debug mode I have changed the value of the fetchSize to 1 and only one
> > message gets marked as SEEN.
> >
> > I'm not sure if this is solving the whole problem.
> >
> > kind regards,
> > Christoph
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733325.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Ok, i see fetchSize is also a parameter.
The only way to don't lose mails in my testcase is to set fetchSize to 1.
But that's not a solution, it's only a workaround.
I'm sorry I couldn't find the problem by myself, I could only say it has
something to do with the folder.isOpen check. After this all fetched mails
are marked as seen, but only 1 message is processed.
cgiera wrote
> Even if I set the maxMessagesPerPoll option to 1, the fetchsize in this
> case is -1.
> So all messages will be looped in this case.
> In debug mode I have changed the value of the fetchSize to 1 and only one
> message gets marked as SEEN.
>
> I'm not sure if this is solving the whole problem.
>
> kind regards,
> Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733325.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
I'm not sure if I'm on the right way, but I've found one thing which doesn't
work correct I think:
Even if I set the maxMessagesPerPoll option to 1, the fetchsize in this case
is -1.
So all messages will be looped in this case.
In debug mode I have changed the value of the fetchSize to 1 and only one
message gets marked as SEEN.
I'm not sure if this is solving the hole problem.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733269.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
I've found the code which forces all fetched messages to get marked as SEEN
even I have set the maxMessagesPerPoll option to 1.
<http://camel.465427.n5.nabble.com/file/n5733267/screenshot.png>
I just can't understand what is going wrong here, after flushBuffer of this
OutputStream all messages are marked as seen.
kind regards,
Christoph
cgiera wrote
> Ok, this was a big mistake, when i set the maxMessagesPerPoll option to 1
> and the pollinginterval to 1 sec only every ~ 5th message will be
> processed!!
> The second test route sends every second 4 messages to the inbox.
>
> I spend a lot of time with debugging now but I wasn't able to find the
> code segment where this goes wrong.
> cgiera wrote
>> The missing mails can be reduces when setting maxMessagesPerPoll to a
>> small number, but it doesn't solve the problem.
>>
>> kind regards,
>> Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733267.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Ok, this was a big mistake, when i set the maxMessagesPerPoll option to 1 and
the pollinginterval to 1 sec only every ~ 5th message will be processed!!
The second test route sends every second 4 messages to the inbox.
I spend a lot of time with debugging now but I wasn't able to find the code
segment where this goes wrong.
cgiera wrote
> The missing mails can be reduces when setting maxMessagesPerPoll to a
> small number, but it doesn't solve the problem.
>
> kind regards,
> Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733264.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
I have upgraded to camel 2.11.0(not all referenced libs but camel mail
component doesn't use any referenced libs) and mail 1.5.
The problem still exists in the new versions.
I will debug the MailConsumer now and I try to find out what's going on.
I believe there is something wrong with this code:
My test routes look like the following:
The missing mails can be reduces when setting maxMessagesPerPoll to a small
number, but it doesn't solve the problem.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733247.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
We are using java mail 1.4.4. I think this is pretty new although ~ 1 month
ago java mail 1.5 was released.
Short update from our testing/analyzing:
It seems the problem occures when mails are arriving on the mailbox while
MailConsumer is working.
I have created a test scenario where I loaded ~320 mails into the mailbox.
The subject of the mails where the numbers 1-320.
Then i created a route where I just take the mails from the mailbox and move
them to a file-endpoint. As filename I took the subject, so it's easy to see
which mails are missing.
In this scenario I started the route after all mails where sent to the
mailbox. Looking at the file-endpoint, there were all numbers from 1-320.
My second test scenario is extended with an extra route.
A route which just sends 10 messages every 5 seconds to this mailbox. The
subject is still numbered.
Looking now at the file-endpoint there were some numbers missing every now
and then, but all mails in the mailbox were marked as seen.
So I can reproduce the problem of our customer.
I will try to update to a new Camel version and run this scenario again,
maybe you can get me a hint how to get all the referenced libs that have
changed since 2.6.0. We don't use maven so it is really hard to upgrade the
referenced libaries for all camel components.
kind regards,
Christoph
Claus Ibsen-2 wrote
> You can also try upgrading the java mail JAR
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733109.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Claus Ibsen <cl...@gmail.com>.
You can also try upgrading the java mail JAR
On Fri, May 24, 2013 at 8:46 AM, cgiera <ch...@mic-cust.com> wrote:
> Hello,
>
> hm, it is not a really big step from 14 messages to 10 ^^.
> Or does this limiting of the messages changes the behavior of the
> MailConsumer significantly?
>
> Thx for your reply.
>
> kind regards,
> Christoph
>
>
> Claus Ibsen-2 wrote
>> Hi
>>
>> You can try using maxMessagesPerPoll=10 or something to limit polling
>> in too many mails.
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733065.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
hm, it is not a really big step from 14 messages to 10 ^^.
Or does this limiting of the messages changes the behavior of the
MailConsumer significantly?
Thx for your reply.
kind regards,
Christoph
Claus Ibsen-2 wrote
> Hi
>
> You can try using maxMessagesPerPoll=10 or something to limit polling
> in too many mails.
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733065.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
You can try using maxMessagesPerPoll=10 or something to limit polling
in too many mails.
On Thu, May 23, 2013 at 11:01 AM, cgiera <ch...@mic-cust.com> wrote:
> Hello,
>
> I'm facing strange problems when consuming mails from a cyrus mail server.
> Some mails were not processed but appear as SEEN in the mailbox.
> I've added the mail debug option to the route and I was pretty confused when
> analyzing the log:
>
> First poll is quite normal: 14 new mails, 14 get fetched and processed.
>
>
>
>
> Second poll is very confusing to me:
> 14 new messages and only 11 get fetched.
> Missing emails are 266, 267 and 268, but they still exist in the
> mailbox(marked as SEEN).
>
> Our camel application was (and still is) the only client which was connected
> to this mailbox.
> So I've to assume that the camel MailComponent has marked as seen(but not
> processed the mail).
>
>
>
> We are using camel 2.6.0 and cyrus-imapd-2.3.16-6.el6_2.5.x86_64.
>
> The route looks like the following:
>
>
> This behaviour is almost everytime reproducable when loading "a lot" of
> messages to the mailbox(~25+).
> Mailserver log shows that our camel application is the only connected
> client.
>
> Any help/advice/hint would be nice, I don't have a clue what I can do or
> where I should start my next analyzes.
>
> kind regards,
> Christoph
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
due customer restrictions to java 5 we couldn't upgrade until now.
This restriction will now fall, so we can upgrade now.
However it is a big step from 2.6.0 to 2.11.0 and I already started the
upgrade process, but I'm facing big problems with the referenced lib's as
they are not integrated in the normal download.
Do you know what is the simplest way to get the referenced libs when we
don't use maven?
Thx for your reply.
kind regards,
Christoph
Christian Mueller wrote
> Camel 2.6.0 is a bit old and not supported anymore. Do you have a chance
> to
> try a newer version (e.g. 2.11.0) and check whether it behaves in the same
> way!?
>
> Best,
> Christian
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733064.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Müller <ch...@gmail.com>.
Camel 2.6.0 is a bit old and not supported anymore. Do you have a chance to
try a newer version (e.g. 2.11.0) and check whether it behaves in the same
way!?
Best,
Christian
Sent from a mobile device
Am 23.05.2013 11:01 schrieb "cgiera" <ch...@mic-cust.com>:
> Hello,
>
> I'm facing strange problems when consuming mails from a cyrus mail server.
> Some mails were not processed but appear as SEEN in the mailbox.
> I've added the mail debug option to the route and I was pretty confused
> when
> analyzing the log:
>
> First poll is quite normal: 14 new mails, 14 get fetched and processed.
>
>
>
>
> Second poll is very confusing to me:
> 14 new messages and only 11 get fetched.
> Missing emails are 266, 267 and 268, but they still exist in the
> mailbox(marked as SEEN).
>
> Our camel application was (and still is) the only client which was
> connected
> to this mailbox.
> So I've to assume that the camel MailComponent has marked as seen(but not
> processed the mail).
>
>
>
> We are using camel 2.6.0 and cyrus-imapd-2.3.16-6.el6_2.5.x86_64.
>
> The route looks like the following:
>
>
> This behaviour is almost everytime reproducable when loading "a lot" of
> messages to the mailbox(~25+).
> Mailserver log shows that our camel application is the only connected
> client.
>
> Any help/advice/hint would be nice, I don't have a clue what I can do or
> where I should start my next analyzes.
>
> kind regards,
> Christoph
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>