You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by cgiera <ch...@mic-cust.com> on 2013/06/06 09:28:38 UTC
Re: MailComponent/MailConsumer misses Mails
A general question:
Is the camel mail consumer transaction secure?
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733918.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Christian Mueller wrote
> Can you send me the entire log file. I need more context than only two
> lines...
Above of the 2 lines you can find a big part of the log file, I thought this
will be enough, but sure I can
send you the 30 MB zip files :).
Christian Mueller wrote
> Do you have multiple routes polling the same INBOX?
No.
Christian Mueller wrote
> Did you stopped/started the routes (may we have an issue not correctly
> stopping the rouet)?
I have stopped the MailProducer routes when testing one time, not the
consumer route.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734383.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Müller <ch...@gmail.com>.
Can you send me the entire log file. I need more context than only two
lines...
Do you have multiple routes polling the same INBOX?
Did you stopped/started the routes (may we have an issue not correctly
stopping the rouet)?
Best,
Christian Müller
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Wed, Jun 19, 2013 at 12:58 PM, cgiera <ch...@mic-cust.com>wrote:
> Here is another part of the logfile where this problem occures:
>
> Here is something really strange:
> INFO | jvm 1 | 2013/06/19 11:30:37 | 2013-06-19 11:30:37,237 [Camel
> (camel) thread #2 - imap://<ipaddress>] DEBUG - Polling mailbox folder:
> imap://<ipaddress>:143, folder=INBOX
> INFO | jvm 1 | 2013/06/19 11:30:37 | 2013-06-19 11:30:37,300 [Camel
> (camel) thread #8 - imap://<ipaddress>] DEBUG - Polling mailbox folder:
> imap://<ipaddress>:143, folder=INBOX
>
> 2 Threads which are polling at the same time.
>
> kind regards,
> Christoph
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734367.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Here is another part of the logfile where this problem occures:
Here is something really strange:
INFO | jvm 1 | 2013/06/19 11:30:37 | 2013-06-19 11:30:37,237 [Camel
(camel) thread #2 - imap://<ipaddress>] DEBUG - Polling mailbox folder:
imap://<ipaddress>:143, folder=INBOX
INFO | jvm 1 | 2013/06/19 11:30:37 | 2013-06-19 11:30:37,300 [Camel
(camel) thread #8 - imap://<ipaddress>] DEBUG - Polling mailbox folder:
imap://<ipaddress>:143, folder=INBOX
2 Threads which are polling at the same time.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734367.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
now with java mail 1.4.5 and log entries(have added slf4j_log4j.jar to
ensure debug messages occure):
It's the same behavior and the same log entries,
missing mail logs:
INFO | jvm 1 | 2013/06/19 11:54:23 | * 2 RECENT
INFO | jvm 1 | 2013/06/19 11:54:23 | * SEARCH
INFO | jvm 1 | 2013/06/19 11:54:23 | A9329 OK Completed (0 msgs in
0.000 secs)
For comparing, here it works:
INFO | jvm 1 | 2013/06/19 11:54:44 | * 2 RECENT
INFO | jvm 1 | 2013/06/19 11:54:44 | * SEARCH 1550 1551
INFO | jvm 1 | 2013/06/19 11:54:44 | A9357 OK Completed (2 msgs in
0.000 secs)
INFO | jvm 1 | 2013/06/19 11:54:44 | 2013-06-19 11:54:44,944 [Camel
(camel) thread #2 - imap://<ipaddress>] DEBUG - Fetching 1 messages. Total
2 messages.
Although it is only fetching 1 of 2 messages here the second message was
processed with the next run.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734364.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Müller <ch...@gmail.com>.
Using
log4j.logger.org.apache.camel.component.mail=DEBUG
should be fine.
I would like to see the log entries at the time Camel mark/flag more
messages as seen as messages were processed.
And could you please check whether do you see the same strange Java Mail
log messages (2 RECENT but no message found) if you use mail.jar 1.4.5.
Because this is the version we are using in Camel 2.11.0...
Best,
Christian Müller
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Tue, Jun 18, 2013 at 5:27 PM, cgiera <ch...@mic-cust.com>wrote:
> Hello,
>
> I've updated our customers qa system to camel 2.11.0 and mail 1.5 now.
> We've added some test routes to send mails to the mail box we are reading
> from and had a look at it
> if mails are missing.
> After ~1 hour we saw that sent mails vs. received mails was not equal
> anymore and analyzed the log files.
> I've found the log entries where something is wrong:
>
> There are 2 recent mails, but they were not fetched:
> INFO | jvm 1 | 2013/06/18 16:32:21 | * 2 RECENT
> INFO | jvm 1 | 2013/06/18 16:32:21 | * SEARCH
> INFO | jvm 1 | 2013/06/18 16:32:21 | A2749 OK Completed (0 msgs in
> 0.000 secs)
>
> For comparing here are log entries where mails are consumed correctly:
>
> 2 Recent and 2 fetched
> INFO | jvm 1 | 2013/06/18 16:32:43 | * 2 RECENT
> INFO | jvm 1 | 2013/06/18 16:32:43 | * SEARCH 1069 1070
> INFO | jvm 1 | 2013/06/18 16:32:43 | A2777 OK Completed (2 msgs in
> 0.000 secs)
>
> Here is the setup of our routes:
> Consumerroute:
>
> imap://<ip-adress>?consumer.delay=3000&consumer.useFixedDelay=true&debugMode=true&fetchSize=1&password=...._&unseen=true&username=....
>
> 2 routes which send to the mailbox:
> timer://foo?fixedRate=true&period=27000
> timer://foo2?fixedRate=true&period=29000
>
>
> I've activated mailDebug option on the consumer route and additionally some
> log4j logging:
> log4j.logger.org.apache.camel.component.mail=TRACE
> log4j.logger.org.apache.camel.impl=TRACE
>
> Pls let me know if I you need other logs or how I should adjust log4j to
> get
> more logs out of the MailComponent.
>
> kind regards,
> Christoph
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734311.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Müller <ch...@gmail.com>.
Hello Christoph!
Thanks for sharing this with us. I hope upgrading to cyrus 2.4.x will solve
your issue.
Best,
Christian
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Fri, Aug 9, 2013 at 7:56 AM, cgiera <ch...@mic-cust.com> wrote:
> Hello,
>
> it looks like we have found the problem:
> http://www.mail-archive.com/cyrus-devel@lists.andrew.cmu.edu/msg01684.html
>
> We use exactly the cyrus version mentioned in the link, it is not 100% sure
> if this is the real problem, but we are setting up a server with cyrus
> 2.4.x
> and run our tests again.
>
> As a simple workaround the option closeFolder=false looks good, we haven't
> lost a single mail in our testings.
>
> Thanks for your time and help, maybe someone will need this information in
> the future.
>
> kind regards,
> Christoph
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5737024.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
it looks like we have found the problem:
http://www.mail-archive.com/cyrus-devel@lists.andrew.cmu.edu/msg01684.html
We use exactly the cyrus version mentioned in the link, it is not 100% sure
if this is the real problem, but we are setting up a server with cyrus 2.4.x
and run our tests again.
As a simple workaround the option closeFolder=false looks good, we haven't
lost a single mail in our testings.
Thanks for your time and help, maybe someone will need this information in
the future.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5737024.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Mueller <ch...@gmail.com>.
I just updated Camel 2.11.1-SNAPSHOT to javax.mail/mail from 1.4.5 to 1.4.7
without any problems. Could you please try Camel 2.11.0 with javax.mail/mail
1.4.7?
I don't think it's a Camel issue, but we have to investigate a bit more...
Best,
Christian
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5735315.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
I've updated our customers qa system to camel 2.11.0 and mail 1.5 now.
We've added some test routes to send mails to the mail box we are reading
from and had a look at it
if mails are missing.
After ~1 hour we saw that sent mails vs. received mails was not equal
anymore and analyzed the log files.
I've found the log entries where something is wrong:
There are 2 recent mails, but they were not fetched:
INFO | jvm 1 | 2013/06/18 16:32:21 | * 2 RECENT
INFO | jvm 1 | 2013/06/18 16:32:21 | * SEARCH
INFO | jvm 1 | 2013/06/18 16:32:21 | A2749 OK Completed (0 msgs in
0.000 secs)
For comparing here are log entries where mails are consumed correctly:
2 Recent and 2 fetched
INFO | jvm 1 | 2013/06/18 16:32:43 | * 2 RECENT
INFO | jvm 1 | 2013/06/18 16:32:43 | * SEARCH 1069 1070
INFO | jvm 1 | 2013/06/18 16:32:43 | A2777 OK Completed (2 msgs in
0.000 secs)
Here is the setup of our routes:
Consumerroute:
imap://<ip-adress>?consumer.delay=3000&consumer.useFixedDelay=true&debugMode=true&fetchSize=1&password=...._&unseen=true&username=....
2 routes which send to the mailbox:
timer://foo?fixedRate=true&period=27000
timer://foo2?fixedRate=true&period=29000
I've activated mailDebug option on the consumer route and additionally some
log4j logging:
log4j.logger.org.apache.camel.component.mail=TRACE
log4j.logger.org.apache.camel.impl=TRACE
Pls let me know if I you need other logs or how I should adjust log4j to get
more logs out of the MailComponent.
kind regards,
Christoph
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734311.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
thx for the reply.
I've updated to camel 2.11.0 and java mail 1.5 before I've begun the
testing.
So all the log's from the previous posts are already from the newest
version.
On my local system I've tested the workaround with fetchSize=1, here both
versions 2.6.0 and 2.11.0 are working for me.
So we decided to get this workaround to the customer system(2.6.0), but here
the workaround is not working.
I have not found out what is the difference between the customer system and
my testings, but I don't think it is fixed in 2.11.0.
I haven't found something new in this log files, all information is also
included in the logs I've posted in the previous posts.
However, if you still need more log entries, I can post them if you want.
kind regards,
Christoph
Christian Mueller wrote
> As I already mentioned, Camel 2.6.0 is really old (released in Jan. 2011)
> and not supported anymore by Apache Camel.
>
> The reason why I ask to upgrade you application is to check whether this
> issue still exists with the actual version or not.
>
> Camel-mail 2.11.0 use the following dependencies (I skipped the test
> dependencies):
> [INFO] org.apache.camel:camel-mail:bundle:2.11.0
> [INFO] +- org.apache.camel:camel-core:jar:2.11.0:compile
> [INFO] | \- org.slf4j:slf4j-api:jar:1.6.6:compile (version managed from
> 1.6.6)
> [INFO] +- javax.mail:mail:jar:1.4.5:compile
>
> Can you re-run the failing tests in DEBUG mode (enable DEBUG log level for
> Camel) and share the log with us?
> At the moment I'm wondering why we have fetchSize and maxMessagesPerPoll
> and what the difference should be...
>
> Best,
>
> Christian Müller
> -----------------
>
> Software Integration Specialist
>
> Apache Camel committer: https://camel.apache.org/team
> V.P. Apache Camel: https://www.apache.org/foundation/
> Apache Member: https://www.apache.org/foundation/members.html
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734253.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Christian Müller <ch...@gmail.com>.
As I already mentioned, Camel 2.6.0 is really old (released in Jan. 2011)
and not supported anymore by Apache Camel.
The reason why I ask to upgrade you application is to check whether this
issue still exists with the actual version or not.
Camel-mail 2.11.0 use the following dependencies (I skipped the test
dependencies):
[INFO] org.apache.camel:camel-mail:bundle:2.11.0
[INFO] +- org.apache.camel:camel-core:jar:2.11.0:compile
[INFO] | \- org.slf4j:slf4j-api:jar:1.6.6:compile (version managed from
1.6.6)
[INFO] +- javax.mail:mail:jar:1.4.5:compile
Can you re-run the failing tests in DEBUG mode (enable DEBUG log level for
Camel) and share the log with us?
At the moment I'm wondering why we have fetchSize and maxMessagesPerPoll
and what the difference should be...
Best,
Christian Müller
-----------------
Software Integration Specialist
Apache Camel committer: https://camel.apache.org/team
V.P. Apache Camel: https://www.apache.org/foundation/
Apache Member: https://www.apache.org/foundation/members.html
https://www.linkedin.com/pub/christian-mueller/11/551/642
On Fri, Jun 7, 2013 at 10:42 AM, cgiera <ch...@mic-cust.com>wrote:
> Hello,
>
> thx for the reply.
> But even if the consumer is not transaction secure, it should not happen
> that mails are marked as seen but don't get processed. I haven't found why
> this happens exactly, but I have described where it happens in my previous
> posts. I'm wondering that no one had this behavior before. Maybe you can
> give me a hint how to fix this problem so that I can probaply apply a patch
> for the community.
> I'm also happy with other suggestions like your environment causes the
> problem(but I don't think it is an environment problem :) ).
>
> kind regards,
> Christoph
>
>
> Claus Ibsen-2 wrote
> > Hi
> >
> > No the mail consumer is not TX per see. When it marks a mail as SEEN
> > it cannot do this in an atomic transaction.
> > Also there is a bit difference when using pop3 or imap.
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734001.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: MailComponent/MailConsumer misses Mails
Posted by cgiera <ch...@mic-cust.com>.
Hello,
thx for the reply.
But even if the consumer is not transaction secure, it should not happen
that mails are marked as seen but don't get processed. I haven't found why
this happens exactly, but I have described where it happens in my previous
posts. I'm wondering that no one had this behavior before. Maybe you can
give me a hint how to fix this problem so that I can probaply apply a patch
for the community.
I'm also happy with other suggestions like your environment causes the
problem(but I don't think it is an environment problem :) ).
kind regards,
Christoph
Claus Ibsen-2 wrote
> Hi
>
> No the mail consumer is not TX per see. When it marks a mail as SEEN
> it cannot do this in an atomic transaction.
> Also there is a bit difference when using pop3 or imap.
--
View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5734001.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MailComponent/MailConsumer misses Mails
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
No the mail consumer is not TX per see. When it marks a mail as SEEN
it cannot do this in an atomic transaction.
Also there is a bit difference when using pop3 or imap.
On Thu, Jun 6, 2013 at 9:28 AM, cgiera <ch...@mic-cust.com> wrote:
> A general question:
> Is the camel mail consumer transaction secure?
>
> kind regards,
> Christoph
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/MailComponent-MailConsumer-misses-Mails-tp5733010p5733918.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen