You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-user@james.apache.org by Jerry Malcolm <te...@malcolms.com> on 2019/10/22 02:55:56 UTC
Moving Servers Wipes Out iPhone's Mail Accounts
I threw the switch this evening to fully move to my new upgraded 3.4
server. Basically, I just changed the IP address of the domain name for
my mail server. I did a bunch of testing on several accounts using
Thunderbird. Everything seems to be working. But then I get a phone
call from a client that says all of his mail disappeared from his
iPhone. I checked my iPhone and, yep, it's all gone from my iPhone on
all of my different accounts except the new mail that has come in during
the last few hours since moving to the new server. No errors or
messages. Just when I go to a folder it briefly shows the email list,
which then disappears and is replaced with a "No Mail" message.
Fortunately, it's not deleting it from the server. All mail is still
there when I go to Tbird.
Is this some kind of 'security' thing that Apple has created to remove
all mail from the app if the server IP address changes? If so, I'm not
impressed with their thought process. I figure I'm going to start
hearing it from other clients tomorrow. I'm about to delete an account
on my iPhone and recreate it and see if loads all of the mail.
Just wondering if anyone else has encountered this and if there is
anything to tell clients.
Thx
Jerry
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Servers Wipes Out iPhone's Mail Accounts
Posted by Jerry Malcolm <te...@malcolms.com>.
Tellier,
I simply stopped the old JAMES instance, copied the entire database,
started the new JAMES instance, and changed the DNS record for the mail
server IP to point to the new instance. I'm not deep enough into mail
RFCs to know anything about a UID VALIDITY. Would the steps I performed
have changed the UID VALIDITY? It appears the iPhone is doing the
reverse of what you described. It's like it automatically reset the
'start' of all folders to a few hours ago when the ip address changed,
and it removed any mail earlier than that from the folder. I honestly
can't think of a rational reason why that would a reasonable behavior.
And there is absolutely no way to tell the folders to go back to the
'real' start date of the folder.... not even by deleting the entire
email account and recreating it from scratch. That new folder start
timestamp is set in concrete now....
So far, only one client (and me) have seen the problem (it's evening
here, it'll hit the fan tomorrow AM). I'm not aware of any way to
delete the cache on the iPhone (hopefully I have just missed it...
education welcome..). I'm not as concerned about the time it takes to
refresh the entire mailbox as I am about the fact that right now, I
can't figure out a way to even see more than the last few hours mail
from the mailbox on the phone. Again, this ONLY iPhone.... TBird is
showing everything just fine. (Don't know about Android)
Is there anything I can do on the database (i.e. with this UID
VALIDATION stuff) to simply say "all bets are off..." and just force all
clients to just re-download the entire folder contents? I'd rather have
that than lose access to years of email on the phone.
Thx as always.
Jerry
On 10/22/2019 1:03 AM, Tellier Benoit wrote:
> Hi Jerry,
>
> What is the UID VALIDITY associated with a given mailbox name before and
> after your migration?
>
> If the UID VALIDITY is different, then the IMAP client will assume the
> two mailboxes are different and will attempt a whole download once
> again, which is a really lengthy process, and implies dropping all the
> emails downloaded before.
>
> This is due to an historical lack of truely immutable identifiers in
> IMAP, a point that this RFC addresses:
>
> - RFC-8474 IMAP Extension for Object Identifiers
>
> MAILBOXID: a unique identifier for a mailbox, stands for the mailbox
> name + UID validity pair
>
> EMAILID: unique identifier for an email
>
> If I'm right a good way to mitigate your production issue is to enforce
> the UID validity is the same before & after the migration, and hope your
> client apps did not all drop their cache.
>
> Cheers,
>
> Benoit
>
> On 22/10/2019 05:25, Jerry Malcolm wrote:
>> Deleting the iPhone email account and recreating does nothing. It says
>> 'all account data will be deleted.' Nope. After recreating the
>> account, it loads only the two emails that came in since switching IP
>> addresses and ignores the 1000+ other emails in the mailbox.
>>
>> Hopefully somebody has a workaround for this.
>>
>>
>> On 10/21/2019 9:55 PM, Jerry Malcolm wrote:
>>> I threw the switch this evening to fully move to my new upgraded 3.4
>>> server. Basically, I just changed the IP address of the domain name
>>> for my mail server. I did a bunch of testing on several accounts
>>> using Thunderbird. Everything seems to be working. But then I get a
>>> phone call from a client that says all of his mail disappeared from
>>> his iPhone. I checked my iPhone and, yep, it's all gone from my
>>> iPhone on all of my different accounts except the new mail that has
>>> come in during the last few hours since moving to the new server. No
>>> errors or messages. Just when I go to a folder it briefly shows the
>>> email list, which then disappears and is replaced with a "No Mail"
>>> message. Fortunately, it's not deleting it from the server. All mail
>>> is still there when I go to Tbird.
>>>
>>> Is this some kind of 'security' thing that Apple has created to remove
>>> all mail from the app if the server IP address changes? If so, I'm
>>> not impressed with their thought process. I figure I'm going to start
>>> hearing it from other clients tomorrow. I'm about to delete an
>>> account on my iPhone and recreate it and see if loads all of the mail.
>>>
>>> Just wondering if anyone else has encountered this and if there is
>>> anything to tell clients.
>>>
>>> Thx
>>>
>>> Jerry
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Servers Wipes Out iPhone's Mail Accounts
Posted by Tellier Benoit <bt...@apache.org>.
Hi Jerry,
What is the UID VALIDITY associated with a given mailbox name before and
after your migration?
If the UID VALIDITY is different, then the IMAP client will assume the
two mailboxes are different and will attempt a whole download once
again, which is a really lengthy process, and implies dropping all the
emails downloaded before.
This is due to an historical lack of truely immutable identifiers in
IMAP, a point that this RFC addresses:
- RFC-8474 IMAP Extension for Object Identifiers
MAILBOXID: a unique identifier for a mailbox, stands for the mailbox
name + UID validity pair
EMAILID: unique identifier for an email
If I'm right a good way to mitigate your production issue is to enforce
the UID validity is the same before & after the migration, and hope your
client apps did not all drop their cache.
Cheers,
Benoit
On 22/10/2019 05:25, Jerry Malcolm wrote:
> Deleting the iPhone email account and recreating does nothing. It says
> 'all account data will be deleted.' Nope. After recreating the
> account, it loads only the two emails that came in since switching IP
> addresses and ignores the 1000+ other emails in the mailbox.
>
> Hopefully somebody has a workaround for this.
>
>
> On 10/21/2019 9:55 PM, Jerry Malcolm wrote:
>> I threw the switch this evening to fully move to my new upgraded 3.4
>> server. Basically, I just changed the IP address of the domain name
>> for my mail server. I did a bunch of testing on several accounts
>> using Thunderbird. Everything seems to be working. But then I get a
>> phone call from a client that says all of his mail disappeared from
>> his iPhone. I checked my iPhone and, yep, it's all gone from my
>> iPhone on all of my different accounts except the new mail that has
>> come in during the last few hours since moving to the new server. No
>> errors or messages. Just when I go to a folder it briefly shows the
>> email list, which then disappears and is replaced with a "No Mail"
>> message. Fortunately, it's not deleting it from the server. All mail
>> is still there when I go to Tbird.
>>
>> Is this some kind of 'security' thing that Apple has created to remove
>> all mail from the app if the server IP address changes? If so, I'm
>> not impressed with their thought process. I figure I'm going to start
>> hearing it from other clients tomorrow. I'm about to delete an
>> account on my iPhone and recreate it and see if loads all of the mail.
>>
>> Just wondering if anyone else has encountered this and if there is
>> anything to tell clients.
>>
>> Thx
>>
>> Jerry
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Servers Wipes Out iPhone's Mail Accounts
Posted by Jerry Malcolm <te...@malcolms.com>.
Deleting the iPhone email account and recreating does nothing. It says
'all account data will be deleted.' Nope. After recreating the
account, it loads only the two emails that came in since switching IP
addresses and ignores the 1000+ other emails in the mailbox.
Hopefully somebody has a workaround for this.
On 10/21/2019 9:55 PM, Jerry Malcolm wrote:
> I threw the switch this evening to fully move to my new upgraded 3.4
> server. Basically, I just changed the IP address of the domain name
> for my mail server. I did a bunch of testing on several accounts
> using Thunderbird. Everything seems to be working. But then I get a
> phone call from a client that says all of his mail disappeared from
> his iPhone. I checked my iPhone and, yep, it's all gone from my
> iPhone on all of my different accounts except the new mail that has
> come in during the last few hours since moving to the new server. No
> errors or messages. Just when I go to a folder it briefly shows the
> email list, which then disappears and is replaced with a "No Mail"
> message. Fortunately, it's not deleting it from the server. All mail
> is still there when I go to Tbird.
>
> Is this some kind of 'security' thing that Apple has created to remove
> all mail from the app if the server IP address changes? If so, I'm
> not impressed with their thought process. I figure I'm going to start
> hearing it from other clients tomorrow. I'm about to delete an
> account on my iPhone and recreate it and see if loads all of the mail.
>
> Just wondering if anyone else has encountered this and if there is
> anything to tell clients.
>
> Thx
>
> Jerry
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
(Resolved): Moving Servers Wipes Out iPhone's Mail Accounts
Posted by Jerry Malcolm <te...@malcolms.com>.
Finally, after 2 weeks, iPhones have all of their mail. The problem was
that my Lucene index was corrupted. Or to be more precise, only emails
from the last two weeks since I first started up James on the new
environment were indexed. Even after renaming (effectively removing) the
/var/store/lucene folder, the index would not fully rebuild. Same
problem... but worse now... now it showed only mail that came in over
the last 10 minutes since I cleaned the lucene index folder and lost the
last 2 weeks' mail that actually was indexed. On a whim, I changed
indexer.xml to use lazyIndex instead of luceneIndex... voila.... all of
the folders were fully indexed, and iPhone has all of the mail for each
folder. Couldn't find much documentation on lazyIndex vs. luceneIndex.
But at this point, I think I'll take having mail back on my client's
iPhones over any downside to using lazyIndex. But.... educate me if
there are serious downsides to using lazyIndex.
Thunderbird populates folders using a different imap command which,
apparently, goes straight to the JPA database instead of using the
Lucene index. So that's why the problem didn't appear in TBird.
I'm going to start another thread in the dev group specifically to
discuss the Lucene issue.
Jerry
On 10/31/2019 12:35 PM, Jerry Malcolm wrote:
> (I'm moving this thread to the dev forum since it appears now to not
> be a user problem)....
>
> More data now... Since iPhone doesn't load the correct email list and
> TBird does, I traced the precise same scenario with the same email
> account with TBird as the previous iPhone trace in the post below. As
> I suspected, in order to do an initial full sync on a folder, TBird is
> asking for the list of UIDs in a different way. Note from the two traces:
>
> TBird: UID fetch 1:* (FLAGS)
> Response: 1 FETCH (FLAGS (\\Seen) UID 5801) (...followed by many
> other UIDs including the ones in iPhone search below)
> ---
> iPhone: UID SEARCH RETURN (COUNT) 1:* NOT DELETED
> Response: ESEARCH (TAG "39") UID ALL 7776,7955:7970,7975:8033
>
> The two queries above are for the same email account/folder. Again,
> I'm not an IMAP expert. But it appears that both should return the
> same list of UIDs even though they ask for it differently.
>
> Summary... the TBird query gets all UIDs for the folder starting at
> 5801 as expected. The iPhone query gets a subset of UIDs starting at
> 7776. This continues to support my current theory that the code that
> processes UID SEARCH RETURN (COUNT) 1:*.... is not correct.
>
> If someone knows what class processes these imap queries, please let
> me know. I'm going to dig into the code now. But a few pointers to
> the package/classes would be helpful.
>
> Thunderbird trace:
>
> 127 Request: 19 select "INBOX"
> 129,Response: * FLAGS (\\Answered \\Deleted \\Draft \\Flagged \\Seen
> "MailFlagBit1 "MailFlagBit0 NonJunk "label3 "label1 "Forwarded)
> 130,Response: 19 OK [READ-WRITE] SELECT completed.
> 132 Request: 20 getquotaroot "INBOX"
> 134 Response: * QUOTAROOT "INBOX" #private&jerry@malcolms.com
> 135,Response: 20 OK GETQUOTAROOT completed.
> 137,Request: 21 UID fetch 1:* (FLAGS)
> 139,Response: * 1 FETCH (FLAGS (\\Seen) UID 5801)
> 140,Response: * 2 FETCH (FLAGS (\\Seen) UID 5802)
> 141,Response: * 4 FETCH (FLAGS (\\Seen) UID 5804)
>
> On 10/29/2019 3:12 PM, Jerry Malcolm wrote:
>> Ok, I need an IMAP expert.... Below is a very brief trace of the
>> communications between iPhone mail and JAMES (3.4). I completely
>> deleted an account on my iPhone, then recreated it while in airplane
>> mode to make sure I didn't miss any communications in my trace. I
>> started the trace, exited airplane mode and let the iPhone do an
>> initial sync with the account. The inbox folder in this account has
>> over 1000 emails going back to early 2019.
>>
>> I'm not an expert in IMAP. But it appears that the iPhone mail app
>> requests all of the emails 1:* (see line 812), but JAMES returns a
>> single id plus two ranges (line 813). But the total count JAMES
>> reports is nowhere near the full 1000. Subsequently (line 822),
>> iPhone requests the emails JAMES told it about in line 813. From
>> what I can tell, the problem is in line 813. JAMES did not report all
>> of the actual mail that is in the INBOX, which is exactly what I'm
>> seeing on the phone. Shouldn't JAMES return all 1000+ email ids that
>> exist in INBOX? Or am I reading the IMAP trace incorrectly?
>>
>> Note that the only emails JAMES reports came in AFTER I migrated to
>> the new server and to the James 3.4 from v3b5. It appears that there
>> is something different about pre-existing emails in the mailbox
>> folder that is causing JAMES not to recognize them. But this problem
>> ONLY exists on the iPhone mail app. Thunderbird gets all of the mail
>> for the inbox. So is iPhone must be sending a slightly different
>> request command syntax than other clients do (??) Just speculating....
>>
>> 798 IMAP Request: 37 SELECT INBOX
>> 802 IMAP Response: 37 OK [READ-WRITE] SELECT completed.
>> 806 IMAP Request: 38 UID SEARCH RETURN (COUNT) 1:* NOT DELETED
>> 808 IMAP Response: * ESEARCH (TAG "38") UID COUNT 76
>> 809 IMAP Response: 38 OK SEARCH completed.
>> 812 IMAP Request: 39 UID SEARCH RETURN (ALL) 1:* NOT DELETED
>> 813 IMAP Response: * ESEARCH (TAG "39") UID ALL
>> 7776,7955:7970,7975:8033
>> 814 IMAP Response: 39 OK SEARCH completed.
>> 817 IMAP Request: 40 UID SEARCH RETURN (ALL) 929:* NOT DELETED
>> 818 IMAP Response: * ESEARCH (TAG "40") UID ALL 7955:7970,7975:8033
>> 819 IMAP Response: 40 OK SEARCH completed.
>> 822 IMAP Request: 41 UID FETCH 7955:7970,7975:8033 (INTERNALDATE UID
>> RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (date subject from
>> content-type to cc bcc message-id in-reply-to references list-id
>> x-universally-unique-identifier)])
>>
>> On 10/22/2019 3:07 AM, Matthieu Baechler wrote:
>>> Hi Jerry,
>>>
>>> I don't think trying to guess what happen will help you much. I would
>>> rather try to capture the IMAP session.
>>>
>>> I see two ways to do that:
>>>
>>> 1. you write a IMAP session logger in James (it should be really easy,
>>> I don't know why it's not already there)
>>>
>>> 2. you disable SSL on the phone and do a tcpdump on the server
>>>
>>> Once you'll have the IMAP session capture, it will be easier to
>>> diagnose your problem.
>>>
>>> Cheers,
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
(Resolved): Moving Servers Wipes Out iPhone's Mail Accounts
Posted by Jerry Malcolm <te...@malcolms.com>.
Finally, after 2 weeks, iPhones have all of their mail. The problem was
that my Lucene index was corrupted. Or to be more precise, only emails
from the last two weeks since I first started up James on the new
environment were indexed. Even after renaming (effectively removing) the
/var/store/lucene folder, the index would not fully rebuild. Same
problem... but worse now... now it showed only mail that came in over
the last 10 minutes since I cleaned the lucene index folder and lost the
last 2 weeks' mail that actually was indexed. On a whim, I changed
indexer.xml to use lazyIndex instead of luceneIndex... voila.... all of
the folders were fully indexed, and iPhone has all of the mail for each
folder. Couldn't find much documentation on lazyIndex vs. luceneIndex.
But at this point, I think I'll take having mail back on my client's
iPhones over any downside to using lazyIndex. But.... educate me if
there are serious downsides to using lazyIndex.
Thunderbird populates folders using a different imap command which,
apparently, goes straight to the JPA database instead of using the
Lucene index. So that's why the problem didn't appear in TBird.
I'm going to start another thread in the dev group specifically to
discuss the Lucene issue.
Jerry
On 10/31/2019 12:35 PM, Jerry Malcolm wrote:
> (I'm moving this thread to the dev forum since it appears now to not
> be a user problem)....
>
> More data now... Since iPhone doesn't load the correct email list and
> TBird does, I traced the precise same scenario with the same email
> account with TBird as the previous iPhone trace in the post below. As
> I suspected, in order to do an initial full sync on a folder, TBird is
> asking for the list of UIDs in a different way. Note from the two traces:
>
> TBird: UID fetch 1:* (FLAGS)
> Response: 1 FETCH (FLAGS (\\Seen) UID 5801) (...followed by many
> other UIDs including the ones in iPhone search below)
> ---
> iPhone: UID SEARCH RETURN (COUNT) 1:* NOT DELETED
> Response: ESEARCH (TAG "39") UID ALL 7776,7955:7970,7975:8033
>
> The two queries above are for the same email account/folder. Again,
> I'm not an IMAP expert. But it appears that both should return the
> same list of UIDs even though they ask for it differently.
>
> Summary... the TBird query gets all UIDs for the folder starting at
> 5801 as expected. The iPhone query gets a subset of UIDs starting at
> 7776. This continues to support my current theory that the code that
> processes UID SEARCH RETURN (COUNT) 1:*.... is not correct.
>
> If someone knows what class processes these imap queries, please let
> me know. I'm going to dig into the code now. But a few pointers to
> the package/classes would be helpful.
>
> Thunderbird trace:
>
> 127 Request: 19 select "INBOX"
> 129,Response: * FLAGS (\\Answered \\Deleted \\Draft \\Flagged \\Seen
> "MailFlagBit1 "MailFlagBit0 NonJunk "label3 "label1 "Forwarded)
> 130,Response: 19 OK [READ-WRITE] SELECT completed.
> 132 Request: 20 getquotaroot "INBOX"
> 134 Response: * QUOTAROOT "INBOX" #private&jerry@malcolms.com
> 135,Response: 20 OK GETQUOTAROOT completed.
> 137,Request: 21 UID fetch 1:* (FLAGS)
> 139,Response: * 1 FETCH (FLAGS (\\Seen) UID 5801)
> 140,Response: * 2 FETCH (FLAGS (\\Seen) UID 5802)
> 141,Response: * 4 FETCH (FLAGS (\\Seen) UID 5804)
>
> On 10/29/2019 3:12 PM, Jerry Malcolm wrote:
>> Ok, I need an IMAP expert.... Below is a very brief trace of the
>> communications between iPhone mail and JAMES (3.4). I completely
>> deleted an account on my iPhone, then recreated it while in airplane
>> mode to make sure I didn't miss any communications in my trace. I
>> started the trace, exited airplane mode and let the iPhone do an
>> initial sync with the account. The inbox folder in this account has
>> over 1000 emails going back to early 2019.
>>
>> I'm not an expert in IMAP. But it appears that the iPhone mail app
>> requests all of the emails 1:* (see line 812), but JAMES returns a
>> single id plus two ranges (line 813). But the total count JAMES
>> reports is nowhere near the full 1000. Subsequently (line 822),
>> iPhone requests the emails JAMES told it about in line 813. From
>> what I can tell, the problem is in line 813. JAMES did not report all
>> of the actual mail that is in the INBOX, which is exactly what I'm
>> seeing on the phone. Shouldn't JAMES return all 1000+ email ids that
>> exist in INBOX? Or am I reading the IMAP trace incorrectly?
>>
>> Note that the only emails JAMES reports came in AFTER I migrated to
>> the new server and to the James 3.4 from v3b5. It appears that there
>> is something different about pre-existing emails in the mailbox
>> folder that is causing JAMES not to recognize them. But this problem
>> ONLY exists on the iPhone mail app. Thunderbird gets all of the mail
>> for the inbox. So is iPhone must be sending a slightly different
>> request command syntax than other clients do (??) Just speculating....
>>
>> 798 IMAP Request: 37 SELECT INBOX
>> 802 IMAP Response: 37 OK [READ-WRITE] SELECT completed.
>> 806 IMAP Request: 38 UID SEARCH RETURN (COUNT) 1:* NOT DELETED
>> 808 IMAP Response: * ESEARCH (TAG "38") UID COUNT 76
>> 809 IMAP Response: 38 OK SEARCH completed.
>> 812 IMAP Request: 39 UID SEARCH RETURN (ALL) 1:* NOT DELETED
>> 813 IMAP Response: * ESEARCH (TAG "39") UID ALL
>> 7776,7955:7970,7975:8033
>> 814 IMAP Response: 39 OK SEARCH completed.
>> 817 IMAP Request: 40 UID SEARCH RETURN (ALL) 929:* NOT DELETED
>> 818 IMAP Response: * ESEARCH (TAG "40") UID ALL 7955:7970,7975:8033
>> 819 IMAP Response: 40 OK SEARCH completed.
>> 822 IMAP Request: 41 UID FETCH 7955:7970,7975:8033 (INTERNALDATE UID
>> RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (date subject from
>> content-type to cc bcc message-id in-reply-to references list-id
>> x-universally-unique-identifier)])
>>
>> On 10/22/2019 3:07 AM, Matthieu Baechler wrote:
>>> Hi Jerry,
>>>
>>> I don't think trying to guess what happen will help you much. I would
>>> rather try to capture the IMAP session.
>>>
>>> I see two ways to do that:
>>>
>>> 1. you write a IMAP session logger in James (it should be really easy,
>>> I don't know why it's not already there)
>>>
>>> 2. you disable SSL on the phone and do a tcpdump on the server
>>>
>>> Once you'll have the IMAP session capture, it will be easier to
>>> diagnose your problem.
>>>
>>> Cheers,
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Servers Wipes Out iPhone's Mail Accounts (IMAP Trace
attached)
Posted by Jerry Malcolm <te...@malcolms.com>.
(I'm moving this thread to the dev forum since it appears now to not be
a user problem)....
More data now... Since iPhone doesn't load the correct email list and
TBird does, I traced the precise same scenario with the same email
account with TBird as the previous iPhone trace in the post below. As I
suspected, in order to do an initial full sync on a folder, TBird is
asking for the list of UIDs in a different way. Note from the two traces:
TBird: UID fetch 1:* (FLAGS)
Response: 1 FETCH (FLAGS (\\Seen) UID 5801) (...followed by many
other UIDs including the ones in iPhone search below)
---
iPhone: UID SEARCH RETURN (COUNT) 1:* NOT DELETED
Response: ESEARCH (TAG "39") UID ALL 7776,7955:7970,7975:8033
The two queries above are for the same email account/folder. Again, I'm
not an IMAP expert. But it appears that both should return the same
list of UIDs even though they ask for it differently.
Summary... the TBird query gets all UIDs for the folder starting at 5801
as expected. The iPhone query gets a subset of UIDs starting at 7776.
This continues to support my current theory that the code that processes
UID SEARCH RETURN (COUNT) 1:*.... is not correct.
If someone knows what class processes these imap queries, please let me
know. I'm going to dig into the code now. But a few pointers to the
package/classes would be helpful.
Thunderbird trace:
127 Request: 19 select "INBOX"
129,Response: * FLAGS (\\Answered \\Deleted \\Draft \\Flagged \\Seen
"MailFlagBit1 "MailFlagBit0 NonJunk "label3 "label1 "Forwarded)
130,Response: 19 OK [READ-WRITE] SELECT completed.
132 Request: 20 getquotaroot "INBOX"
134 Response: * QUOTAROOT "INBOX" #private&jerry@malcolms.com
135,Response: 20 OK GETQUOTAROOT completed.
137,Request: 21 UID fetch 1:* (FLAGS)
139,Response: * 1 FETCH (FLAGS (\\Seen) UID 5801)
140,Response: * 2 FETCH (FLAGS (\\Seen) UID 5802)
141,Response: * 4 FETCH (FLAGS (\\Seen) UID 5804)
On 10/29/2019 3:12 PM, Jerry Malcolm wrote:
> Ok, I need an IMAP expert.... Below is a very brief trace of the
> communications between iPhone mail and JAMES (3.4). I completely
> deleted an account on my iPhone, then recreated it while in airplane
> mode to make sure I didn't miss any communications in my trace. I
> started the trace, exited airplane mode and let the iPhone do an
> initial sync with the account. The inbox folder in this account has
> over 1000 emails going back to early 2019.
>
> I'm not an expert in IMAP. But it appears that the iPhone mail app
> requests all of the emails 1:* (see line 812), but JAMES returns a
> single id plus two ranges (line 813). But the total count JAMES
> reports is nowhere near the full 1000. Subsequently (line 822),
> iPhone requests the emails JAMES told it about in line 813. From what
> I can tell, the problem is in line 813. JAMES did not report all of
> the actual mail that is in the INBOX, which is exactly what I'm seeing
> on the phone. Shouldn't JAMES return all 1000+ email ids that exist
> in INBOX? Or am I reading the IMAP trace incorrectly?
>
> Note that the only emails JAMES reports came in AFTER I migrated to
> the new server and to the James 3.4 from v3b5. It appears that there
> is something different about pre-existing emails in the mailbox folder
> that is causing JAMES not to recognize them. But this problem ONLY
> exists on the iPhone mail app. Thunderbird gets all of the mail for
> the inbox. So is iPhone must be sending a slightly different request
> command syntax than other clients do (??) Just speculating....
>
> 798 IMAP Request: 37 SELECT INBOX
> 802 IMAP Response: 37 OK [READ-WRITE] SELECT completed.
> 806 IMAP Request: 38 UID SEARCH RETURN (COUNT) 1:* NOT DELETED
> 808 IMAP Response: * ESEARCH (TAG "38") UID COUNT 76
> 809 IMAP Response: 38 OK SEARCH completed.
> 812 IMAP Request: 39 UID SEARCH RETURN (ALL) 1:* NOT DELETED
> 813 IMAP Response: * ESEARCH (TAG "39") UID ALL 7776,7955:7970,7975:8033
> 814 IMAP Response: 39 OK SEARCH completed.
> 817 IMAP Request: 40 UID SEARCH RETURN (ALL) 929:* NOT DELETED
> 818 IMAP Response: * ESEARCH (TAG "40") UID ALL 7955:7970,7975:8033
> 819 IMAP Response: 40 OK SEARCH completed.
> 822 IMAP Request: 41 UID FETCH 7955:7970,7975:8033 (INTERNALDATE UID
> RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (date subject from
> content-type to cc bcc message-id in-reply-to references list-id
> x-universally-unique-identifier)])
>
> On 10/22/2019 3:07 AM, Matthieu Baechler wrote:
>> Hi Jerry,
>>
>> I don't think trying to guess what happen will help you much. I would
>> rather try to capture the IMAP session.
>>
>> I see two ways to do that:
>>
>> 1. you write a IMAP session logger in James (it should be really easy,
>> I don't know why it's not already there)
>>
>> 2. you disable SSL on the phone and do a tcpdump on the server
>>
>> Once you'll have the IMAP session capture, it will be easier to
>> diagnose your problem.
>>
>> Cheers,
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Moving Servers Wipes Out iPhone's Mail Accounts (IMAP Trace
attached)
Posted by Jerry Malcolm <te...@malcolms.com>.
(I'm moving this thread to the dev forum since it appears now to not be
a user problem)....
More data now... Since iPhone doesn't load the correct email list and
TBird does, I traced the precise same scenario with the same email
account with TBird as the previous iPhone trace in the post below. As I
suspected, in order to do an initial full sync on a folder, TBird is
asking for the list of UIDs in a different way. Note from the two traces:
TBird: UID fetch 1:* (FLAGS)
Response: 1 FETCH (FLAGS (\\Seen) UID 5801) (...followed by many
other UIDs including the ones in iPhone search below)
---
iPhone: UID SEARCH RETURN (COUNT) 1:* NOT DELETED
Response: ESEARCH (TAG "39") UID ALL 7776,7955:7970,7975:8033
The two queries above are for the same email account/folder. Again, I'm
not an IMAP expert. But it appears that both should return the same
list of UIDs even though they ask for it differently.
Summary... the TBird query gets all UIDs for the folder starting at 5801
as expected. The iPhone query gets a subset of UIDs starting at 7776.
This continues to support my current theory that the code that processes
UID SEARCH RETURN (COUNT) 1:*.... is not correct.
If someone knows what class processes these imap queries, please let me
know. I'm going to dig into the code now. But a few pointers to the
package/classes would be helpful.
Thunderbird trace:
127 Request: 19 select "INBOX"
129,Response: * FLAGS (\\Answered \\Deleted \\Draft \\Flagged \\Seen
"MailFlagBit1 "MailFlagBit0 NonJunk "label3 "label1 "Forwarded)
130,Response: 19 OK [READ-WRITE] SELECT completed.
132 Request: 20 getquotaroot "INBOX"
134 Response: * QUOTAROOT "INBOX" #private&jerry@malcolms.com
135,Response: 20 OK GETQUOTAROOT completed.
137,Request: 21 UID fetch 1:* (FLAGS)
139,Response: * 1 FETCH (FLAGS (\\Seen) UID 5801)
140,Response: * 2 FETCH (FLAGS (\\Seen) UID 5802)
141,Response: * 4 FETCH (FLAGS (\\Seen) UID 5804)
On 10/29/2019 3:12 PM, Jerry Malcolm wrote:
> Ok, I need an IMAP expert.... Below is a very brief trace of the
> communications between iPhone mail and JAMES (3.4). I completely
> deleted an account on my iPhone, then recreated it while in airplane
> mode to make sure I didn't miss any communications in my trace. I
> started the trace, exited airplane mode and let the iPhone do an
> initial sync with the account. The inbox folder in this account has
> over 1000 emails going back to early 2019.
>
> I'm not an expert in IMAP. But it appears that the iPhone mail app
> requests all of the emails 1:* (see line 812), but JAMES returns a
> single id plus two ranges (line 813). But the total count JAMES
> reports is nowhere near the full 1000. Subsequently (line 822),
> iPhone requests the emails JAMES told it about in line 813. From what
> I can tell, the problem is in line 813. JAMES did not report all of
> the actual mail that is in the INBOX, which is exactly what I'm seeing
> on the phone. Shouldn't JAMES return all 1000+ email ids that exist
> in INBOX? Or am I reading the IMAP trace incorrectly?
>
> Note that the only emails JAMES reports came in AFTER I migrated to
> the new server and to the James 3.4 from v3b5. It appears that there
> is something different about pre-existing emails in the mailbox folder
> that is causing JAMES not to recognize them. But this problem ONLY
> exists on the iPhone mail app. Thunderbird gets all of the mail for
> the inbox. So is iPhone must be sending a slightly different request
> command syntax than other clients do (??) Just speculating....
>
> 798 IMAP Request: 37 SELECT INBOX
> 802 IMAP Response: 37 OK [READ-WRITE] SELECT completed.
> 806 IMAP Request: 38 UID SEARCH RETURN (COUNT) 1:* NOT DELETED
> 808 IMAP Response: * ESEARCH (TAG "38") UID COUNT 76
> 809 IMAP Response: 38 OK SEARCH completed.
> 812 IMAP Request: 39 UID SEARCH RETURN (ALL) 1:* NOT DELETED
> 813 IMAP Response: * ESEARCH (TAG "39") UID ALL 7776,7955:7970,7975:8033
> 814 IMAP Response: 39 OK SEARCH completed.
> 817 IMAP Request: 40 UID SEARCH RETURN (ALL) 929:* NOT DELETED
> 818 IMAP Response: * ESEARCH (TAG "40") UID ALL 7955:7970,7975:8033
> 819 IMAP Response: 40 OK SEARCH completed.
> 822 IMAP Request: 41 UID FETCH 7955:7970,7975:8033 (INTERNALDATE UID
> RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (date subject from
> content-type to cc bcc message-id in-reply-to references list-id
> x-universally-unique-identifier)])
>
> On 10/22/2019 3:07 AM, Matthieu Baechler wrote:
>> Hi Jerry,
>>
>> I don't think trying to guess what happen will help you much. I would
>> rather try to capture the IMAP session.
>>
>> I see two ways to do that:
>>
>> 1. you write a IMAP session logger in James (it should be really easy,
>> I don't know why it's not already there)
>>
>> 2. you disable SSL on the phone and do a tcpdump on the server
>>
>> Once you'll have the IMAP session capture, it will be easier to
>> diagnose your problem.
>>
>> Cheers,
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: iPhone/Android mail apps with JAMES
Posted by Jerry Malcolm <te...@malcolms.com>.
Benoit,
Thank you so much for the info. I'm going to start pulling together
some wireshark captures and logs and hopefully we can figure out what it
going wrong.
I really appreciate your help.
Jerry
On 4/26/2022 9:40 PM, btellier@linagora.com wrote:
> Hello Jerry,
>
> I confirm that different email applications uses the IMAP protocol in
> different ways.
>
> I confirm that some clients uses the IMAP SEARCH results upon
> resynchronisation, and thus that inacurate results could result in bad
> synchronisation.
>
> That being said, having traffic capture for bith apps would be
> valuable to diagnose what is going on. Things like wireshark of James
> debug logs (that include IMAP command).
>
> FYI I succeeded to reinde email on the JPA Guice based distribution on
> the 3.7.0 demo image.
>
> Regards,
>
> Benoit
>
> On 4/26/22 11:41, Jerry Malcolm wrote:
>> This is a critical problem. I really need some direction on this.
>> Please!!!....
>>
>> I started this thread about two and half years ago when I moved my
>> James installation to AWS EC2. All of my clients lost all of their
>> mail on their mobile devices, even though all of the mail still shows
>> up fine on desktop Thunderbird. I tried all of the suggestions, but
>> never could get the older emails restored. Since then I have added
>> another very large customer account on a completely different james
>> installation, and I've upgraded to pretty much the latest GitHub
>> version of James on my original installation. For the new client,
>> there was no migration. It was completely start from scratch on a
>> new domain. Yet across the board, NONE of my clients on either
>> system can reliably get their mail on iPhone or Android. On my own
>> iPhone, I have replaced the native iPhone mail app with Edison mail
>> app and later with Outlook mail app. I get one or two emails
>> downloaded periodically on each app on my various email accounts, and
>> some email accounts just start saying there's no mail in the last
>> month or two (and there's actually typically 10-20 emails each day on
>> those accounts). Today, a client configured their mail account on an
>> iPhone. It immediately said 21 unread emails, and the inbox then
>> downloaded 3 emails. Open TBird on the same account. There's
>> there's the 21 emails.
>>
>> I've tried to re-index the Lucene cache. But I'm still getting the
>> same error that I got in Oct 2019 about wrong parameters or something
>> when I try to do that. So I completely erased the Lucene cache
>> folder. No change. I'm running Spring. I tried to move to Guice a
>> month or so ago, but gave up when I couldn't fix the errors I was
>> getting , so I moved back to Spring. Is this whole iPhone problem
>> due to me using the Spring build? I have no problem trying again to
>> get Guice up and running. But I don't want to waste a week trying to
>> get Guice up and find out the same problem exists in the Guice build.
>>
>> It's obvious that for some reason all of the mobile email apps
>> (native, Edison, Outlook) ask for email differently than
>> Thunderbird. But I'm at a loss to explain why JAMES refuses to send
>> the mail that is there when these same email client apps have no
>> problem getting mail from other mail servers.
>>
>> It's hard for me to understand how every other JAMES user in the
>> world is working totally successfully with mobile phones when I have
>> two completely independent JAMES environments with a huge number of
>> clients on each and NOT ONE of them can get more than 5% of their
>> real mail on their phone.
>>
>> If somebody can just educate me just a little on the differences
>> between how JAMES responds to IMAP queries on mobile devices vs. IMAP
>> queries from Thunderbird, and point me to the handling code, I'll
>> start seeing what I can do to resolve this. Or better yet, is there
>> someway to disable all of the Lucene or whatever caching completely
>> and just make JAMES think it's talking to Thunderbird instead of an
>> iPhone?
>>
>> I'm to the point that my major client is refusing to use the JAMES
>> environment for their company mail accounts since their phone email
>> apps are not receiving most of their critical corporate emails. To
>> say the least, they are NOT happy. Somebody PLEASE respond before my
>> customer fires me.
>>
>> Please HELP!!!! Give me SOMETHING I can work with!!!! I just want to
>> get a conversation going.
>>
>> Thanks
>>
>> Jerry
>>
>>
>> On 11/8/2019 8:03 AM, Matthieu Baechler wrote:
>>> Hi Jerry,
>>>
>>> On Tue, 2019-10-29 at 15:12 -0500, Jerry Malcolm wrote:
>>>> Ok, I need an IMAP expert.... Below is a very brief trace of the
>>>> communications between iPhone mail and JAMES (3.4). I completely
>>>> deleted an account on my iPhone, then recreated it while in airplane
>>>> mode to make sure I didn't miss any communications in my trace. I
>>>> started the trace, exited airplane mode and let the iPhone do an
>>>> initial
>>>> sync with the account. The inbox folder in this account has over
>>>> 1000
>>>> emails going back to early 2019.
>>>>
>>>> I'm not an expert in IMAP. But it appears that the iPhone mail app
>>>> requests all of the emails 1:* (see line 812), but JAMES returns a
>>>> single id plus two ranges (line 813). But the total count JAMES
>>>> reports
>>>> is nowhere near the full 1000. Subsequently (line 822), iPhone
>>>> requests
>>>> the emails JAMES told it about in line 813. From what I can tell,
>>>> the
>>>> problem is in line 813. JAMES did not report all of the actual mail
>>>> that is in the INBOX, which is exactly what I'm seeing on the
>>>> phone.
>>>> Shouldn't JAMES return all 1000+ email ids that exist in INBOX? Or
>>>> am I
>>>> reading the IMAP trace incorrectly?
>>> You are right, if there's 1000+ messages, it should be returned by
>>> these calls.
>>>
>>>> Note that the only emails JAMES reports came in AFTER I migrated to
>>>> the
>>>> new server and to the James 3.4 from v3b5. It appears that there is
>>>> something different about pre-existing emails in the mailbox folder
>>>> that
>>>> is causing JAMES not to recognize them. But this problem ONLY exists
>>>> on
>>>> the iPhone mail app. Thunderbird gets all of the mail for the
>>>> inbox.
>>>> So is iPhone must be sending a slightly different request command
>>>> syntax
>>>> than other clients do (??) Just speculating....
>>>>
>>> The best thing to do is to get a trace from Thunderbird for the same
>>> scenario.
>>>
>>> My best guess is that one client trigger a search whereas the other is
>>> just triggering a full-scan.
>>>
>>> Are you using JPA? With with indexing backend?
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: iPhone/Android mail apps with JAMES
Posted by "btellier@linagora.com" <bt...@linagora.com>.
Hello Jerry,
I confirm that different email applications uses the IMAP protocol in
different ways.
I confirm that some clients uses the IMAP SEARCH results upon
resynchronisation, and thus that inacurate results could result in bad
synchronisation.
That being said, having traffic capture for bith apps would be valuable
to diagnose what is going on. Things like wireshark of James debug logs
(that include IMAP command).
FYI I succeeded to reinde email on the JPA Guice based distribution on
the 3.7.0 demo image.
Regards,
Benoit
On 4/26/22 11:41, Jerry Malcolm wrote:
> This is a critical problem. I really need some direction on this.
> Please!!!....
>
> I started this thread about two and half years ago when I moved my
> James installation to AWS EC2. All of my clients lost all of their
> mail on their mobile devices, even though all of the mail still shows
> up fine on desktop Thunderbird. I tried all of the suggestions, but
> never could get the older emails restored. Since then I have added
> another very large customer account on a completely different james
> installation, and I've upgraded to pretty much the latest GitHub
> version of James on my original installation. For the new client,
> there was no migration. It was completely start from scratch on a new
> domain. Yet across the board, NONE of my clients on either system can
> reliably get their mail on iPhone or Android. On my own iPhone, I
> have replaced the native iPhone mail app with Edison mail app and
> later with Outlook mail app. I get one or two emails downloaded
> periodically on each app on my various email accounts, and some email
> accounts just start saying there's no mail in the last month or two
> (and there's actually typically 10-20 emails each day on those
> accounts). Today, a client configured their mail account on an
> iPhone. It immediately said 21 unread emails, and the inbox then
> downloaded 3 emails. Open TBird on the same account. There's there's
> the 21 emails.
>
> I've tried to re-index the Lucene cache. But I'm still getting the
> same error that I got in Oct 2019 about wrong parameters or something
> when I try to do that. So I completely erased the Lucene cache
> folder. No change. I'm running Spring. I tried to move to Guice a
> month or so ago, but gave up when I couldn't fix the errors I was
> getting , so I moved back to Spring. Is this whole iPhone problem due
> to me using the Spring build? I have no problem trying again to get
> Guice up and running. But I don't want to waste a week trying to get
> Guice up and find out the same problem exists in the Guice build.
>
> It's obvious that for some reason all of the mobile email apps
> (native, Edison, Outlook) ask for email differently than Thunderbird.
> But I'm at a loss to explain why JAMES refuses to send the mail that
> is there when these same email client apps have no problem getting
> mail from other mail servers.
>
> It's hard for me to understand how every other JAMES user in the world
> is working totally successfully with mobile phones when I have two
> completely independent JAMES environments with a huge number of
> clients on each and NOT ONE of them can get more than 5% of their real
> mail on their phone.
>
> If somebody can just educate me just a little on the differences
> between how JAMES responds to IMAP queries on mobile devices vs. IMAP
> queries from Thunderbird, and point me to the handling code, I'll
> start seeing what I can do to resolve this. Or better yet, is there
> someway to disable all of the Lucene or whatever caching completely
> and just make JAMES think it's talking to Thunderbird instead of an
> iPhone?
>
> I'm to the point that my major client is refusing to use the JAMES
> environment for their company mail accounts since their phone email
> apps are not receiving most of their critical corporate emails. To say
> the least, they are NOT happy. Somebody PLEASE respond before my
> customer fires me.
>
> Please HELP!!!! Give me SOMETHING I can work with!!!! I just want to
> get a conversation going.
>
> Thanks
>
> Jerry
>
>
> On 11/8/2019 8:03 AM, Matthieu Baechler wrote:
>> Hi Jerry,
>>
>> On Tue, 2019-10-29 at 15:12 -0500, Jerry Malcolm wrote:
>>> Ok, I need an IMAP expert.... Below is a very brief trace of the
>>> communications between iPhone mail and JAMES (3.4). I completely
>>> deleted an account on my iPhone, then recreated it while in airplane
>>> mode to make sure I didn't miss any communications in my trace. I
>>> started the trace, exited airplane mode and let the iPhone do an
>>> initial
>>> sync with the account. The inbox folder in this account has over
>>> 1000
>>> emails going back to early 2019.
>>>
>>> I'm not an expert in IMAP. But it appears that the iPhone mail app
>>> requests all of the emails 1:* (see line 812), but JAMES returns a
>>> single id plus two ranges (line 813). But the total count JAMES
>>> reports
>>> is nowhere near the full 1000. Subsequently (line 822), iPhone
>>> requests
>>> the emails JAMES told it about in line 813. From what I can tell,
>>> the
>>> problem is in line 813. JAMES did not report all of the actual mail
>>> that is in the INBOX, which is exactly what I'm seeing on the
>>> phone.
>>> Shouldn't JAMES return all 1000+ email ids that exist in INBOX? Or
>>> am I
>>> reading the IMAP trace incorrectly?
>> You are right, if there's 1000+ messages, it should be returned by
>> these calls.
>>
>>> Note that the only emails JAMES reports came in AFTER I migrated to
>>> the
>>> new server and to the James 3.4 from v3b5. It appears that there is
>>> something different about pre-existing emails in the mailbox folder
>>> that
>>> is causing JAMES not to recognize them. But this problem ONLY exists
>>> on
>>> the iPhone mail app. Thunderbird gets all of the mail for the
>>> inbox.
>>> So is iPhone must be sending a slightly different request command
>>> syntax
>>> than other clients do (??) Just speculating....
>>>
>> The best thing to do is to get a trace from Thunderbird for the same
>> scenario.
>>
>> My best guess is that one client trigger a search whereas the other is
>> just triggering a full-scan.
>>
>> Are you using JPA? With with indexing backend?
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
iPhone/Android mail apps with JAMES
Posted by Jerry Malcolm <te...@malcolms.com>.
This is a critical problem. I really need some direction on this.
Please!!!....
I started this thread about two and half years ago when I moved my James
installation to AWS EC2. All of my clients lost all of their mail on
their mobile devices, even though all of the mail still shows up fine on
desktop Thunderbird. I tried all of the suggestions, but never could
get the older emails restored. Since then I have added another very
large customer account on a completely different james installation, and
I've upgraded to pretty much the latest GitHub version of James on my
original installation. For the new client, there was no migration. It
was completely start from scratch on a new domain. Yet across the
board, NONE of my clients on either system can reliably get their mail
on iPhone or Android. On my own iPhone, I have replaced the native
iPhone mail app with Edison mail app and later with Outlook mail app. I
get one or two emails downloaded periodically on each app on my various
email accounts, and some email accounts just start saying there's no
mail in the last month or two (and there's actually typically 10-20
emails each day on those accounts). Today, a client configured their
mail account on an iPhone. It immediately said 21 unread emails, and
the inbox then downloaded 3 emails. Open TBird on the same account.
There's there's the 21 emails.
I've tried to re-index the Lucene cache. But I'm still getting the same
error that I got in Oct 2019 about wrong parameters or something when I
try to do that. So I completely erased the Lucene cache folder. No
change. I'm running Spring. I tried to move to Guice a month or so ago,
but gave up when I couldn't fix the errors I was getting , so I moved
back to Spring. Is this whole iPhone problem due to me using the Spring
build? I have no problem trying again to get Guice up and running. But
I don't want to waste a week trying to get Guice up and find out the
same problem exists in the Guice build.
It's obvious that for some reason all of the mobile email apps (native,
Edison, Outlook) ask for email differently than Thunderbird. But I'm at
a loss to explain why JAMES refuses to send the mail that is there when
these same email client apps have no problem getting mail from other
mail servers.
It's hard for me to understand how every other JAMES user in the world
is working totally successfully with mobile phones when I have two
completely independent JAMES environments with a huge number of clients
on each and NOT ONE of them can get more than 5% of their real mail on
their phone.
If somebody can just educate me just a little on the differences between
how JAMES responds to IMAP queries on mobile devices vs. IMAP queries
from Thunderbird, and point me to the handling code, I'll start seeing
what I can do to resolve this. Or better yet, is there someway to
disable all of the Lucene or whatever caching completely and just make
JAMES think it's talking to Thunderbird instead of an iPhone?
I'm to the point that my major client is refusing to use the JAMES
environment for their company mail accounts since their phone email apps
are not receiving most of their critical corporate emails. To say the
least, they are NOT happy. Somebody PLEASE respond before my customer
fires me.
Please HELP!!!! Give me SOMETHING I can work with!!!! I just want to
get a conversation going.
Thanks
Jerry
On 11/8/2019 8:03 AM, Matthieu Baechler wrote:
> Hi Jerry,
>
> On Tue, 2019-10-29 at 15:12 -0500, Jerry Malcolm wrote:
>> Ok, I need an IMAP expert.... Below is a very brief trace of the
>> communications between iPhone mail and JAMES (3.4). I completely
>> deleted an account on my iPhone, then recreated it while in airplane
>> mode to make sure I didn't miss any communications in my trace. I
>> started the trace, exited airplane mode and let the iPhone do an
>> initial
>> sync with the account. The inbox folder in this account has over
>> 1000
>> emails going back to early 2019.
>>
>> I'm not an expert in IMAP. But it appears that the iPhone mail app
>> requests all of the emails 1:* (see line 812), but JAMES returns a
>> single id plus two ranges (line 813). But the total count JAMES
>> reports
>> is nowhere near the full 1000. Subsequently (line 822), iPhone
>> requests
>> the emails JAMES told it about in line 813. From what I can tell,
>> the
>> problem is in line 813. JAMES did not report all of the actual mail
>> that is in the INBOX, which is exactly what I'm seeing on the
>> phone.
>> Shouldn't JAMES return all 1000+ email ids that exist in INBOX? Or
>> am I
>> reading the IMAP trace incorrectly?
> You are right, if there's 1000+ messages, it should be returned by
> these calls.
>
>> Note that the only emails JAMES reports came in AFTER I migrated to
>> the
>> new server and to the James 3.4 from v3b5. It appears that there is
>> something different about pre-existing emails in the mailbox folder
>> that
>> is causing JAMES not to recognize them. But this problem ONLY exists
>> on
>> the iPhone mail app. Thunderbird gets all of the mail for the
>> inbox.
>> So is iPhone must be sending a slightly different request command
>> syntax
>> than other clients do (??) Just speculating....
>>
> The best thing to do is to get a trace from Thunderbird for the same
> scenario.
>
> My best guess is that one client trigger a search whereas the other is
> just triggering a full-scan.
>
> Are you using JPA? With with indexing backend?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Moving Servers Wipes Out iPhone's Mail Accounts (IMAP Trace
attached)
Posted by Matthieu Baechler <ma...@apache.org>.
Hi Jerry,
On Tue, 2019-10-29 at 15:12 -0500, Jerry Malcolm wrote:
> Ok, I need an IMAP expert.... Below is a very brief trace of the
> communications between iPhone mail and JAMES (3.4). I completely
> deleted an account on my iPhone, then recreated it while in airplane
> mode to make sure I didn't miss any communications in my trace. I
> started the trace, exited airplane mode and let the iPhone do an
> initial
> sync with the account. The inbox folder in this account has over
> 1000
> emails going back to early 2019.
>
> I'm not an expert in IMAP. But it appears that the iPhone mail app
> requests all of the emails 1:* (see line 812), but JAMES returns a
> single id plus two ranges (line 813). But the total count JAMES
> reports
> is nowhere near the full 1000. Subsequently (line 822), iPhone
> requests
> the emails JAMES told it about in line 813. From what I can tell,
> the
> problem is in line 813. JAMES did not report all of the actual mail
> that is in the INBOX, which is exactly what I'm seeing on the
> phone.
> Shouldn't JAMES return all 1000+ email ids that exist in INBOX? Or
> am I
> reading the IMAP trace incorrectly?
You are right, if there's 1000+ messages, it should be returned by
these calls.
>
> Note that the only emails JAMES reports came in AFTER I migrated to
> the
> new server and to the James 3.4 from v3b5. It appears that there is
> something different about pre-existing emails in the mailbox folder
> that
> is causing JAMES not to recognize them. But this problem ONLY exists
> on
> the iPhone mail app. Thunderbird gets all of the mail for the
> inbox.
> So is iPhone must be sending a slightly different request command
> syntax
> than other clients do (??) Just speculating....
>
The best thing to do is to get a trace from Thunderbird for the same
scenario.
My best guess is that one client trigger a search whereas the other is
just triggering a full-scan.
Are you using JPA? With with indexing backend?
--
Matthieu Baechler
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Servers Wipes Out iPhone's Mail Accounts (IMAP Trace
attached)
Posted by Jerry Malcolm <te...@malcolms.com>.
Ok, I need an IMAP expert.... Below is a very brief trace of the
communications between iPhone mail and JAMES (3.4). I completely
deleted an account on my iPhone, then recreated it while in airplane
mode to make sure I didn't miss any communications in my trace. I
started the trace, exited airplane mode and let the iPhone do an initial
sync with the account. The inbox folder in this account has over 1000
emails going back to early 2019.
I'm not an expert in IMAP. But it appears that the iPhone mail app
requests all of the emails 1:* (see line 812), but JAMES returns a
single id plus two ranges (line 813). But the total count JAMES reports
is nowhere near the full 1000. Subsequently (line 822), iPhone requests
the emails JAMES told it about in line 813. From what I can tell, the
problem is in line 813. JAMES did not report all of the actual mail
that is in the INBOX, which is exactly what I'm seeing on the phone.
Shouldn't JAMES return all 1000+ email ids that exist in INBOX? Or am I
reading the IMAP trace incorrectly?
Note that the only emails JAMES reports came in AFTER I migrated to the
new server and to the James 3.4 from v3b5. It appears that there is
something different about pre-existing emails in the mailbox folder that
is causing JAMES not to recognize them. But this problem ONLY exists on
the iPhone mail app. Thunderbird gets all of the mail for the inbox.
So is iPhone must be sending a slightly different request command syntax
than other clients do (??) Just speculating....
798 IMAP Request: 37 SELECT INBOX
802 IMAP Response: 37 OK [READ-WRITE] SELECT completed.
806 IMAP Request: 38 UID SEARCH RETURN (COUNT) 1:* NOT DELETED
808 IMAP Response: * ESEARCH (TAG "38") UID COUNT 76
809 IMAP Response: 38 OK SEARCH completed.
812 IMAP Request: 39 UID SEARCH RETURN (ALL) 1:* NOT DELETED
813 IMAP Response: * ESEARCH (TAG "39") UID ALL 7776,7955:7970,7975:8033
814 IMAP Response: 39 OK SEARCH completed.
817 IMAP Request: 40 UID SEARCH RETURN (ALL) 929:* NOT DELETED
818 IMAP Response: * ESEARCH (TAG "40") UID ALL 7955:7970,7975:8033
819 IMAP Response: 40 OK SEARCH completed.
822 IMAP Request: 41 UID FETCH 7955:7970,7975:8033 (INTERNALDATE UID
RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (date subject from
content-type to cc bcc message-id in-reply-to references list-id
x-universally-unique-identifier)])
On 10/22/2019 3:07 AM, Matthieu Baechler wrote:
> Hi Jerry,
>
> I don't think trying to guess what happen will help you much. I would
> rather try to capture the IMAP session.
>
> I see two ways to do that:
>
> 1. you write a IMAP session logger in James (it should be really easy,
> I don't know why it's not already there)
>
> 2. you disable SSL on the phone and do a tcpdump on the server
>
> Once you'll have the IMAP session capture, it will be easier to
> diagnose your problem.
>
> Cheers,
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Servers Wipes Out iPhone's Mail Accounts
Posted by Matthieu Baechler <ma...@apache.org>.
Hi,
On Tue, 2019-10-22 at 17:55 -0500, Jerry Malcolm wrote:
> Hi Matthieu,
>
> I can add some log/trace statements into the low-level imap
> processing.
> I tried digging into the source, but got pretty bogged down. If you
> can
> point me to the best java class to put the logging code, I can make
> that
> happen.
>
> In the meantime, to try to keep moving on the problem, I just did a
> tcpdump on my phone's ip address. Not quite as clean as a
> formatted
> imap trace. But the info is there. Hopefully someone knows what
> to
> look for in the trace. It was too large to include inline. It's at
> https://jwmhosting.com/imaptrace.txt
Do you have a raw tcpdump file (instead of a text version) so that I
can open it in wireshark?
> Let me know where to insert the imap trace in the code, and I'll try
> to
> get a cleaner trace tomorrow.
>
I had a second look at that but it's not as easy as I thought:
actually, Netty is providing a ChannelBuffer to
NettyImapRequestLineReader but this class, despite its name, is not
line-oriented, so it's not straightforward to have line-oriented logs.
I'm reluctant to modify this code for now as I have little knowledge of
the way it work.
So I suggest we go ahead with tcpdump strategy.
Cheers,
--
Matthieu Baechler
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Servers Wipes Out iPhone's Mail Accounts
Posted by Jerry Malcolm <te...@malcolms.com>.
Hi Matthieu,
I can add some log/trace statements into the low-level imap processing.
I tried digging into the source, but got pretty bogged down. If you can
point me to the best java class to put the logging code, I can make that
happen.
In the meantime, to try to keep moving on the problem, I just did a
tcpdump on my phone's ip address. Not quite as clean as a formatted
imap trace. But the info is there. Hopefully someone knows what to
look for in the trace. It was too large to include inline. It's at
https://jwmhosting.com/imaptrace.txt
Let me know where to insert the imap trace in the code, and I'll try to
get a cleaner trace tomorrow.
Thanks.
Jerry
On 10/22/2019 3:07 AM, Matthieu Baechler wrote:
> Hi Jerry,
>
> I don't think trying to guess what happen will help you much. I would
> rather try to capture the IMAP session.
>
> I see two ways to do that:
>
> 1. you write a IMAP session logger in James (it should be really easy,
> I don't know why it's not already there)
>
> 2. you disable SSL on the phone and do a tcpdump on the server
>
> Once you'll have the IMAP session capture, it will be easier to
> diagnose your problem.
>
> Cheers,
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Servers Wipes Out iPhone's Mail Accounts
Posted by Matthieu Baechler <ma...@apache.org>.
Hi Jerry,
I don't think trying to guess what happen will help you much. I would
rather try to capture the IMAP session.
I see two ways to do that:
1. you write a IMAP session logger in James (it should be really easy,
I don't know why it's not already there)
2. you disable SSL on the phone and do a tcpdump on the server
Once you'll have the IMAP session capture, it will be easier to
diagnose your problem.
Cheers,
--
Matthieu Baechler
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org