You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-user@james.apache.org by Jerry Malcolm <te...@malcolms.com> on 2017/03/25 19:35:24 UTC
Moving Archived Messages Out of Database
I have a client who has some folders with 150,000+ messages in them.
When he accesses these folders via Thunderbird or other clients, James
gets swamped with returning the message list and slows down responding
to all my other clients. Most of these messages are old and could be
archived. But just moving them to an archive folder just means that if
he opens the archive folder, I'm back to the same problem. I know I
could just tell him to delete about 90% of them. But I don't want to
impose limits. I thought about asking him to unsubscribe to these
folders. That'll work in Tbird. But iPhone mail doesn't honor the
subscribe bit and displays the folder anyway. Maybe better than nothing...
What I'd really like to do is ask him to store old archived messages
locally on his machine and just get them out of my server completely.
But I want to get some advice on the best way to do that.
What I'm thinking about doing is:
1) Create a completely separate 'archive' user account for him
2) Using SQL, move the super large folders to this archive account
3) Have him create a matching 'archive' Thunderbird account, but make it
POP3 type rather than IMAP and set it to 'delete mail on server after
retrieval'
4) Have him open/download (and inherently erase) the messages from the
server.
I'm pretty sure this will work. (Will probably take two days to
download....) But my question is.... is this the best way to do this.
Surely other JAMES users have encountered this problem as well. Am I
reinventing the wheel? Is there a better way?
Thx.
Jerry
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Archived Messages Out of Database
Posted by Matthieu Baechler <mb...@linagora.com>.
A possible way to do that is to create another account for your user and
moving the emails on this account, I guess.
You then can make sure this account is only used with POP3.
--
Matthieu Baechler
Le 27/03/2017 � 18:41, Jerry Malcolm a �crit :
> I'm using MySQL.
>
> I'd love to get better performance on large folders. However, I'd
> prefer to make clients move old archive folders completely off the
> server and store locally (the way POP3 does). Since IMAP doesn't
> support doing that directly, I'm looking for suggestions on how to
> move hundreds of thousands of emails off the server completely.
>
> Thanks.
>
> Jerry
>
> On 3/27/2017 2:37 AM, Matthieu Baechler wrote:
>> Hi,
>>
>> I can confirm what Benoit said : making large mailboxes fast is not a
>> easy task.
>>
>> However, it's not impossible to do, it's just a matter of preparing
>> the data for SELECT to being fast.
>>
>> What mailbox backend are you using ?
>>
>> Regards,
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Archived Messages Out of Database
Posted by Jerry Malcolm <te...@malcolms.com>.
I'm using MySQL.
I'd love to get better performance on large folders. However, I'd
prefer to make clients move old archive folders completely off the
server and store locally (the way POP3 does). Since IMAP doesn't
support doing that directly, I'm looking for suggestions on how to move
hundreds of thousands of emails off the server completely.
Thanks.
Jerry
On 3/27/2017 2:37 AM, Matthieu Baechler wrote:
> Hi,
>
> I can confirm what Benoit said : making large mailboxes fast is not a
> easy task.
>
> However, it's not impossible to do, it's just a matter of preparing
> the data for SELECT to being fast.
>
> What mailbox backend are you using ?
>
> Regards,
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Archived Messages Out of Database
Posted by Matthieu Baechler <mb...@linagora.com>.
Hi,
I can confirm what Benoit said : making large mailboxes fast is not a
easy task.
However, it's not impossible to do, it's just a matter of preparing the
data for SELECT to being fast.
What mailbox backend are you using ?
Regards,
--
Matthieu Baechler
Le 27/03/2017 � 08:23, Benoit Tellier a �crit :
> Hi Jerry,
>
> We also have some performance troubles while opening large mailboxes as
> well. This is something the team is currently working on.
>
> To make it short :
> - Opening a new, large mailbox implies removing RECENT flags from all
> the stored messages (no choices)
> - Then you have to build the MSN <-> UID mapping, which is linear with
> the mailbox size.
>
> Both of them are limitations of the IMAP protocol. Both of them implies
> reading the entire mailbox, and potentially updating flags for the
> entire mailbox.
>
> We currently work on performance (Cassandra) for SELECT on such large
> mailboxes. (Because our IMAP-sync scripts are pretty slow)
>
> I am not aware of shortcuts for dealing with such problems.
>
> Cheers,
>
> Benoit Tellier
>
> Le 26/03/2017 � 02:35, Jerry Malcolm a �crit :
>> I have a client who has some folders with 150,000+ messages in them.
>> When he accesses these folders via Thunderbird or other clients, James
>> gets swamped with returning the message list and slows down responding
>> to all my other clients. Most of these messages are old and could be
>> archived. But just moving them to an archive folder just means that if
>> he opens the archive folder, I'm back to the same problem. I know I
>> could just tell him to delete about 90% of them. But I don't want to
>> impose limits. I thought about asking him to unsubscribe to these
>> folders. That'll work in Tbird. But iPhone mail doesn't honor the
>> subscribe bit and displays the folder anyway. Maybe better than nothing...
>>
>> What I'd really like to do is ask him to store old archived messages
>> locally on his machine and just get them out of my server completely.
>> But I want to get some advice on the best way to do that.
>>
>> What I'm thinking about doing is:
>>
>> 1) Create a completely separate 'archive' user account for him
>>
>> 2) Using SQL, move the super large folders to this archive account
>>
>> 3) Have him create a matching 'archive' Thunderbird account, but make it
>> POP3 type rather than IMAP and set it to 'delete mail on server after
>> retrieval'
>>
>> 4) Have him open/download (and inherently erase) the messages from the
>> server.
>>
>> I'm pretty sure this will work. (Will probably take two days to
>> download....) But my question is.... is this the best way to do this.
>> Surely other JAMES users have encountered this problem as well. Am I
>> reinventing the wheel? Is there a better way?
>>
>> Thx.
>>
>> Jerry
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Moving Archived Messages Out of Database
Posted by Benoit Tellier <bt...@linagora.com>.
Hi Jerry,
We also have some performance troubles while opening large mailboxes as
well. This is something the team is currently working on.
To make it short :
- Opening a new, large mailbox implies removing RECENT flags from all
the stored messages (no choices)
- Then you have to build the MSN <-> UID mapping, which is linear with
the mailbox size.
Both of them are limitations of the IMAP protocol. Both of them implies
reading the entire mailbox, and potentially updating flags for the
entire mailbox.
We currently work on performance (Cassandra) for SELECT on such large
mailboxes. (Because our IMAP-sync scripts are pretty slow)
I am not aware of shortcuts for dealing with such problems.
Cheers,
Benoit Tellier
Le 26/03/2017 � 02:35, Jerry Malcolm a �crit :
> I have a client who has some folders with 150,000+ messages in them.
> When he accesses these folders via Thunderbird or other clients, James
> gets swamped with returning the message list and slows down responding
> to all my other clients. Most of these messages are old and could be
> archived. But just moving them to an archive folder just means that if
> he opens the archive folder, I'm back to the same problem. I know I
> could just tell him to delete about 90% of them. But I don't want to
> impose limits. I thought about asking him to unsubscribe to these
> folders. That'll work in Tbird. But iPhone mail doesn't honor the
> subscribe bit and displays the folder anyway. Maybe better than nothing...
>
> What I'd really like to do is ask him to store old archived messages
> locally on his machine and just get them out of my server completely.
> But I want to get some advice on the best way to do that.
>
> What I'm thinking about doing is:
>
> 1) Create a completely separate 'archive' user account for him
>
> 2) Using SQL, move the super large folders to this archive account
>
> 3) Have him create a matching 'archive' Thunderbird account, but make it
> POP3 type rather than IMAP and set it to 'delete mail on server after
> retrieval'
>
> 4) Have him open/download (and inherently erase) the messages from the
> server.
>
> I'm pretty sure this will work. (Will probably take two days to
> download....) But my question is.... is this the best way to do this.
> Surely other JAMES users have encountered this problem as well. Am I
> reinventing the wheel? Is there a better way?
>
> Thx.
>
> Jerry
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org