You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "btellier@apache.org" <bt...@apache.org> on 2021/09/05 12:58:16 UTC
Time to retire maildir from James 3.7.X ?
Hello all,
While reviewing overall file path sanitizing (JAMES-3646) I found
maildir to incorrectly sanitize some file path.
Playing around, I discovered some magic names that leads to maildir
errors namely MAILBOX-406, and a NPE with an empty mailbox (MAILBOX-407).
Moreover, over time the list of maildir defects keeps on growing:
- MAILBOX-389 Mailbox rename fails with Maildir
- MAILBOX-393 mailboxId support for mailDir is partial
Now looking on the bug tracker I notice people encounter (and nobody
answers):
- MAILBOX-390 Unexpected error accessing mailbox
- MAILBOX-344 Maildir do not support MODSEQ search
- MAILBOX-299 Maildir should fail gracefully when mailbox name is too long
- MAILBOX-241 Maildir implementation does not check namespace on existance
- MAILBOX-183 readUidFile is taking too much time for large uid file
and blocks other threads
- JAMES-3612 Cryptic errors upon bad permissions
- JAMES-1515 Maildirstorage backend crashes with Exception due to
"wrong" filename
With such a high number of critical issues that no one dare fixing, it
sounds reasonable to me to state that maildir implementation do not
match quality standards that one would expect from an Apache project.
Other mail servers do have great maildir implenentation, which
mistakenly lead users to think the maildir implementation is a quality
one, which is very far from the truth.
Moreover, maildir code is outdated, hard to work with, and pulls us
back in many way. Given the currently small amount of contributors, we
do not have time and resources to give it the love it deserves.
Also, alternatives exist: embedded JPA database do also run well in
single instance mode, and could server as a maildir replacement. People
could use utilities like imapsync to plan their migration.
As such, I propose myself to retire maildir and remove related code from
the repository.
Regards,
Benoit
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Time to retire maildir from James 3.7.X ?
Posted by "btellier@apache.org" <bt...@apache.org>.
Hello Jean,
On 08/09/2021 20:58, Jean Helou wrote:
> Hello devs,
>
> I switch threads because I agree with the removal of the FS based mailqueue
> but still find the maildir removal to be a bit more difficult.
>
> Quoting from the other thread at
> https://www.mail-archive.com/server-dev@james.apache.org/msg70960.html
>
> What protocols are supported by the 2.3 release line? As far as I am
>> aware of, it mostly is POP3, and - again to my own experience - POP3
>> user do not tend to have a large mailbox (as POP3 causes it to be read
>> again and again and over again), I doubt that we are speaking of the
>> same volumetry than for instance an IMAP server migration.
>>
> Undoubtedly I don't know james code as well as you do but while gary
> mentioned a 2.3 migration, it doesn't really matter ...
> The maildir implementation currently exists in james 3.x[1], it is listed
> as an official mailbox implementation and not marked deprecated [2]
> Therefore it would be legitimate for users to use it with James 3 and
> dropping it from 3.7 without a deprecation period sounds a bit cavalier to
> me.
I agree on the 'cavalier' aspect of my initial proposal (whose purposes
was to try to move lines on that topic).
I would agree on marking it deprecated in 3.7.0 and remove
mailbox/maildir in the 3.8.0 release line.
>
> Also maildir is a fairly standard mailbox format, support for it allows for
> migration from other software to james
1. It is not because it is standard that we have to implement it.
2. We could imagine having a "Import from maildir" tool instead of a
full-fledged mailbox implementation, this would lower greatly
maintenance costs.
3. I have not seen any tests regarding maildir inter-operability. I have
serious doubts when I see critical files having a james- prefix
(james-acl, james-uidvalidity, james-uidlist)
4. If it was truly useful, maybe we would have some contributions on
this component?
5. As stated, I provide an alternative way to perform such a migration
using imapsync.
>
> Now you state:
>
>> With such a high number of critical issues that no one dare fixing
>>
> I may be mistaken as I didn't investigate in details but it sounds like
> most of the issues stem from a lack of sanitizing in FS paths.
No it's not. Maildir suffer from many other flows. For instance you
cannot rename mailboxes which is nonetheless a basic IMAP feature.
> You mention
> in JAMES-3646[3] that you are going to fix the sieve and FileRepositories.
> Considering the practices in the james codebase most of the sanitizing code
> is going to end up in a shared service.
Maildir accesses files directly without ending up using any "common
service". We would have to design a fix just for it.
As such, we could say "maildir has issues that no-ones dare to fix so we
deprecate and soon remove it".
> Once that's in place it would be
> easier to fix a few of the maildir bugs using this code and it could become
> a nice first contrib no ?
>
> On a side note MAILBOX-183[4] "readUidFile is taking too much time for
> large uid file and blocks other threads" is open but marked as a duplicate
> of a closed bug ... not sure if it is still applicable.
>
> Jean
>
> [1] https://github.com/apache/james-project/tree/master/mailbox/maildir
> [2] https://github.com/apache/james-project/blob/master/mailbox/README.md
> [3] https://issues.apache.org/jira/browse/JAMES-3646
> [4] https://issues.apache.org/jira/browse/MAILBOX-183
>
> On Mon, Sep 6, 2021 at 5:47 AM Rene Cordier <rc...@apache.org> wrote:
>
>> Hello,
>>
>> I think you are making good points. It's probably better to focus on
>> what works well if it's enough of an alternative to maildir (like
>> embedded JPA as you mentionned), with little resources we have.
>>
>> +1 !
>>
>> Rene.
>>
>> On 05/09/2021 19:58, btellier@apache.org wrote:
>>> Hello all,
>>>
>>> While reviewing overall file path sanitizing (JAMES-3646) I found
>>> maildir to incorrectly sanitize some file path.
>>>
>>> Playing around, I discovered some magic names that leads to maildir
>>> errors namely MAILBOX-406, and a NPE with an empty mailbox (MAILBOX-407).
>>>
>>> Moreover, over time the list of maildir defects keeps on growing:
>>>
>>> - MAILBOX-389 Mailbox rename fails with Maildir
>>> - MAILBOX-393 mailboxId support for mailDir is partial
>>>
>>> Now looking on the bug tracker I notice people encounter (and nobody
>>> answers):
>>>
>>> - MAILBOX-390 Unexpected error accessing mailbox
>>> - MAILBOX-344 Maildir do not support MODSEQ search
>>> - MAILBOX-299 Maildir should fail gracefully when mailbox name is too
>> long
>>> - MAILBOX-241 Maildir implementation does not check namespace on
>> existance
>>> - MAILBOX-183 readUidFile is taking too much time for large uid file
>>> and blocks other threads
>>> - JAMES-3612 Cryptic errors upon bad permissions
>>> - JAMES-1515 Maildirstorage backend crashes with Exception due to
>>> "wrong" filename
>>>
>>> With such a high number of critical issues that no one dare fixing, it
>>> sounds reasonable to me to state that maildir implementation do not
>>> match quality standards that one would expect from an Apache project.
>>>
>>> Other mail servers do have great maildir implenentation, which
>>> mistakenly lead users to think the maildir implementation is a quality
>>> one, which is very far from the truth.
>>>
>>> Moreover, maildir code is outdated, hard to work with, and pulls us
>>> back in many way. Given the currently small amount of contributors, we
>>> do not have time and resources to give it the love it deserves.
>>>
>>> Also, alternatives exist: embedded JPA database do also run well in
>>> single instance mode, and could server as a maildir replacement. People
>>> could use utilities like imapsync to plan their migration.
>>>
>>> As such, I propose myself to retire maildir and remove related code from
>>> the repository.
>>>
>>> Regards,
>>>
>>> Benoit
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Time to retire maildir from James 3.7.X ?
Posted by Jean Helou <je...@gmail.com>.
Hello devs,
I switch threads because I agree with the removal of the FS based mailqueue
but still find the maildir removal to be a bit more difficult.
Quoting from the other thread at
https://www.mail-archive.com/server-dev@james.apache.org/msg70960.html
What protocols are supported by the 2.3 release line? As far as I am
> aware of, it mostly is POP3, and - again to my own experience - POP3
> user do not tend to have a large mailbox (as POP3 causes it to be read
> again and again and over again), I doubt that we are speaking of the
> same volumetry than for instance an IMAP server migration.
>
Undoubtedly I don't know james code as well as you do but while gary
mentioned a 2.3 migration, it doesn't really matter ...
The maildir implementation currently exists in james 3.x[1], it is listed
as an official mailbox implementation and not marked deprecated [2]
Therefore it would be legitimate for users to use it with James 3 and
dropping it from 3.7 without a deprecation period sounds a bit cavalier to
me.
Also maildir is a fairly standard mailbox format, support for it allows for
migration from other software to james
Now you state:
> With such a high number of critical issues that no one dare fixing
>
I may be mistaken as I didn't investigate in details but it sounds like
most of the issues stem from a lack of sanitizing in FS paths. You mention
in JAMES-3646[3] that you are going to fix the sieve and FileRepositories.
Considering the practices in the james codebase most of the sanitizing code
is going to end up in a shared service. Once that's in place it would be
easier to fix a few of the maildir bugs using this code and it could become
a nice first contrib no ?
On a side note MAILBOX-183[4] "readUidFile is taking too much time for
large uid file and blocks other threads" is open but marked as a duplicate
of a closed bug ... not sure if it is still applicable.
Jean
[1] https://github.com/apache/james-project/tree/master/mailbox/maildir
[2] https://github.com/apache/james-project/blob/master/mailbox/README.md
[3] https://issues.apache.org/jira/browse/JAMES-3646
[4] https://issues.apache.org/jira/browse/MAILBOX-183
On Mon, Sep 6, 2021 at 5:47 AM Rene Cordier <rc...@apache.org> wrote:
> Hello,
>
> I think you are making good points. It's probably better to focus on
> what works well if it's enough of an alternative to maildir (like
> embedded JPA as you mentionned), with little resources we have.
>
> +1 !
>
> Rene.
>
> On 05/09/2021 19:58, btellier@apache.org wrote:
> > Hello all,
> >
> > While reviewing overall file path sanitizing (JAMES-3646) I found
> > maildir to incorrectly sanitize some file path.
> >
> > Playing around, I discovered some magic names that leads to maildir
> > errors namely MAILBOX-406, and a NPE with an empty mailbox (MAILBOX-407).
> >
> > Moreover, over time the list of maildir defects keeps on growing:
> >
> > - MAILBOX-389 Mailbox rename fails with Maildir
> > - MAILBOX-393 mailboxId support for mailDir is partial
> >
> > Now looking on the bug tracker I notice people encounter (and nobody
> > answers):
> >
> > - MAILBOX-390 Unexpected error accessing mailbox
> > - MAILBOX-344 Maildir do not support MODSEQ search
> > - MAILBOX-299 Maildir should fail gracefully when mailbox name is too
> long
> > - MAILBOX-241 Maildir implementation does not check namespace on
> existance
> > - MAILBOX-183 readUidFile is taking too much time for large uid file
> > and blocks other threads
> > - JAMES-3612 Cryptic errors upon bad permissions
> > - JAMES-1515 Maildirstorage backend crashes with Exception due to
> > "wrong" filename
> >
> > With such a high number of critical issues that no one dare fixing, it
> > sounds reasonable to me to state that maildir implementation do not
> > match quality standards that one would expect from an Apache project.
> >
> > Other mail servers do have great maildir implenentation, which
> > mistakenly lead users to think the maildir implementation is a quality
> > one, which is very far from the truth.
> >
> > Moreover, maildir code is outdated, hard to work with, and pulls us
> > back in many way. Given the currently small amount of contributors, we
> > do not have time and resources to give it the love it deserves.
> >
> > Also, alternatives exist: embedded JPA database do also run well in
> > single instance mode, and could server as a maildir replacement. People
> > could use utilities like imapsync to plan their migration.
> >
> > As such, I propose myself to retire maildir and remove related code from
> > the repository.
> >
> > Regards,
> >
> > Benoit
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-dev-help@james.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
Re: Time to retire maildir from James 3.7.X ?
Posted by Rene Cordier <rc...@apache.org>.
Hello,
I think you are making good points. It's probably better to focus on
what works well if it's enough of an alternative to maildir (like
embedded JPA as you mentionned), with little resources we have.
+1 !
Rene.
On 05/09/2021 19:58, btellier@apache.org wrote:
> Hello all,
>
> While reviewing overall file path sanitizing (JAMES-3646) I found
> maildir to incorrectly sanitize some file path.
>
> Playing around, I discovered some magic names that leads to maildir
> errors namely MAILBOX-406, and a NPE with an empty mailbox (MAILBOX-407).
>
> Moreover, over time the list of maildir defects keeps on growing:
>
> - MAILBOX-389 Mailbox rename fails with Maildir
> - MAILBOX-393 mailboxId support for mailDir is partial
>
> Now looking on the bug tracker I notice people encounter (and nobody
> answers):
>
> - MAILBOX-390 Unexpected error accessing mailbox
> - MAILBOX-344 Maildir do not support MODSEQ search
> - MAILBOX-299 Maildir should fail gracefully when mailbox name is too long
> - MAILBOX-241 Maildir implementation does not check namespace on existance
> - MAILBOX-183 readUidFile is taking too much time for large uid file
> and blocks other threads
> - JAMES-3612 Cryptic errors upon bad permissions
> - JAMES-1515 Maildirstorage backend crashes with Exception due to
> "wrong" filename
>
> With such a high number of critical issues that no one dare fixing, it
> sounds reasonable to me to state that maildir implementation do not
> match quality standards that one would expect from an Apache project.
>
> Other mail servers do have great maildir implenentation, which
> mistakenly lead users to think the maildir implementation is a quality
> one, which is very far from the truth.
>
> Moreover, maildir code is outdated, hard to work with, and pulls us
> back in many way. Given the currently small amount of contributors, we
> do not have time and resources to give it the love it deserves.
>
> Also, alternatives exist: embedded JPA database do also run well in
> single instance mode, and could server as a maildir replacement. People
> could use utilities like imapsync to plan their migration.
>
> As such, I propose myself to retire maildir and remove related code from
> the repository.
>
> Regards,
>
> Benoit
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org