You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "btellier@apache.org" <bt...@apache.org> on 2021/09/05 13:03:16 UTC
Retiring the File mail queue from James 3.7.x
Hello there,
While working on JAMES-3646 incorrect file validation, I came to
discover the file mail queue incorrectly sanitizes file paths.
This is one of the many flows of this component. Here are some others:
FileMailQueue is an outdated unmaintained component suffering incomplete
features and is not thread safe" +
"This includes: " +
" - JAMES-2298 Unsupported remove management feature" +
" - JAMES-2954 Incomplete browse implementation" +
" - JAMES-2544 Mixing concurrent operation might lead to a deadlock and
missing fields" +
" - JAMES-2979 dequeue is not thread safe
(To quote its test)
It is not directly configurable with any exosting products.
There are some alternatives: embedded activeMQ has zero dependencies.
Furthermore migrating is easy: just migrate with an empty queue - which
can be done easily with a SMTP downtime.
Also the component is deprecated for 1.5 years, given the many flows it
have, given that no contributors shows up to keep it alive, I advocate
removal.
Feedback?
Regards,
Benoit
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Retiring the File mail queue from James 3.7.x
Posted by "btellier@apache.org" <bt...@apache.org>.
Hi Garry,
On 07/09/2021 20:29, Jean Helou wrote:
> Hi garry,
>
> I think this concern is more valid for the retirement of the maildir[1]
> implementation (mailbox) than for the mailqueue.
> I don't think messages are expected to remain in the mailqueue for a very
> long time, which is why the advocated migration path includes downtime :
> - stop accepting new email traffic
> - let the mailqueue process in-flight messages
> - switch mailqueue implementation
> - start accepting traffic again
>
> for maildir (fs based mailbox implementation) it is more tricky as you need
> to do a data migration :
> read the whole mailbox content and rewrite it in the new store
What protocols are supported by the 2.3 release line? As far as I am
aware of, it mostly is POP3, and - again to my own experience - POP3
user do not tend to have a large mailbox (as POP3 causes it to be read
again and again and over again), I doubt that we are speaking of the
same volumetry than for instance an IMAP server migration.
Regarding migrating across servers, I did personnally run several time
imapsync [1] migration for 5000+ accounts and 3TB data, we could have
done easily more. Similar tools exists for POP3 -> IMAP migration [2]
and we do already have some documentation detailing the process [3].
Now regarding the James 2.3.x -> 3.x migration process leveraging the
maildir storage format, are we sure that it is a drop in replacement?
Are we sure we did not introduce some subttle incompatibilities? Are we
sure we want to end up maintaining people with a MailDir format for the
years to come in James 3.x ? (I don't) The topic of migrating across
mailbox stores is undocumented, untested, and require a lot of custom
setup. I would not be able to do it myself, so supporting people doing
those?
On the James 2.3 -> 3.x migration topic, rather than trying at all cost
to maintain a James specific solution leveraging outdated technology,
why not rely on the wonderful tools offered by the mail eco system?
[1] https://imapsync.lamiral.info/
[2] http://www.linux-france.org/prj/pop2imap/
[3]
https://james.staged.apache.org/james-project/3.7.0/servers/distributed/operate/migrating.html
Cheers,
Benoit
>
> [1]
> http://mail-archives.apache.org/mod_mbox/james-server-dev/202109.mbox/browser
>
> On Tue, Sep 7, 2021 at 3:19 PM Garry Hurley <ga...@gmail.com>
> wrote:
>
>> My only concern would be those people who are still, for some reason, not
>> yet ready to move to James 3.x - yes, there are still some out there. They
>> are more likely to be using the file repo than 3.x users. If they are
>> using a file repo, we might want to keep a migration tool around to migrate
>> their 'old messages' to one of the newer repos. I know we don't always
>> think about it, but some organizations (particularly government ones) have
>> a records retention policy that mandates they keep old emails for years
>> rather than days. Those users need a - maintained - way of migrating from
>> the file repo to the newer one.
>>
>>
>> Just my two cents.
>>
>> On Mon, Sep 6, 2021 at 10:18 AM Raphaël Ouazana-Sustowski <
>> rouazana@apache.org> wrote:
>>
>>> +1
>>>
>>> Le 06/09/2021 à 14:54, Jean Helou a écrit :
>>>> awesome :)
>>>>
>>>> On Mon, Sep 6, 2021 at 1:52 PM btellier@apache.org <
>> btellier@apache.org>
>>>> wrote:
>>>>
>>>>> Hello Jean,
>>>>>
>>>>> On 06/09/2021 18:42, Jean Helou wrote:
>>>>>> Hello benoit,
>>>>>>
>>>>>>
>>>>>>> There are some alternatives: embedded activeMQ has zero
>> dependencies.
>>>>>>> Furthermore migrating is easy: just migrate with an empty queue -
>>> which
>>>>>>> can be done easily with a SMTP downtime.
>>>>>>>
>>>>>>> Also the component is deprecated for 1.5 years, given the many flows
>>> it
>>>>>>> have, given that no contributors shows up to keep it alive, I
>> advocate
>>>>>>> removal.
>>>>>>>
>>>>>> Sounds fine to me. Maybe make sure the documentation/sample server
>>>>> explains
>>>>>> how to properly configure activemq to have filesystem based msg
>>>>> persistence
>>>>>> (https://activemq.apache.org/kahadb) as an inmemory only activemq
>>> would
>>>>>> reduce the safety of the mailqueue ?
>>>>> KahaDB with an embedded AcctiveMQ is already the default for MailQueue
>>>>> (ActiveMQMailQueueFactory) for Spring and JPA-guice products.
>>>>>
>>>>> Regards
>>>>>> jean
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Retiring the File mail queue from James 3.7.x
Posted by Jean Helou <je...@gmail.com>.
Hi garry,
I think this concern is more valid for the retirement of the maildir[1]
implementation (mailbox) than for the mailqueue.
I don't think messages are expected to remain in the mailqueue for a very
long time, which is why the advocated migration path includes downtime :
- stop accepting new email traffic
- let the mailqueue process in-flight messages
- switch mailqueue implementation
- start accepting traffic again
for maildir (fs based mailbox implementation) it is more tricky as you need
to do a data migration :
read the whole mailbox content and rewrite it in the new store
[1]
http://mail-archives.apache.org/mod_mbox/james-server-dev/202109.mbox/browser
On Tue, Sep 7, 2021 at 3:19 PM Garry Hurley <ga...@gmail.com>
wrote:
> My only concern would be those people who are still, for some reason, not
> yet ready to move to James 3.x - yes, there are still some out there. They
> are more likely to be using the file repo than 3.x users. If they are
> using a file repo, we might want to keep a migration tool around to migrate
> their 'old messages' to one of the newer repos. I know we don't always
> think about it, but some organizations (particularly government ones) have
> a records retention policy that mandates they keep old emails for years
> rather than days. Those users need a - maintained - way of migrating from
> the file repo to the newer one.
>
>
> Just my two cents.
>
> On Mon, Sep 6, 2021 at 10:18 AM Raphaël Ouazana-Sustowski <
> rouazana@apache.org> wrote:
>
> > +1
> >
> > Le 06/09/2021 à 14:54, Jean Helou a écrit :
> > > awesome :)
> > >
> > > On Mon, Sep 6, 2021 at 1:52 PM btellier@apache.org <
> btellier@apache.org>
> > > wrote:
> > >
> > >> Hello Jean,
> > >>
> > >> On 06/09/2021 18:42, Jean Helou wrote:
> > >>> Hello benoit,
> > >>>
> > >>>
> > >>>> There are some alternatives: embedded activeMQ has zero
> dependencies.
> > >>>> Furthermore migrating is easy: just migrate with an empty queue -
> > which
> > >>>> can be done easily with a SMTP downtime.
> > >>>>
> > >>>> Also the component is deprecated for 1.5 years, given the many flows
> > it
> > >>>> have, given that no contributors shows up to keep it alive, I
> advocate
> > >>>> removal.
> > >>>>
> > >>> Sounds fine to me. Maybe make sure the documentation/sample server
> > >> explains
> > >>> how to properly configure activemq to have filesystem based msg
> > >> persistence
> > >>> (https://activemq.apache.org/kahadb) as an inmemory only activemq
> > would
> > >>> reduce the safety of the mailqueue ?
> > >> KahaDB with an embedded AcctiveMQ is already the default for MailQueue
> > >> (ActiveMQMailQueueFactory) for Spring and JPA-guice products.
> > >>
> > >> Regards
> > >>>
> > >>> jean
> > >>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> > >> For additional commands, e-mail: server-dev-help@james.apache.org
> > >>
> > >>
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-dev-help@james.apache.org
> >
> >
>
Re: Retiring the File mail queue from James 3.7.x
Posted by Garry Hurley <ga...@gmail.com>.
My only concern would be those people who are still, for some reason, not
yet ready to move to James 3.x - yes, there are still some out there. They
are more likely to be using the file repo than 3.x users. If they are
using a file repo, we might want to keep a migration tool around to migrate
their 'old messages' to one of the newer repos. I know we don't always
think about it, but some organizations (particularly government ones) have
a records retention policy that mandates they keep old emails for years
rather than days. Those users need a - maintained - way of migrating from
the file repo to the newer one.
Just my two cents.
On Mon, Sep 6, 2021 at 10:18 AM Raphaël Ouazana-Sustowski <
rouazana@apache.org> wrote:
> +1
>
> Le 06/09/2021 à 14:54, Jean Helou a écrit :
> > awesome :)
> >
> > On Mon, Sep 6, 2021 at 1:52 PM btellier@apache.org <bt...@apache.org>
> > wrote:
> >
> >> Hello Jean,
> >>
> >> On 06/09/2021 18:42, Jean Helou wrote:
> >>> Hello benoit,
> >>>
> >>>
> >>>> There are some alternatives: embedded activeMQ has zero dependencies.
> >>>> Furthermore migrating is easy: just migrate with an empty queue -
> which
> >>>> can be done easily with a SMTP downtime.
> >>>>
> >>>> Also the component is deprecated for 1.5 years, given the many flows
> it
> >>>> have, given that no contributors shows up to keep it alive, I advocate
> >>>> removal.
> >>>>
> >>> Sounds fine to me. Maybe make sure the documentation/sample server
> >> explains
> >>> how to properly configure activemq to have filesystem based msg
> >> persistence
> >>> (https://activemq.apache.org/kahadb) as an inmemory only activemq
> would
> >>> reduce the safety of the mailqueue ?
> >> KahaDB with an embedded AcctiveMQ is already the default for MailQueue
> >> (ActiveMQMailQueueFactory) for Spring and JPA-guice products.
> >>
> >> Regards
> >>>
> >>> jean
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> >> For additional commands, e-mail: server-dev-help@james.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
Re: Retiring the File mail queue from James 3.7.x
Posted by Raphaël Ouazana-Sustowski <ro...@apache.org>.
+1
Le 06/09/2021 à 14:54, Jean Helou a écrit :
> awesome :)
>
> On Mon, Sep 6, 2021 at 1:52 PM btellier@apache.org <bt...@apache.org>
> wrote:
>
>> Hello Jean,
>>
>> On 06/09/2021 18:42, Jean Helou wrote:
>>> Hello benoit,
>>>
>>>
>>>> There are some alternatives: embedded activeMQ has zero dependencies.
>>>> Furthermore migrating is easy: just migrate with an empty queue - which
>>>> can be done easily with a SMTP downtime.
>>>>
>>>> Also the component is deprecated for 1.5 years, given the many flows it
>>>> have, given that no contributors shows up to keep it alive, I advocate
>>>> removal.
>>>>
>>> Sounds fine to me. Maybe make sure the documentation/sample server
>> explains
>>> how to properly configure activemq to have filesystem based msg
>> persistence
>>> (https://activemq.apache.org/kahadb) as an inmemory only activemq would
>>> reduce the safety of the mailqueue ?
>> KahaDB with an embedded AcctiveMQ is already the default for MailQueue
>> (ActiveMQMailQueueFactory) for Spring and JPA-guice products.
>>
>> Regards
>>>
>>> jean
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Retiring the File mail queue from James 3.7.x
Posted by Jean Helou <je...@gmail.com>.
awesome :)
On Mon, Sep 6, 2021 at 1:52 PM btellier@apache.org <bt...@apache.org>
wrote:
> Hello Jean,
>
> On 06/09/2021 18:42, Jean Helou wrote:
> > Hello benoit,
> >
> >
> >> There are some alternatives: embedded activeMQ has zero dependencies.
> >> Furthermore migrating is easy: just migrate with an empty queue - which
> >> can be done easily with a SMTP downtime.
> >>
> >> Also the component is deprecated for 1.5 years, given the many flows it
> >> have, given that no contributors shows up to keep it alive, I advocate
> >> removal.
> >>
> > Sounds fine to me. Maybe make sure the documentation/sample server
> explains
> > how to properly configure activemq to have filesystem based msg
> persistence
> > (https://activemq.apache.org/kahadb) as an inmemory only activemq would
> > reduce the safety of the mailqueue ?
> KahaDB with an embedded AcctiveMQ is already the default for MailQueue
> (ActiveMQMailQueueFactory) for Spring and JPA-guice products.
>
> Regards
> >
> > jean
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
Re: Retiring the File mail queue from James 3.7.x
Posted by "btellier@apache.org" <bt...@apache.org>.
Hello Jean,
On 06/09/2021 18:42, Jean Helou wrote:
> Hello benoit,
>
>
>> There are some alternatives: embedded activeMQ has zero dependencies.
>> Furthermore migrating is easy: just migrate with an empty queue - which
>> can be done easily with a SMTP downtime.
>>
>> Also the component is deprecated for 1.5 years, given the many flows it
>> have, given that no contributors shows up to keep it alive, I advocate
>> removal.
>>
> Sounds fine to me. Maybe make sure the documentation/sample server explains
> how to properly configure activemq to have filesystem based msg persistence
> (https://activemq.apache.org/kahadb) as an inmemory only activemq would
> reduce the safety of the mailqueue ?
KahaDB with an embedded AcctiveMQ is already the default for MailQueue
(ActiveMQMailQueueFactory) for Spring and JPA-guice products.
Regards
>
> jean
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Retiring the File mail queue from James 3.7.x
Posted by Jean Helou <je...@gmail.com>.
Hello benoit,
> There are some alternatives: embedded activeMQ has zero dependencies.
> Furthermore migrating is easy: just migrate with an empty queue - which
> can be done easily with a SMTP downtime.
>
> Also the component is deprecated for 1.5 years, given the many flows it
> have, given that no contributors shows up to keep it alive, I advocate
> removal.
>
Sounds fine to me. Maybe make sure the documentation/sample server explains
how to properly configure activemq to have filesystem based msg persistence
(https://activemq.apache.org/kahadb) as an inmemory only activemq would
reduce the safety of the mailqueue ?
jean
Re: Retiring the File mail queue from James 3.7.x
Posted by Rene Cordier <rc...@apache.org>.
Hello,
Sounds legit to me, +1!
Thanks for looking into that deeply.
Rene.
On 05/09/2021 20:03, btellier@apache.org wrote:
> Hello there,
>
> While working on JAMES-3646 incorrect file validation, I came to
> discover the file mail queue incorrectly sanitizes file paths.
>
> This is one of the many flows of this component. Here are some others:
>
> FileMailQueue is an outdated unmaintained component suffering incomplete
> features and is not thread safe" +
> "This includes: " +
> " - JAMES-2298 Unsupported remove management feature" +
> " - JAMES-2954 Incomplete browse implementation" +
> " - JAMES-2544 Mixing concurrent operation might lead to a deadlock and
> missing fields" +
> " - JAMES-2979 dequeue is not thread safe
>
> (To quote its test)
>
> It is not directly configurable with any exosting products.
>
> There are some alternatives: embedded activeMQ has zero dependencies.
> Furthermore migrating is easy: just migrate with an empty queue - which
> can be done easily with a SMTP downtime.
>
> Also the component is deprecated for 1.5 years, given the many flows it
> have, given that no contributors shows up to keep it alive, I advocate
> removal.
>
> Feedback?
>
> Regards,
>
> Benoit
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org