Ponymail foal - why not use elastic.py and ponymailconfig.py?

[sebb <se...@gmail.com>]

I notice that the only tool that has been ported to ponymail-foal is
archive.py (and setup.py).

However rather than porting elastic.py and ponymailconfig.py,
archive.py does its own config checking and database setup.

Why is this?

S.

Re: Ponymail foal - why not use elastic.py and ponymailconfig.py?

[sebb <se...@gmail.com>]

On Mon, 17 Aug 2020 at 12:17, Daniel Gruno <hu...@apache.org> wrote:
>
> On 17/08/2020 11.31, sebb wrote:
> > On Mon, 17 Aug 2020 at 10:16, sebb <se...@gmail.com> wrote:
> >>
> >> I notice that the only tool that has been ported to ponymail-foal is
> >> archive.py (and setup.py).
> >>
> >> However rather than porting elastic.py and ponymailconfig.py,
> >> archive.py does its own config checking and database setup.
> >>
> >> Why is this?
> >
> > Sorry, I see now that archiver.py was never converted to use the
> > elastic.py interface.
> > I thought I had done all the tools.
> >
> > However it would still make sense to use the standard config access,
> > suitably modified for yaml.
> > There would have been no need to convert the config calls in the
> > original code, making it easier to see what has actually changed.
> >
> >> S.
>
> You mean mimicking the configparser code as a wrapper for yaml access?
> That could make sense I suppose :)

Just done it.

> As for import-mbox, I hadn't added that yet as ES 7.x is quite different
> from <=5.x, so we have to make changes to utilize it, and changes to how
> we store the data via the bulk operations. I'll look into that as time
> allows this week.
>
> Perhaps some sort of extra module with some default database index names
> would be useful, so we don't replicate that throughout the standard
> archive/import code.

The elastic.py utility is fairly generic (I think).
It mainly helps with handling database credentials and URI.
I've added that as well.

Re: Ponymail foal - why not use elastic.py and ponymailconfig.py?

[Daniel Gruno <hu...@apache.org>]

On 17/08/2020 11.31, sebb wrote:
> On Mon, 17 Aug 2020 at 10:16, sebb <se...@gmail.com> wrote:
>>
>> I notice that the only tool that has been ported to ponymail-foal is
>> archive.py (and setup.py).
>>
>> However rather than porting elastic.py and ponymailconfig.py,
>> archive.py does its own config checking and database setup.
>>
>> Why is this?
> 
> Sorry, I see now that archiver.py was never converted to use the
> elastic.py interface.
> I thought I had done all the tools.
> 
> However it would still make sense to use the standard config access,
> suitably modified for yaml.
> There would have been no need to convert the config calls in the
> original code, making it easier to see what has actually changed.
> 
>> S.

You mean mimicking the configparser code as a wrapper for yaml access? 
That could make sense I suppose :)

As for import-mbox, I hadn't added that yet as ES 7.x is quite different 
from <=5.x, so we have to make changes to utilize it, and changes to how 
we store the data via the bulk operations. I'll look into that as time 
allows this week.

Perhaps some sort of extra module with some default database index names 
would be useful, so we don't replicate that throughout the standard 
archive/import code.

Re: Ponymail foal - why not use elastic.py and ponymailconfig.py?

[sebb <se...@gmail.com>]

On Mon, 17 Aug 2020 at 10:16, sebb <se...@gmail.com> wrote:
>
> I notice that the only tool that has been ported to ponymail-foal is
> archive.py (and setup.py).
>
> However rather than porting elastic.py and ponymailconfig.py,
> archive.py does its own config checking and database setup.
>
> Why is this?

Sorry, I see now that archiver.py was never converted to use the
elastic.py interface.
I thought I had done all the tools.

However it would still make sense to use the standard config access,
suitably modified for yaml.
There would have been no need to convert the config calls in the
original code, making it easier to see what has actually changed.

> S.