Consolidating languages (lua+python -> python) ?

[Daniel Gruno <hu...@apache.org>]

Hi folks,
I've been pondering about the difficulties PM faces in terms of getting
new contributors, and one of the core issues, as I see it, is our choice
of using Lua for the web application. Apart from locking people to
apache/nginx, it seems that people are - for lack of another word -
scared to touch the lua part of it, leaving only 2-3 people working on it.

Therefore, I would like to propose that we switch (after 0.10) to a pure
python setup for Pony Mail. Thus, both the archivers and the web app
would be python. This would also rid us of a lot of homegrown
self-invented wheels within the project.

I will of course do my best to help with the bulk of the work here, and
I would be immensely grateful if others would support this and help out
with the process of converting. I also have acknowledgements from a few
people that would like to help with the transition, should we agree on this.

So....thoughts?


With regards,
Daniel.

Re: Consolidating languages (lua+python -> python) ?

[Daniel Gruno <hu...@apache.org>]

Here are my thoughts (and progress) so far on the subject:

I have ported about half of the API at home, and am testing it while
also building some simple unit tests.

Once I am satisfied that the port covers the complete current API, I
will donate this to Apache for review and whatever we then choose what
to do next.

While this is not a 'rewrite' but rather a port, there are some changes
made to the way it works internally:

Improvements/changes:

- Abstract AAA away from endpoints. Don't trust stats.lua (or stats.py)
to handle security, put that in a core mbox library for fetching emails.
- No or very little raw ES queries in the end points (have a library for
this, "fetchEmail", "fetchList" etc which handles the AAA aspect for all
end points)
- Refactor the fuzzy search logic for stats.py, separate to its own
library, so both stats, mbox and atom can use it. Imagine downloading an
mbox file with emails matching a search/range, or subscribing to an
email search as an atom feed.
- Ditch debug/timing outputs entirely, JS can handle much of this.

Proposed structure:

Standard WSGI application, can be run with mod_wsgi in apache, or as
standalone via gunicorn or similar app.

wsgi --\
         - handler.py (WSGI handler, grab session/DB and run end point)
         - pages --\
                   - stats.py
                   - atom.py
                   - preferences.py
                   - etc
          - plugins --\
                       - defuzzer.py (search logic for stats/atom/mbox)
                       - mbox.py (fetch one/more email(s), check AAA)
                       - aaa.py (AAA lib)
                       - aaa_*.py (additional AAA libs, not tracked)
                       - database.py (abstract ES object)
                       - session.py (session object)
           - tests --\
                       - defuzzer.py (tests for defuzzer)
                       - aaa.py (tests for AAA)
                       - etc

This would be backwards compatible in that the new endpoints would be
/api/stats etc, but /api/stats.lua would also be accepted.

I could also be persuaded to just dump everything I have into the PM
repo under a branch (it's in VCS already, so history would be preserved)
if someone thinks that's a good idea. People could then test this very
easily with gunicorn or similar.

With regards,
Daniel.

Re: Consolidating languages (lua+python -> python) ?

[sebb <se...@gmail.com>]

On 4 June 2017 at 20:26, Daniel Gruno <hu...@apache.org> wrote:
> Hi folks,
> I've been pondering about the difficulties PM faces in terms of getting
> new contributors, and one of the core issues, as I see it, is our choice
> of using Lua for the web application. Apart from locking people to
> apache/nginx, it seems that people are - for lack of another word -
> scared to touch the lua part of it, leaving only 2-3 people working on it.
>
> Therefore, I would like to propose that we switch (after 0.10) to a pure
> python setup for Pony Mail. Thus, both the archivers and the web app
> would be python. This would also rid us of a lot of homegrown
> self-invented wheels within the project.
>
> I will of course do my best to help with the bulk of the work here, and
> I would be immensely grateful if others would support this and help out
> with the process of converting. I also have acknowledgements from a few
> people that would like to help with the transition, should we agree on this.
>
> So....thoughts?

At present most of the oustanding bugs/enhancements are in the GUI.

I think that should take priority.

Are there (m)any changes which need to be made to the Lua code?

I am -1 on converting the Lua code until the GUI has been sorted out.

If at some stage there is a *need* to revamp the Lua backend, I think
it needs to be approached very carefully.
Mistakes in server code can be very costly in terms of security.
Also much of the Lua code is effectively part of the public API.

There needs to be some exhaustive unit test data which can be proved
against the existing Lua code.
This can then be used to help weed out the inevitable bugs in the conversion.

>
> With regards,
> Daniel.

Re: Consolidating languages (lua+python -> python) ?

["John D. Ament" <jo...@apache.org>]

On Sun, Jun 4, 2017 at 4:04 PM Daniel Gruno <hu...@apache.org> wrote:

> On 06/04/2017 09:59 PM, John D. Ament wrote:
> > If we use python as the webapp, would you expect to use a python web
> > framework (ala django) for the UI?
>
> No, I would not expect us to use something as heavy as Django - I don't
> see the need at present. Our web app is primary a JSON REST API with a
> JS overlay, and as such, I think the frames needed are very small.
>
>
I said ala Django.  Not only Django, but other frameworks that may be of
interest.

The reason being - simpler code, lower barrier of entry.

I'm curious, how would you consider returning JSON over HTTP with Python?


> There is also backwards compat to think about. If we switched to a
> python web framework with batteries and kitchen sink, older
> installations would need a converter, and likely another database
> installed (assuming you meant to imply that django could handle sessions
> and such).
>
>
As far as I can tell, Elasticsearch could be that runtime.  I don't believe
there's any hard requirement to use another database.  Where are sessions
stored right now? Cookie only?


> It is entirely possible that I'm ignorant about what Django would have
> to offer here.
>
>
The way I see it, it (it being Django or another framework like Django)
becomes a consistent development platform, that handles all of our web
handling use cases and is simple enough to run.


> With regards,
> Daniel.
>
> >
> > On Sun, Jun 4, 2017 at 3:26 PM Daniel Gruno <hu...@apache.org>
> wrote:
> >
> >> Hi folks,
> >> I've been pondering about the difficulties PM faces in terms of getting
> >> new contributors, and one of the core issues, as I see it, is our choice
> >> of using Lua for the web application. Apart from locking people to
> >> apache/nginx, it seems that people are - for lack of another word -
> >> scared to touch the lua part of it, leaving only 2-3 people working on
> it.
> >>
> >> Therefore, I would like to propose that we switch (after 0.10) to a pure
> >> python setup for Pony Mail. Thus, both the archivers and the web app
> >> would be python. This would also rid us of a lot of homegrown
> >> self-invented wheels within the project.
> >>
> >> I will of course do my best to help with the bulk of the work here, and
> >> I would be immensely grateful if others would support this and help out
> >> with the process of converting. I also have acknowledgements from a few
> >> people that would like to help with the transition, should we agree on
> >> this.
> >>
> >> So....thoughts?
> >>
> >>
> >> With regards,
> >> Daniel.
> >>
> >
>
>

Re: Consolidating languages (lua+python -> python) ?

[Daniel Gruno <hu...@apache.org>]

On 06/04/2017 09:59 PM, John D. Ament wrote:
> If we use python as the webapp, would you expect to use a python web
> framework (ala django) for the UI?

No, I would not expect us to use something as heavy as Django - I don't
see the need at present. Our web app is primary a JSON REST API with a
JS overlay, and as such, I think the frames needed are very small.

There is also backwards compat to think about. If we switched to a
python web framework with batteries and kitchen sink, older
installations would need a converter, and likely another database
installed (assuming you meant to imply that django could handle sessions
and such).

It is entirely possible that I'm ignorant about what Django would have
to offer here.

With regards,
Daniel.

> 
> On Sun, Jun 4, 2017 at 3:26 PM Daniel Gruno <hu...@apache.org> wrote:
> 
>> Hi folks,
>> I've been pondering about the difficulties PM faces in terms of getting
>> new contributors, and one of the core issues, as I see it, is our choice
>> of using Lua for the web application. Apart from locking people to
>> apache/nginx, it seems that people are - for lack of another word -
>> scared to touch the lua part of it, leaving only 2-3 people working on it.
>>
>> Therefore, I would like to propose that we switch (after 0.10) to a pure
>> python setup for Pony Mail. Thus, both the archivers and the web app
>> would be python. This would also rid us of a lot of homegrown
>> self-invented wheels within the project.
>>
>> I will of course do my best to help with the bulk of the work here, and
>> I would be immensely grateful if others would support this and help out
>> with the process of converting. I also have acknowledgements from a few
>> people that would like to help with the transition, should we agree on
>> this.
>>
>> So....thoughts?
>>
>>
>> With regards,
>> Daniel.
>>
> 

Re: Consolidating languages (lua+python -> python) ?

["John D. Ament" <jo...@apache.org>]

If we use python as the webapp, would you expect to use a python web
framework (ala django) for the UI?

On Sun, Jun 4, 2017 at 3:26 PM Daniel Gruno <hu...@apache.org> wrote:

> Hi folks,
> I've been pondering about the difficulties PM faces in terms of getting
> new contributors, and one of the core issues, as I see it, is our choice
> of using Lua for the web application. Apart from locking people to
> apache/nginx, it seems that people are - for lack of another word -
> scared to touch the lua part of it, leaving only 2-3 people working on it.
>
> Therefore, I would like to propose that we switch (after 0.10) to a pure
> python setup for Pony Mail. Thus, both the archivers and the web app
> would be python. This would also rid us of a lot of homegrown
> self-invented wheels within the project.
>
> I will of course do my best to help with the bulk of the work here, and
> I would be immensely grateful if others would support this and help out
> with the process of converting. I also have acknowledgements from a few
> people that would like to help with the transition, should we agree on
> this.
>
> So....thoughts?
>
>
> With regards,
> Daniel.
>