TODO for mod_mbox

[Paul Querna <ch...@force-elite.com>]

Shorter Term List:
* Remove automake dependency (pure autoconf+Makefile.in)
* Rename mod-mbox-util. (mboxutil? something else?  All I know is that
the current name will change)
* Update trunk/scripts to use mod-mbox-util
* Add some *basic* documentation on configuration

Longer Term Things:

* Interface.  This is a SoC project.  We need some thoughts on if we
want to use some kind of Theme system (eg, ClearSilver or XSLT) or stick
with hard coded HTML.  I think a massive addition of CSS is a given, but
the details are still up in the air.

* Searching.  This is a SoC project.  I do have some of the low level
indexing bits done for Lucene4c in trunk.  We need further discussion if
we want to even use Lucene4c.  Some have suggested using Postgres or
MySQL with their full text indexing tools.  Personally, I like using
Lucene4c since it runs right off the file system, and is a
eat-your-own-dog-food type thing.

* General Design.  Currently, mod_mbox is tied to a single .mbox file,
or optionally, a directory of .mbox files.  As we add searching and
improve the interface, we will want the ability to search all mailing
lists, a single list, or maybe all lists under a domain.  This means
mod_mbox needs a knowledge of what other lists exist.  This is really
about turning mod_mbox into a 'Mail Archive Application' vs just
something that runs off of mbox files.  There are advantages and
disadvantages to both directions.  For the ASF's use cases, I believe
making it more of a unified application is good.

Other ideas?

-Paul

Re: TODO for mod_mbox

[Justin Erenkrantz <ju...@erenkrantz.com>]

On Thu, Jun 02, 2005 at 11:35:09AM -0700, Paul Querna wrote:
> Shorter Term List:
> * Remove automake dependency (pure autoconf+Makefile.in)

Yes, this is a condition before any release given my -1 for automake.  =)

> * Rename mod-mbox-util. (mboxutil? something else?  All I know is that
> the current name will change)

I'm not a fan of 'busybox' style programs.  I think that style of programming
makes it too difficult for people to understand as the program tries to do too
much.  However, I won't push too hard.

> * Update trunk/scripts to use mod-mbox-util

These should be done in concert with updating the mod_mbox install on ajax.
Perhaps this is something we can coordinate at the ApacheCon Hackathon next
month?  (Honestly, I probably won't have the full two-three days to
re-initializing everything until ApacheCon.)

> * Add some *basic* documentation on configuration
> 
> Longer Term Things:
> 
> * Interface.  This is a SoC project.  We need some thoughts on if we
> want to use some kind of Theme system (eg, ClearSilver or XSLT) or stick
> with hard coded HTML.  I think a massive addition of CSS is a given, but
> the details are still up in the air.

Yup.

> * Searching.  This is a SoC project.  I do have some of the low level
> indexing bits done for Lucene4c in trunk.  We need further discussion if
> we want to even use Lucene4c.  Some have suggested using Postgres or
> MySQL with their full text indexing tools.  Personally, I like using
> Lucene4c since it runs right off the file system, and is a
> eat-your-own-dog-food type thing.

Part of the intention with mod_mbox was not to be tied to a relational
database.  So, I'm against MySQL or Postgres being mandatory in any form.
Lucene4c is by far the best solution and eating our dog food is goodness.

> * General Design.  Currently, mod_mbox is tied to a single .mbox file,
> or optionally, a directory of .mbox files.  As we add searching and
> improve the interface, we will want the ability to search all mailing
> lists, a single list, or maybe all lists under a domain.  This means
> mod_mbox needs a knowledge of what other lists exist.  This is really
> about turning mod_mbox into a 'Mail Archive Application' vs just
> something that runs off of mbox files.  There are advantages and
> disadvantages to both directions.  For the ASF's use cases, I believe
> making it more of a unified application is good.

I feel a lot of the power of mod_mbox comes from its core simplicity.  

I'm ambivalent about joining everything together in a 'single' application.
In the past when we did searching (via htdig, etc.), we kept that disconnected
from mod_mbox.  I'm not sure why searching means that we have to have a 'Mail
Archive Application.'

So, I would really want to see a lot of thought into why we'd go from a
handler based module into an incredibly more complicated (and error-prone!)
catch-all system that would override Apache's file logic (groan!).  -- justin

Re: TODO for mod_mbox

[Justin Erenkrantz <ju...@erenkrantz.com>]

On Thu, Jun 02, 2005 at 11:35:09AM -0700, Paul Querna wrote:
> Other ideas?

Probably worth adding these to STATUS as to-do items.  ;-)

I'm on my way out to view Star Wars now; but I'll take a look at your list
later tonight...  -- justin

Re: TODO for mod_mbox

[Justin Erenkrantz <ju...@erenkrantz.com>]

On Fri, Jun 03, 2005 at 02:28:40AM +0200, Maxime Petazzoni wrote:
> My name is Maxime Petazzoni, I'm a french student interested by the
> SoC project about redesigning the mod_mbox interface (httpd-mbox-if).
> I'm willing to get involved in Apache's development, and I believe
> this project is a good start. Not that big, but with some challenge
> (along with the "time attack" involved by the SoC).

Hello!  Welcome.

> I'm not yet entirely familiar with the mod_mbox code, but I have to
> say that I find the parts I've seen a bit messy. It maybe because I'm
> not used to ASF coding style and it messes my overall understanding of
> the code.

Perhaps.  I initially wrote most of the code back in 2001 - I really haven't
gone through the code since then.  Most of my time on mod_mbox since then has
been to write the supporting scripts for it.  Paul's picked it up recently and
added some nifty features.

The style guide is at: http://httpd.apache.org/dev/styleguide.html.

> > * General Design.  Currently, mod_mbox is tied to a single .mbox
> > file, or optionally, a directory of .mbox files.  As we add
> > searching and improve the interface, we will want the ability to
> > search all mailing lists, a single list, or maybe all lists under a
> > domain.  This means mod_mbox needs a knowledge of what other lists
> > exist.  This is really about turning mod_mbox into a 'Mail Archive
> > Application' vs just something that runs off of mbox files.  There
> > are advantages and disadvantages to both directions.  For the ASF's
> > use cases, I believe making it more of a unified application is
> > good.
> 
> This is an interesting point :) As you said, actually mod_mbox serves
> only one mailing list archives. If we want it to handle mutliple
> lists, we are not talking about design, but redesign. All the actual
> mod_mbox code would only be a small part of this new mod_mbox.
> 
> The question that must be asked is : does a so complex application
> should be an Apache module ? Imho, actual mod_mbox as an module is
> great because it's very fast, but I don't believe C is the best
> language for the large application you are talking about. Maybe
> something could mix both together ?

C *is* the language of mod_mbox.  =)

If you want to write mod_mbox in Java, go look at Eyebrowse or other mail
archiving systems.  But, the general design and choice of language of mod_mbox
is not really open to debate at this point in time.  mod_mbox is an Apache
httpd module and is also highly tied into a RESTful architecture in a way that
few other mail archivers have even contemplated.  -- justin

Re: TODO for mod_mbox

[Maxime Petazzoni <ma...@bulix.org>]

Hi,

This is my first post to the list, so I think I'll begin with a little
introduction about myself.

My name is Maxime Petazzoni, I'm a french student interested by the
SoC project about redesigning the mod_mbox interface (httpd-mbox-if).
I'm willing to get involved in Apache's development, and I believe
this project is a good start. Not that big, but with some challenge
(along with the "time attack" involved by the SoC).

> * Interface.  This is a SoC project.  We need some thoughts on if we
> want to use some kind of Theme system (eg, ClearSilver or XSLT) or
> stick with hard coded HTML.  I think a massive addition of CSS is a
> given, but the details are still up in the air.

As I said, I'm interested in this project. I've started looking at the
mod_mbox code, and chipig helped me setting this thing up and running
on my box (we almost took down my system in the meanwhile, but anyway,
it works).

I've already made some changes to the mod_mbox_index.c file so it
outputs a clean XML file, along with an appropriate stylesheet I got a
nice (but very simple) result : http://skikda.bulix.org/archives/dev/

I'm not yet using any Apache integrated XSL Transformation, so you'll
need a Gecko-based browser to be able to have an overview of this work
in progress.

From what I've seen from the mod_mbox source code, the HTML output it
really hardcoded and that's why I've choose XML output : I prefer
hardcoded XML than hardcoded HTML, although I'm not really satisfied
by all this "hardcoded" stuff. But, is there another way ? We have to
output something :)

I'm not yet entirely familiar with the mod_mbox code, but I have to
say that I find the parts I've seen a bit messy. It maybe because I'm
not used to ASF coding style and it messes my overall understanding of
the code.

> * General Design.  Currently, mod_mbox is tied to a single .mbox
> file, or optionally, a directory of .mbox files.  As we add
> searching and improve the interface, we will want the ability to
> search all mailing lists, a single list, or maybe all lists under a
> domain.  This means mod_mbox needs a knowledge of what other lists
> exist.  This is really about turning mod_mbox into a 'Mail Archive
> Application' vs just something that runs off of mbox files.  There
> are advantages and disadvantages to both directions.  For the ASF's
> use cases, I believe making it more of a unified application is
> good.

This is an interesting point :) As you said, actually mod_mbox serves
only one mailing list archives. If we want it to handle mutliple
lists, we are not talking about design, but redesign. All the actual
mod_mbox code would only be a small part of this new mod_mbox.

The question that must be asked is : does a so complex application
should be an Apache module ? Imho, actual mod_mbox as an module is
great because it's very fast, but I don't believe C is the best
language for the large application you are talking about. Maybe
something could mix both together ?

Well, these are my first thaughts about mod_mbox, I think I'll have
more to say in next few days while ideas come up.

Comments and ideas are welcome, of course !

Regards,
- Sam
-- 
/ Maxime Petazzoni - <ma...@bulix.org> - bulix.org |
| Zwe (zwe.bulix.org) - Gobelins : http://gobelins.nekeme.net |
| Gpg Id: 0x83E6AE0D  - Jabber: sam@jabber.dk ________________/

Re: TODO for mod_mbox

[Maxime Petazzoni <ma...@bulix.org>]

Hi,

> checking for Lucene4c library in no... no
> configure: *** Lucene4c library not found.
> checking for Apache 2.0 version >= 2.0.40... yes
> configure: creating ./config.status
> config.status: creating Makefile
> config.status: error: cannot find input file: Makefile.in

For God sake, read ! The absence of Lucene4c is not fatal ! The
problem comes from Makefile.in.

Here's how I compiled it :

% cd mod_mbox
% ./autogen.sh
% ./configure --prefix=/home/users/sam/.local \
   --without-lucene4c                         \
   --with-apxs=/usr/bin/apxs2
% make


- Sam

PS: could you send answers only to the list, unless it is really
private ? I hate receiving messages twice. Thanks!

-- 
/ Maxime Petazzoni - <ma...@bulix.org> - bulix.org |
| Zwe (zwe.bulix.org) - Gobelins : http://gobelins.nekeme.net |
| Gpg Id: 0x83E6AE0D  - Jabber: sam@jabber.dk ________________/

Re: TODO for mod_mbox

[Florian Heinemann <fl...@fheinemann.de>]

Maxime Petazzoni wrote:
> Hi,
> 
> 
>>Btw: I'm trying to compile lucene (GCJ version) for mod_mbox but he
>>misses the <org/apache/lucene/document/Field.h> file. Can somebody
>>tell me where I can get this file or which package I forgot?
> 
> 
> Unless you are working on the search project, Lucene is no longer
> needed to compile mod_mbox. Just use --without-lucene on the
> ./configure line.
> 
> - Sam

Neither --without-lucene nor --without-lucene4c works.

checking for Lucene4c library in no... no
configure: *** Lucene4c library not found.
checking for Apache 2.0 version >= 2.0.40... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: error: cannot find input file: Makefile.in

When I try to compile the Lucene-Lib I get the old errror, that he can't
find the <org/apache/lucene/document/Field.h> file. Where does this file
come from? I can't find it in any repositories(?)

It would be nice, if someone could help me with one of this problems :-)

- Florian -

Re: TODO for mod_mbox

[Maxime Petazzoni <ma...@bulix.org>]

Hi,

> Btw: I'm trying to compile lucene (GCJ version) for mod_mbox but he
> misses the <org/apache/lucene/document/Field.h> file. Can somebody
> tell me where I can get this file or which package I forgot?

Unless you are working on the search project, Lucene is no longer
needed to compile mod_mbox. Just use --without-lucene on the
./configure line.

- Sam
-- 
/ Maxime Petazzoni - <ma...@bulix.org> - bulix.org |
| Zwe (zwe.bulix.org) - Gobelins : http://gobelins.nekeme.net |
| Gpg Id: 0x83E6AE0D  - Jabber: sam@jabber.dk ________________/

Re: TODO for mod_mbox

[Rolando Abarca <ma...@rolando.cl>]

On Fri, Jun 03, 2005 at 02:13:25AM +0200, Florian Heinemann wrote:
> 
> Good idea, but do you really think that this works in practice? I 
> estimate that more than a half of the mails will be put into the default 
> categorie, because the people who don't use the web-interface won't 
> think about that categories. Or don't I get it?

of course that more than a half of the mails will be on the default
category, but the idea is that to publish content on the web *and* send
it to the mailing list send a special subject.
You have to warn the mailing list users to think before posting if they
want to put their messages on the web.
Even more, the "default" category doesn't need to be be on the web at
all...
Thinking it now, I see that it's too far from the original mod_mbox
intention, and maybe it's not bad, but perhaps it should be a separate
module...

> But generally I think the idea of some filter is extendable.

of course.

-- 
Rolando Abarca M. [rabarca.arroba@dcc.punto.uchile.punto.cl]

Re: TODO for mod_mbox

[Florian Heinemann <fl...@fheinemann.de>]

Hey, I'm one of the people who applied for this SoC project :-)

Rolando Abarca wrote:
> - I belong to a MUG (Macintosh User Group) and every time we wanted to
> create a web page it failed, mainly because the power of the user group
> was in the mailing list. So, the ideal website would've been a web based
> email archive, that could sort the emails in different categories based
> on some pattern in the subject, for example, all emails whose subject
> matches the r.e. '\[AAA\].*' should fit the category named 'AAA'. Of
> course, the mailing manager can create different categories, and a
> default category, where all emails whose subject doesn't match any, will
> be put.

Good idea, but do you really think that this works in practice? I 
estimate that more than a half of the mails will be put into the default 
categorie, because the people who don't use the web-interface won't 
think about that categories. Or don't I get it?

But generally I think the idea of some filter is extendable.

Paul Querna wrote:
> * Interface.  This is a SoC project.  We need some thoughts on if we
> want to use some kind of Theme system (eg, ClearSilver or XSLT) or stick
> with hard coded HTML.  I think a massive addition of CSS is a given, but
> the details are still up in the air.

XSLT is a good solution, because we need no additional module, it is a 
widely-used standard and simple to use. But what will we do with the 
people who haven't a XML-ready browser in combination with a server 
which hasn't a XML parser?

Btw: I'm trying to compile lucene (GCJ version) for mod_mbox but he 
misses the <org/apache/lucene/document/Field.h> file. Can somebody tell 
me where I can get this file or which package I forgot?


Florian

Re: TODO for mod_mbox

[Rolando Abarca <ma...@rolando.cl>]

Sorry to break into the thread like this...
I wanted to apply for this project in the SoC, but other project took my
attention.
Anyway, the assignment of the other project is not guaranteed.

On Thu, Jun 02, 2005 at 11:35:09AM -0700, Paul Querna wrote:
> Other ideas?

- maildir support
- I belong to a MUG (Macintosh User Group) and every time we wanted to
create a web page it failed, mainly because the power of the user group
was in the mailing list. So, the ideal website would've been a web based
email archive, that could sort the emails in different categories based
on some pattern in the subject, for example, all emails whose subject
matches the r.e. '\[AAA\].*' should fit the category named 'AAA'. Of
course, the mailing manager can create different categories, and a
default category, where all emails whose subject doesn't match any, will
be put.

> -Paul

If I don't get the other project, I might work on the idea above by my
own, and I wanted to know if that idea is too far from the main
objectives of the mod_mbox...
-- 
Rolando Abarca M. [rabarca.arroba@dcc.punto.uchile.punto.cl]
 "Tam pro papa quam pro rege bibunt omnes sine lege."