mod_smtpd project planning

[Paul Querna <ch...@force-elite.com>]

As some of you might be aware, one of the Summer of Code Projects is an 
SMTP protocol module for httpd 2.x.

We don't have much code yet, but we need a place to put it, and for the 
SoC Students to have commit bits too.

I suggest that we treat mod_smtpd like mod_python, as an httpd 
subproject.  If it later grows into its own TLP/PMC, thats great, if 
not, from a developer community perspective, I believe it fits best 
inside httpd.

So, to accommodate this, we would need a mod_smtpd space in subversion 
and a smtpd-dev@httpd.apache.org mailing list.

Thoughts, comments, alternatives?

-Paul

Re: mod_smtpd project planning

[Paul A Houle <ph...@cornell.edu>]

Paul Querna wrote:

> As some of you might be aware, one of the Summer of Code Projects is 
> an SMTP protocol module for httpd 2.x.
>
    Huh?

Re: mod_smtpd project planning

[Colm MacCarthaigh <co...@stdlib.net>]

On Fri, Jul 01, 2005 at 02:10:19PM +0100, Nick Kew wrote:
> Apart from what others (particularly Rian) have said, it'll want
> an additional ap_hook_smtp_envelope where things like a mod_rbl
> can reject things before DATA.

Take a look at the stages at which exim can ACL. It's not uncommon to
use them all. It should be possible to ACL after every single SMTP
command, incorporating all input until that point, and even beyond this
- for example many people find it useful for the receiver to do MIME
  decode and run an ACL per attachment.

Tony Finch is both a (though inactive) httpd and (active) exim committer, 
he's probably good person to bounce some of this off of.

-- 
Colm MacCárthaigh                        Public Key: colm+pgp@stdlib.net

Re: mod_smtpd project planning

[Nick Kew <ni...@webthing.com>]

Paul A Houle wrote:

>    You read this?
> 
> http://www.acme.com/mail_filtering/
> 
>    One thing that's critical isn't just having access to information
> from early stages of mail processing,  but being able to intervene at
> early stages in the processing so to avoid the CPU and bandwidth waste
> at advanced stages.

Apart from what others (particularly Rian) have said, it'll want
an additional ap_hook_smtp_envelope where things like a mod_rbl
can reject things before DATA.

-- 
Nick Kew

Re: mod_smtpd project planning

[Paul A Houle <ph...@cornell.edu>]

Jem Berkes wrote:

>This is the problem encountered by many spam filters, as to be most 
>effective they really need to be _involved_ in the SMTP transaction and not 
>just stage 2, after receipt happens. Think greylisting as an example.
>
>  
>
    You read this?

http://www.acme.com/mail_filtering/

    One thing that's critical isn't just having access to information 
from early stages of mail processing,  but being able to intervene at 
early stages in the processing so to avoid the CPU and bandwidth waste 
at advanced stages.  This particularly matters during a computer virus 
outbreak:  I remember hitting on many of Jeff's solutions when a mail 
server I managed was getting hammered by an incredible volume of 
viruses,  and I wrote scripts that picked up bad addresses from the 
virus filter output and put them into the software firewall.
 

Re: mod_smtpd project planning

[Jem Berkes <jb...@users.pc9.org>]

> It's a SMTP protocol frontend for httpd. It will have the power to be a
> sendmail replacer or to supply content via SMTP because it will delegate
> most of the actual handling to other modules. All the details haven't
> been worked out yet, but it will make use of the Apache 2.x filters and
> handlers. For Instance:
> 
> [core] -> [mod_smtpd] -> [mod_insert_special_use_mail_handler]
> a setup like that could be used, but let's say you want to filter out
> junk mail. Use an input filter!
> 
> [core] -> [mod_smtpd] -> [mod_junk_mail_filter] -> [mod_other_thing]

Just to give an idea of the added flexibility of SMTP support within an 
httpd module; if we were just using pipes to/from sendmail then you have 
very limited information - basically just the mail data after receipt and 
no ability to talk back to the peer during the mail transaction.

This is the problem encountered by many spam filters, as to be most 
effective they really need to be _involved_ in the SMTP transaction and not 
just stage 2, after receipt happens. Think greylisting as an example.

Also, since mod_smtpd will receive emails from MTAs itself, and prepare its 
own data structures for passing on the data to other modules, this means 
that it can pass along useful information that are difficult or impossible 
to determine from just message headers/body. For example, IP address of the 
incoming or outgoing relay, helo/intro identification of peer, protocol 
violations or warnings, connection data rate perhaps... brainstorming.

Re: mod_smtpd project planning

[Rian Hunter <ri...@MIT.EDU>]

Paul A Houle wrote:
> Jem Berkes wrote:
> 
>> Hi all, I'm another student working on mod_smtpd
>>
>> Been running httpd 2.x since it appeared, but am new to development.
>>  
>>
>    What does mod_smtpd do?  Is it a sendmail replacer or does it let 
> people request content via smtp or what?

It's a SMTP protocol frontend for httpd. It will have the power to be a 
sendmail replacer or to supply content via SMTP because it will delegate 
most of the actual handling to other modules. All the details haven't 
been worked out yet, but it will make use of the Apache 2.x filters and 
handlers. For Instance:

[core] -> [mod_smtpd] -> [mod_insert_special_use_mail_handler]

a setup like that could be used, but let's say you want to filter out 
junk mail. Use an input filter!

[core] -> [mod_smtpd] -> [mod_junk_mail_filter] -> [mod_other_thing]

No rewrite! The handler can be anything and the possibilities are 
endless (just like in httpd). As a proof of concept we will probably 
write or reuse a basic delivery module.

Another upside to this is that lets say you want to implement a new 
proprietary feature for SMTP (like a content supplier). Your modules 
just has to register which extensions it supports (for a response to the 
ehlo command), and you don't have to worry about the details of writing 
a new SMTP server when most of the tedious work has already been taken 
care of.
-rian

Re: mod_smtpd project planning

[Paul A Houle <ph...@cornell.edu>]

Jem Berkes wrote:

>Hi all, I'm another student working on mod_smtpd
>
>Been running httpd 2.x since it appeared, but am new to development.
>  
>
    What does mod_smtpd do?  Is it a sendmail replacer or does it let 
people request content via smtp or what?

Re: mod_smtpd project planning

[Jem Berkes <jb...@users.pc9.org>]

> As one of the students I can definitely appreciate that!
> 
> To everyone managing SoC: about how long until our svn accounts are
> activated? I know there are a lot details being worked out still, but I
> still feel a little in the dark.

Hi all, I'm another student working on mod_smtpd

Been running httpd 2.x since it appeared, but am new to development.

- Jem Berkes

Re: mod_smtpd project planning

[rian <ri...@MIT.EDU>]

Roy T. Fielding wrote:
> This isn't the first time that new people working on httpd-related
> projects have been given limited commit access.  I don't give a
> rat's ass whether they finish their assigned projects or not.
> What I want them to do is become part of our community long enough
> to figure out how fun it can be to be part of a smart group of
> people working towards a greater goal.
> 
> The dev list needs some new blood (and new ideas) and these folks
> aren't going to learn anything about real Apache development if
> they are stuck out in the suburbs.

As one of the students I can definitely appreciate that!

To everyone managing SoC: about how long until our svn accounts are 
activated? I know there are a lot details being worked out still, but I 
still feel a little in the dark.
-rian

Re: mod_smtpd project planning

[Justin Erenkrantz <ju...@erenkrantz.com>]

On Wed, Jun 29, 2005 at 02:39:43PM -0700, Roy Fielding wrote:
> The dev list needs some new blood (and new ideas) and these folks
> aren't going to learn anything about real Apache development if
> they are stuck out in the suburbs.

++1.  -- justin

Re: mod_smtpd project planning

["Roy T. Fielding" <fi...@gbiv.com>]

On Jun 29, 2005, at 6:09 AM, Ian Holsman wrote:

> and where do we put the code?

Wherever you want the code to be, so long as it isn't released
until you get three binding +1s.

This isn't the first time that new people working on httpd-related
projects have been given limited commit access.  I don't give a
rat's ass whether they finish their assigned projects or not.
What I want them to do is become part of our community long enough
to figure out how fun it can be to be part of a smart group of
people working towards a greater goal.

The dev list needs some new blood (and new ideas) and these folks
aren't going to learn anything about real Apache development if
they are stuck out in the suburbs.

....Roy

Re: mod_smtpd project planning

[Ian Holsman <li...@holsman.net>]

Roy T. Fielding wrote:
> On Jun 28, 2005, at 4:20 PM, Paul Querna wrote:
> 
>> So, to accommodate this, we would need a mod_smtpd space in subversion 
>> and a smtpd-dev@httpd.apache.org mailing list.
> 
> 
> -1.  The point of SoC is to get more people involved in open
> source projects, not to encourage the ASF to accept
> non-collaborative code bases in exchange for a bit of cash.
> Use the main dev list and encourage the students to participate
> in the larger project as well as the specific one.
> 
> ....Roy
> 
and where do we put the code?

we've got another SoC project in the same spot.
httpd-bandwidth-limit. (one to limit DoS/bandwidth hogs)


Roy.. are you recommending we put both modules in the experimental area?

Re: mod_smtpd project planning

[Paul Querna <ch...@force-elite.com>]

Roy T. Fielding wrote:

> On Jun 28, 2005, at 4:20 PM, Paul Querna wrote:
>
>> So, to accommodate this, we would need a mod_smtpd space in 
>> subversion and a smtpd-dev@httpd.apache.org mailing list.
>
>
> -1.  The point of SoC is to get more people involved in open
> source projects, not to encourage the ASF to accept
> non-collaborative code bases in exchange for a bit of cash.
> Use the main dev list and encourage the students to participate
> in the larger project as well as the specific one.

Oh, I agree about the point of SoC, but I didn't think the main dev list 
would be interested in the details of SMTP.  If people don't mind, then 
using the main dev@httpd list is fine by me.

-Paul

Re: mod_smtpd project planning

["Roy T. Fielding" <fi...@gbiv.com>]

On Jun 28, 2005, at 4:20 PM, Paul Querna wrote:

> So, to accommodate this, we would need a mod_smtpd space in subversion 
> and a smtpd-dev@httpd.apache.org mailing list.

-1.  The point of SoC is to get more people involved in open
source projects, not to encourage the ASF to accept
non-collaborative code bases in exchange for a bit of cash.
Use the main dev list and encourage the students to participate
in the larger project as well as the specific one.

....Roy