Mailets and dependencies (Was: [PLANNING] Road map)

[Stefano Bagnara <ap...@bago.org>]

Robert Burrell Donkin ha scritto:
> i agree in principle but trunk has a lot of JAMES-specific mailets
> (but i suspect that many of these could be decoupled)

I would say "some" instead of "many" ;-)

> the problem is that many common problems solved by mailets require
> access to basic services which are environment specific (for example,
> delivery to a logical mailbox or access to user information). ATM
> mailets are coupled to basic service APIs defined by JAMES. IMHO it
> would be better if mailets were decoupled by exposing their own API
> interfaces rather than using JAMES service APIs.

During 2004-2005 (before I joined this project) some of the repository 
api landed the mailet api for what was called mailet api v3. Both 
usersrepository and mailrepository were exposed in that version.
I don't know exactly what happened because when I joined the project it 
was no more there. I think the main cause was that adding repositories 
to the mailet apis would have made more complicate to create mailet 
containers. And probably repositories api was not mature enough, also.

 > i think that it would be a good plan to pull out those mailets which
 > are conceptually independent into separate a subproject (standard
 > mailets, say)

I did some homework...
here is a list of matcher/mailets that (I think) do not depends on james 
or avalon:

matchers/FetchedFrom
matchers/RecipientIsRegex
matchers/RecipientIs
matchers/CommandListservMatcher
matchers/SMTPAuthSuccessful
matchers/SenderIsRegex
matchers/AbstractQuotaMatcher
matchers/SMTPIsAuthNetwork
matchers/HasAttachment
matchers/SenderIs
matchers/CommandForListserv
matchers/HasMailAttributeWithValueRegex
matchers/RecipientIsLocal
matchers/CompareNumericHeaderValue
matchers/SubjectStartsWith
matchers/AttachmentFileNameIs
matchers/All
matchers/FileRegexMatcher
matchers/SizeGreaterThan
matchers/GenericRegexMatcher
matchers/SenderIsNull
matchers/UserIs
matchers/HostIs
matchers/HasMailAttribute
matchers/HostIsLocal
matchers/SMTPAuthUserIs
matchers/SubjectIs
matchers/RelayLimit
matchers/HasHeader
matchers/IsSingleRecipient
matchers/HasHabeasWarrantMark
matchers/SenderHostIsLocal
matchers/HasMailAttributeWithValue
matchers/SenderHostIs
matchers/NESSpamCheck
matchers/smime/IsX509CertificateSubject
matchers/smime/IsSMIMESigned
matchers/smime/IsSMIMEEncrypted
mailets/MailAttributesToMimeHeaders
mailets/RemoveMimeHeader
mailets/debug/Counter
mailets/debug/ExceptionThrowingMailet
mailets/debug/DumpSystemErr
mailets/debug/Identity
mailets/OnlyText
mailets/GenericListservManager
mailets/AddFooter
mailets/RemoveAllMailAttributes
mailets/UseHeaderRecipients
mailets/SetMimeHeader
mailets/ServerTime
mailets/Null
mailets/ToProcessor
mailets/RemoveMailAttribute
mailets/PostmasterAlias
mailets/SetMailAttribute
mailets/AbstractAddFooter
mailets/AddSubjectPrefix
mailets/AddHabeasWarrantMark

-----

Then we have some mailets with dependencies on utility classes that 
could be separated from James:

mailets/ReplaceContent
- import org.apache.james.util.mailet.StringUtils;

mailets/ReplaceContent
- depends on org.apache.james.util.mailet.MailetUtil

mailets/UnwrapText
mailets/WrapText
- depend on org.apache.james.util.mailet.FlowedMessageUtils

mailets/smime/SMIMECheckSignature
mailets/smime/Sign
mailets/smime/AbstractSign
mailets/smime/SMIMESign
mailets/smime/SMIMEDecrypt
- smime stuff. depends on org.apache.james.security

mailets/ClamAVScan
- import org.apache.james.util.io.IOUtil;

mailets/SpamAssassin
- import org.apache.james.util.SpamAssassinInvoker;

mailets/LogMessage
- import org.apache.james.core.MailImpl but I think this might be a 
mistake and we could replace MailImpl with Mail.

------------

Then we have the jspf mailet that simply depends on the jspf library:
mailets/SPF

---------

And we have an important set of mailets that currently depends on some 
James core classes (Constants, core.MailImpl, core.MimeMessageUtil) and 
it would be important to make them agnostic from JAMES:

mailets/AbstractRedirect
mailets/NotifyPostmaster
mailets/Forward
mailets/Bounce
mailets/Resend
mailets/NotifySender
mailets/AbstractNotify
mailets/Redirect
mailets/DSNBounce (also depends on util.mail.dsn.DSNStatus and 
util.mail.MimeMultipartReport)

The dependency on MimeMessageUtil is in AbastractRedirect (every other 
mailet extends this one) and is only because of 
MimeMessageUtil.writeMessageBodyTo that is James agnostic and could be 
moved to an util class with no heavy dependency (MimeMessageUtil depends 
on core James classes like MimeMessageWrapper).

---------

Then we have a bunch of mailets depending on AvalonServiceManager and 
the james DNSService because they need some dns lookup service:

matchers/RemoteAddrInNetwork
matchers/RemoteAddrNotInNetwork
matchers/AbstractNetworkMatcher
matchers/SenderInFakeDomain
matchers/InSpammerBlacklist

They use: DNSServer.getByName(String host);
They also depends on the o.a.james.util.NetMatcher that simply depends 
on the DNSServer for the getByName(String host) service).

It seems to me that adding a getByName to the MailetAPI we could make 
all of this matchers generic.

----------

Then we have mailets depending on avalon configuration, on james 
UsersRepositories and MailRepositories and more. But I think the above 
list should be easier to take care as a first step.

-------------

Most of the listed stuff (excluding the last 2 groups) could be moved to 
its own module along with some utility dependencies.
Should it be a function or a library? How do we package the utilities 
that are used both by this mailets and by the core-library module?

Stefano


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Mailets and dependencies (Was: [PLANNING] Road map)

[Robert Burrell Donkin <ro...@gmail.com>]

On Feb 3, 2008 12:03 AM, Danny Angus <da...@apache.org> wrote:
> I did the "v3" stuff and the sandbox stuff isn't very different, look
> at it, I can do it all again.
> The problems are really only...
>
> 1/ some of the interfaces that you need to move to mailet aren't
> totally normalised, IOW they are a bit hacked to make them work in
> James not the way you would design them from scratch,

probably better not to move the interfaces but to create new ones in
new packages

> and
> 2/ RemoteDelivery is a total hack right now

+1

> and needs a) a complete
> rethink and b) a fully featured implementation of javamail with
> callbacks (to Listeners messages) from events that occur after they
> have been queued, possibly upto to several days and several restarts
> after they enter the outgoing queue.

looks nasty and seems a bit beyond me at first sight but i'll take a
stab anyway. maybe this function would be better pushed into the
mailet engine (eg deliver("smtp://user@mail.example.org") using the
URL proposal). a pluggable extension point for SMTP transport would
allow an alternative commons mail based implementation.

but there's no need to move everything all at once. stefano has
identified contents for a viable decoupled standard mailet sub-project
which can be started right now. the security SMIME stuff is also
viable but would work best as a separate sub-project (the PGP/MIME
mailets from the branch would be merged in). should be easier to fix
the corner cases once we've moved the easy stuff out of trunk.

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Mailets and dependencies (Was: [PLANNING] Road map)

[Danny Angus <da...@apache.org>]

I did the "v3" stuff and the sandbox stuff isn't very different, look
at it, I can do it all again.
The problems are really only...

1/ some of the interfaces that you need to move to mailet aren't
totally normalised, IOW they are a bit hacked to make them work in
James not the way you would design them from scratch,
and
2/ RemoteDelivery is a total hack right now and needs a) a complete
rethink and b) a fully featured implementation of javamail with
callbacks (to Listeners messages) from events that occur after they
have been queued, possibly upto to several days and several restarts
after they enter the outgoing queue.

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Mailets and dependencies (Was: [PLANNING] Road map)

[Robert Burrell Donkin <ro...@gmail.com>]

On Feb 2, 2008 4:51 PM, Stefano Bagnara <ap...@bago.org> wrote:
> Robert Burrell Donkin ha scritto:
> > i agree in principle but trunk has a lot of JAMES-specific mailets
> > (but i suspect that many of these could be decoupled)
>
> I would say "some" instead of "many" ;-)

:-P

> > the problem is that many common problems solved by mailets require
> > access to basic services which are environment specific (for example,
> > delivery to a logical mailbox or access to user information). ATM
> > mailets are coupled to basic service APIs defined by JAMES. IMHO it
> > would be better if mailets were decoupled by exposing their own API
> > interfaces rather than using JAMES service APIs.
>
> During 2004-2005 (before I joined this project) some of the repository
> api landed the mailet api for what was called mailet api v3. Both
> usersrepository and mailrepository were exposed in that version.
> I don't know exactly what happened because when I joined the project it
> was no more there. I think the main cause was that adding repositories
> to the mailet apis would have made more complicate to create mailet
> containers. And probably repositories api was not mature enough, also.
>
>  > i think that it would be a good plan to pull out those mailets which
>  > are conceptually independent into separate a subproject (standard
>  > mailets, say)
>
> I did some homework...
> here is a list of matcher/mailets that (I think) do not depends on james
> or avalon:
>
> matchers/FetchedFrom
> matchers/RecipientIsRegex
> matchers/RecipientIs
> matchers/CommandListservMatcher
> matchers/SMTPAuthSuccessful
> matchers/SenderIsRegex
> matchers/AbstractQuotaMatcher
> matchers/SMTPIsAuthNetwork
> matchers/HasAttachment
> matchers/SenderIs
> matchers/CommandForListserv
> matchers/HasMailAttributeWithValueRegex
> matchers/RecipientIsLocal
> matchers/CompareNumericHeaderValue
> matchers/SubjectStartsWith
> matchers/AttachmentFileNameIs
> matchers/All
> matchers/FileRegexMatcher
> matchers/SizeGreaterThan
> matchers/GenericRegexMatcher
> matchers/SenderIsNull
> matchers/UserIs
> matchers/HostIs
> matchers/HasMailAttribute
> matchers/HostIsLocal
> matchers/SMTPAuthUserIs
> matchers/SubjectIs
> matchers/RelayLimit
> matchers/HasHeader
> matchers/IsSingleRecipient
> matchers/HasHabeasWarrantMark
> matchers/SenderHostIsLocal
> matchers/HasMailAttributeWithValue
> matchers/SenderHostIs
> matchers/NESSpamCheck
> matchers/smime/IsX509CertificateSubject
> matchers/smime/IsSMIMESigned
> matchers/smime/IsSMIMEEncrypted
> mailets/MailAttributesToMimeHeaders
> mailets/RemoveMimeHeader
> mailets/debug/Counter
> mailets/debug/ExceptionThrowingMailet
> mailets/debug/DumpSystemErr
> mailets/debug/Identity
> mailets/OnlyText
> mailets/GenericListservManager
> mailets/AddFooter
> mailets/RemoveAllMailAttributes
> mailets/UseHeaderRecipients
> mailets/SetMimeHeader
> mailets/ServerTime
> mailets/Null
> mailets/ToProcessor
> mailets/RemoveMailAttribute
> mailets/PostmasterAlias
> mailets/SetMailAttribute
> mailets/AbstractAddFooter
> mailets/AddSubjectPrefix
> mailets/AddHabeasWarrantMark

that'd be a decent start for a standard mailet sub-project

> -----
>
> Then we have some mailets with dependencies on utility classes that
> could be separated from James:
>
> mailets/ReplaceContent
> - import org.apache.james.util.mailet.StringUtils;
>
> mailets/ReplaceContent
> - depends on org.apache.james.util.mailet.MailetUtil
>
> mailets/UnwrapText
> mailets/WrapText
> - depend on org.apache.james.util.mailet.FlowedMessageUtils

yes, this code should be moved into standard mailets

> mailets/smime/SMIMECheckSignature
> mailets/smime/Sign
> mailets/smime/AbstractSign
> mailets/smime/SMIMESign
> mailets/smime/SMIMEDecrypt
> - smime stuff. depends on org.apache.james.security

as you noted, this code probably belongs with the mailets

> mailets/ClamAVScan
> - import org.apache.james.util.io.IOUtil;
>
> mailets/SpamAssassin
> - import org.apache.james.util.SpamAssassinInvoker;
>
> mailets/LogMessage
> - import org.apache.james.core.MailImpl but I think this might be a
> mistake and we could replace MailImpl with Mail.

+1

> ------------
>
> Then we have the jspf mailet that simply depends on the jspf library:
> mailets/SPF

separate sub-project?

> ---------
>
> And we have an important set of mailets that currently depends on some
> James core classes (Constants, core.MailImpl, core.MimeMessageUtil) and
> it would be important to make them agnostic from JAMES:
>
> mailets/AbstractRedirect
> mailets/NotifyPostmaster
> mailets/Forward
> mailets/Bounce
> mailets/Resend
> mailets/NotifySender
> mailets/AbstractNotify
> mailets/Redirect
> mailets/DSNBounce (also depends on util.mail.dsn.DSNStatus and
> util.mail.MimeMultipartReport)
>
> The dependency on MimeMessageUtil is in AbastractRedirect (every other
> mailet extends this one) and is only because of
> MimeMessageUtil.writeMessageBodyTo that is James agnostic and could be
> moved to an util class with no heavy dependency (MimeMessageUtil depends
> on core James classes like MimeMessageWrapper).

+1

> ---------
>
> Then we have a bunch of mailets depending on AvalonServiceManager and
> the james DNSService because they need some dns lookup service:
>
> matchers/RemoteAddrInNetwork
> matchers/RemoteAddrNotInNetwork
> matchers/AbstractNetworkMatcher
> matchers/SenderInFakeDomain
> matchers/InSpammerBlacklist
>
> They use: DNSServer.getByName(String host);
> They also depends on the o.a.james.util.NetMatcher that simply depends
> on the DNSServer for the getByName(String host) service).
>
> It seems to me that adding a getByName to the MailetAPI we could make
> all of this matchers generic.


sounds reasonable to me

> ----------
>
> Then we have mailets depending on avalon configuration, on james
> UsersRepositories and MailRepositories and more. But I think the above
> list should be easier to take care as a first step.

+1

> -------------
>
> Most of the listed stuff (excluding the last 2 groups) could be moved to
> its own module along with some utility dependencies.
> Should it be a function or a library? How do we package the utilities
> that are used both by this mailets and by the core-library module?

i'd prefer to move all that stuff out of server and into separate
sub-projects with their own release cycles in sync with the mailet API

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org