Using a matcher before message delivery?

[Lyubomir Dokov <l....@efficient-soft.com>]

Hi.

Actually, this is both malet API and fetchmail question.

The case:
I am using fetchmail, but there is a problem with the SEEN flag - it is 
not permanent, thus, resulting in all mails fetched each time the fetch 
job starts. I've read about that in the mail list archives but the only 
solution I found was to delete the messages from server. However, this 
is not a workaround for me - to leave messages on server is a valid 
option for my users. (Is there any other workaround which I don't know of?).
What I thought is to implement a matcher/mailet which intercepts the 
incoming mails fetched by fetchmail and decides whether to discard them 
or not. I have written the matcher and it works OK(parses the JAMES 
configuration and checks for the fetch job names in the attributes of 
intercepted messages). The only thing I have to implement is to have 
some way to understand whether the message has been already delivered or 
not(The only thing I could thing of now is to implement some db storage 
with the recieved message IDs - any ideas appreciated).

The problem:
This is something that could ruin my whole approach to the problem: What 
happens if I have a 10MB(or 100MB) message in my INBOX? Is it downloaded 
by the fetcmail job and then given to the mailet chain? If so, I will 
download hundreds of megabytes just to discard them from the 
mailet/matcher. That would be stupid. Is this the way it works, or is 
there some place where I could put mailets which are invoked with just 
the headers of the message(I know this is possible in the mail protocols 
- to get the headers only). Or, is my whole understanding of how the 
things work wrong?

Even if my approach to solving the fetchmail problem is wrong, the case 
where you need to have mailets to match headers only is still valid.

I appreciate any help. Thank you.

Re: Using a matcher before message delivery?

[Danny Angus <da...@apache.org>]

On 9/21/07, Robert Burrell Donkin <ro...@gmail.com> wrote:

> IMHO this is really an implementation issue. AIUI the API is agnostic
> about whether a message should be loaded lazily or preemptively.
>
> if people disagree then hopefully they'll jump in...

I agree. but in any case if it did specific anything it would be more
likely to favour lazy loading,  there's no reason why a lot of use
cases couldn't be completely satisfied without ever looking at the
message itself.

d.

Re: Using a matcher before message delivery?

[Robert Burrell Donkin <ro...@gmail.com>]

On 9/21/07, Lyubomir Dokov <l....@efficient-soft.com> wrote:

<snip>

> P.S.
> I continued the topic here and not in the server mailing lists because I
> still think it is a matter of how mailets are processed, which in turn
> may affect the mailet API. Do you agree on that?

IMHO this is really an implementation issue. AIUI the API is agnostic
about whether a message should be loaded lazily or preemptively.

if people disagree then hopefully they'll jump in...

- robert

Re: Using a matcher before message delivery?

[Robert Burrell Donkin <ro...@gmail.com>]

On 9/21/07, Lyubomir Dokov <l....@efficient-soft.com> wrote:
> Robert, thanks for your reply.
>
> I would be happy to help and contribute. However, I need to look through
> the code first and see how much time would it take - I have deadlines to
> catch in my project.

of course

> On the question:
> > the approach sounds reasonable to me. it could be made to work by
> > modifying fetchmail so that the message is lazily loaded.
> No matter how fetchmail works, I think that having a way to process
> headers only is a good idea for JAMES as a whole. Meaning, not only
> messages fetched by fetchmail, but also recieved the ordinary way - when
> they are sent to JAMES. Does that make sense? I was thinking about
> adding a separate processor of some kind that would be invoked with the
> message headers only(meaning malets and matchers have access only to
> headers, not the whole MimeMessage). Do you think that is possible and
> good solution? I am working with JAMES for 3-4 days now and I am still
> new. Thats why I am so unsure of myself.

i feel that MailMessage is a little heavyweight and that a lighter
abstraction may be better but this is a minority opinion. providing
that the MailMessage implementation is lazy loaded (in other words,
the headers and the body are loaded only when needed) then there is
really no need for a separate abstraction to achieve what you need.

> > ATM fetchmail uses javamail so this usually depends on sun's
> > implementation. it is possible (though unlikely) that the body will
> > not be downloaded until it's needed. sniffing the protocol exchange
> > should tell you whether it does.
> Today, I didn't had much time to investigate, but as far as I remember
> there was sample mailets adding text to the body of the message, thus,
> meaning the whole message is downloaded and then given to the mailet
> chain. However, I will look more when I have the time.

the question is whether sun's MimeMessage implementation for POP3 is
lazy loading (and so will only load the body when a method call
requires it).  if the MimeMessage implementation is lazy loading then
the content will not be downloaded until needed.

if sun's implementation is not lazy loading then it may be necessary
to create a new implementation of MimeMessage based on commons net
which is.

- robert

Re: Using a matcher before message delivery?

[Lyubomir Dokov <l....@efficient-soft.com>]

Robert, thanks for your reply.

I would be happy to help and contribute. However, I need to look through 
the code first and see how much time would it take - I have deadlines to 
catch in my project.

On the question:
> the approach sounds reasonable to me. it could be made to work by
> modifying fetchmail so that the message is lazily loaded.
No matter how fetchmail works, I think that having a way to process 
headers only is a good idea for JAMES as a whole. Meaning, not only 
messages fetched by fetchmail, but also recieved the ordinary way - when 
they are sent to JAMES. Does that make sense? I was thinking about 
adding a separate processor of some kind that would be invoked with the 
message headers only(meaning malets and matchers have access only to 
headers, not the whole MimeMessage). Do you think that is possible and 
good solution? I am working with JAMES for 3-4 days now and I am still 
new. Thats why I am so unsure of myself.

> ATM fetchmail uses javamail so this usually depends on sun's
> implementation. it is possible (though unlikely) that the body will
> not be downloaded until it's needed. sniffing the protocol exchange
> should tell you whether it does.
Today, I didn't had much time to investigate, but as far as I remember 
there was sample mailets adding text to the body of the message, thus, 
meaning the whole message is downloaded and then given to the mailet 
chain. However, I will look more when I have the time.

One other thing about fetchmail:
A store for message IDs is needed for solving the SEEN flag problem. I 
was thinking of adding some sql statements to the "sqlResources.xml" and 
use the JAMES internal repository. Creating a new one(Derby or whatever) 
just for storing the already fetched message IDs would be a waste. Do 
you think this is possible? Could you give me some clues on how to 
access the repository?

As I said, I would be glad to help and contribute, but I will definitely 
need some help from the other contributors - otherwise it will take me 
too much time to reinvent all and I just don't have it right now.

Thank you.

P.S.
I continued the topic here and not in the server mailing lists because I 
still think it is a matter of how mailets are processed, which in turn 
may affect the mailet API. Do you agree on that?

Re: Using a matcher before message delivery?

[Robert Burrell Donkin <ro...@gmail.com>]

On 9/20/07, Lyubomir Dokov <l....@efficient-soft.com> wrote:
> Hi.
>
> Actually, this is both malet API and fetchmail question.

(i'm going to approach this as a fetchmail problem. if other see
better solutions using mailets then hopefully they'll jump in.)

> The problem:
> This is something that could ruin my whole approach to the problem: What
> happens if I have a 10MB(or 100MB) message in my INBOX? Is it downloaded
> by the fetcmail job and then given to the mailet chain?

ATM fetchmail uses javamail so this usually depends on sun's
implementation. it is possible (though unlikely) that the body will
not be downloaded until it's needed. sniffing the protocol exchange
should tell you whether it does.

> If so, I will
> download hundreds of megabytes just to discard them from the
> mailet/matcher. That would be stupid. Is this the way it works, or is
> there some place where I could put mailets which are invoked with just
> the headers of the message(I know this is possible in the mail protocols
> - to get the headers only). Or, is my whole understanding of how the
> things work wrong?
>
> Even if my approach to solving the fetchmail problem is wrong, the case
> where you need to have mailets to match headers only is still valid.

the approach sounds reasonable to me. it could be made to work by
modifying fetchmail so that the message is lazily loaded. this will
most likely require some rewriting of the fetchmail module. i'm not
sure if there are any developers with time and enthusiasm to take this
on but if you want to develop it, i'm sure be happy to help out
answering questions and so on.

if rewriting fetchmail is on the agenda, then the most performant
solution would be to address this directly in the fetchmail module.
would probably mean porting (a version of) fetchmail to use commons
net pop3 client (but this would be a good thing anyway).

- robert