You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Aaron Knauf <ak...@xtra.co.nz> on 2002/11/14 03:28:45 UTC
MailRepository
Hi All,
I am new to this list, but not new to JAMES. I have been developing
against JAMES 2.0a1 and I have a few observations regarding the design
of the mail repository system. I understand that things have changed
since 2.0a1 and I would like to try to ensure that future JAMES version
upgrades are as painless for us as possible, as our reliance on JAMES is
becoming deeper. I recently posted on the james-user list and it was
suggested that I join the dev list - so here I am. If my understanding
of JAMES is out of date, or just plain incorrect, please forgive me.
Here goes:
I think the JAMES mail repository needs to be exposed to mailet authors.
At present mailet authors must import packages from Avalon and from
JAMES code that is not in the mailet jar file in order to manipulate
mail repositories. This introduces a dependency on an internal
implementation, which creates problems when that implementation change
in the next release (which is exactly why I am still working with 2.0a1).
Our requirement is this:
We would like to be able to define an arbitrary mail repository, store
incoming mail into it and process that mail asynchronously. (I would
like be quite clear that I have no interest at this point in the
physical implementation of the repository. Whether mail is stored in a
RDBMS or the file system or under the mattress is of no consequence.)
It strikes me that two approaches are available here:
1) Expose the MailRepository interface in the mailet jar file and
provide a standard way to get a reference to to a particular repository.
2) Expose the ability to define mail repositories through config.xml
*and also to process the mail in them*. At present there is the
ToRepository mailet to put mail in a repository. The trouble is that
nothing happens to the mail once it is written to a repository.
A combination of both is the best solution, I believe.
As I understand it this is the way it works right now:
1 - Mail is received by JAMES and written to the spool repository.
2 - The spool manager reads mail from the spool repository and passes it
to a processor, giving the mailets their chance.
3 - A mailet may pass the mail to another processor, or to a repository,
or not.
I am thinking that this could be extended to include the ability to
define a processor that is associated with a named repository. This
processor would have its own spool manager that read mail from the named
repository and passed it to the processor for processing. Or perhaps it
would be better to define the repository as being an active repository
and associating a processor to handle the mail from that repository.
(i.e. repository references processor, not processor references repository.)
If all of this could be done declaratively, through config.xml, that
would be vastly preferable to having to write code to create all of the
necessary repository instances, spool managers, threads, etc.
A major problem with manipulating repositories in code is the
requirement to cast Mail to MailImpl in order to use the repository.
The repository interface should deal with Mail objects. (An internal
cast to MailImpl is also undesirable because I should be able to
substitute one Mail implementation for another - that is what interfaces
are for.)
The ToRepository mailet performs this cast at the the moment. This
(IMHO) demonstrates that the design is not right. Should a servlet have
to cast a HttpRequest to a container-specific HttpRequestImpl? JAMES
mailets should be able to survive with the same mailet API that everyone
else's mailets work with. We need to be able to eat our own dogfood here.
I realise the at the moment all of this repository stuff is considered
internal to JAMES. The point that I wish to make is that I think it
should be exposed. I have tried to suggest a approach to this. Right
now my approach is not very carefully thought out and probably has
plenty of holes in it. Please try to see the spirit of this suggestion,
rather than the holes.
Cheers
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: MailRepository
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Noel J. Bergman wrote:
> Just have the mailet do this when the service method is called. If you need
> something else, just clone what RemoteDelivery does, which is basically the
> same as your mailet. It uses an internal spool when it can't immediately
> deliver. We could probably refactor RemoteDelivery into a base class for a
> certain category of mailet, and then provide methods for performing the
> remote delivery, e.g., to another SMTP server, to your EJB server, to
> another server.
I seem to recall reading through RemoteDelivery at the time we were
writing our mail repository access code. We do esentially do the same
thing in our mailet - and for the same reasons.
Making RemoteDelivery into a generic base class would give us the
independence from the JAMES internals that we are after for our
immediate requirements. However, the next phase of our app requires
storage of all forwarded messages for a minimum of several days,
regardless of transmission success/failure. A subclass of
RemoteDelivery would not give that kind of flexibility, I imagine.
>
>
>>I think the JAMES mail repository needs to be exposed to mailet authors.
>
>
> I agree that repository access needs to be provided, but I do NOT that the
> Repository interface currently present is what ought to be used, either by
> James or by Mailets. I also agree that whatever the interface to
> repositories, James and Mailets ought to use the same one.
Agreed - the actual interface provided should not be MailRepository
without modification. I'm sure you've thought that aspect out far more
thoroughly than I have.
>
>
>>We would like to be able to define an arbitrary mail repository, store
>>incoming mail into it and process that mail asynchronously.
>
>
> See above. It sounds as if you team is going about it all wrong. Please
> read through the RemoteDelivery mailet.
See above - we did read RemoteDelivery. ;-)
>
>
>>1) Expose the MailRepository interface in the mailet jar file
>> and provide a standard way to get a reference to to a
>> particular repository.
>
>
> That depends upon what form the mail repository takes.
????? Why should the 'form the mail repository takes' affect the
interface or the method used to get a reference? If I have two
repositories, one of which stores its mail in a RDBMS while the other
uses the filesystem, why should I not be able to say
MailRepository rep1 = getRepository("MyDbRepository");
MailRepository rep2 = getRepository("MyFileRepository");
This simply requires that the repositories be configured and named via some
other mechanism, such as a config file.
>
> However, although I am really kind of itching to get into these issues, we
> have a bit of an agreement to keep a bit of a lid on James v3 discussions
> until after we get release 2.1 out the door. Danny also has a lot of ideas
> that he wants to post, as do others.
Fair enough. I won't push the point any further at this stage. We will
be wanting to hold off on any JAMES version upgrades until this bit
settles down, though (unless something falls apart on 2.0a1).
> Right now Spool repositories are uniquely handled by the linear processor,
> and internally by RemoteDelivery. I generalized what you are describing
> into further decoupling things to use use repository storage in similar
> fashion to tuple-spaces. I have been thinking about that since looking at
> concurrency, since tuple-spaces and JSR 166 have some similar concurrency
> notions. This approach would permit some of the more interesting ideas
> people have asked about, such as load balancing. James could actually run
> on a network of machines, providing for amazing scalability.
Yes, essentially you are correct about the tuple spaces.
The key aspect of my suggestion is the idea that the spool repository
would no longer be unique. There could be any number of repositories.
Some of these could be spool repositories. A spool repository has an
associated processor, others do not. There would always be an incomming
mail spool repository, associated with the root processor. The
RemoteDelivery mailet is then replaced by another spool repository
associated with a processor containing a RemoteDelivery mailet that
performs remote deliveries, but does not need to perform any repository
handling of its own, or thread management. The semantics around removal
of a message from a spool repository and locking of a message while
processing it would need to be defined carefully, but would probably be
very similar to those used by JMS queues. In fact, a JMS queue could
easily be used as a spool repository implementation. (Ah ha! Now
there's an interesting thought - that would make spool repositories
quite different to other repositories. JMS is a queueing mechanism, not
a persistent storage mechanism.)
>
> My view is that this is the time to consider any radical restructurings. We
> can maintain a James v2 architecture release with bug fixes, but allow James
> v3 to build on new ideas that have come up based upon experience with James
> v3. But first we have to get v2.1 out the door.
Ok.
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: MailRepository
Posted by Aaron Knauf <ak...@xtra.co.nz>.
>
> I agree with Noel - I don't understand why the special spool is
> necessary. Use of a custom mailet to generate your EJB calls seems more
> than sufficient.
>
See my reply to Noel's post.
>>I agree that repository access needs to be provided, but I do NOT that
>
> the
>
>>Repository interface currently present is what ought to be used,
>
> either by
>
>>James or by Mailets. I also agree that whatever the interface to
>>repositories, James and Mailets ought to use the same one.
>
>
> I agree with all of this. I'll take it one step further. There needs
> to be a way for mailets to access all components provided by the server.
> This should be supported by the mailet API. If a required component is
> not available the mailet needs to be able to throw an exception on
> startup to notify the mailet container.
>
I agree that this is the next logical step. It should be done.
>
> > > We would like to be able to define an arbitrary mail repository,
> store
>
>>>incoming mail into it and process that mail asynchronously.
>>
>>See above. It sounds as if you team is going about it all wrong.
>
> Please
>
>>read through the RemoteDelivery mailet.
>
>
> You should be able to do this, but I concur with Noel. Why would you
> want to do this for your app?
>
>
Guaranteed delivery and the ability to back off and wait before a retry
attempt.
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: MailRepository
Posted by "Noel J. Bergman" <no...@devtech.com>.
> Seems like a lot of changes are being held back
> because of reasons that CVS could address.
So are the discussions about what the changes out to be.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: MailRepository
Posted by Serge Knystautas <se...@lokitech.com>.
Danny Angus wrote:
> Just to kind of pre-empt this thread, don't forget that I've already done a lot of this work, but held it back from cvs because of the iminent release.
>
> d.
Can't we just tag and branch the pending release? I forget when the
release plan said we were going to do this. Seems like a lot of changes
are being held back because of reasons that CVS could address.
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: MailRepository
Posted by Danny Angus <da...@apache.org>.
"Noel J. Bergman" <no...@devtech.com> wrote in message
news:NBBBJGEAGJAKLIDBKJOPCEAHOKAA.noel@devtech.com...
> Danny,
>
> I haven't seen it, and I doubt that I would veto it right away if it is
any
> improvement. We could replace it again later if necessary, after we go
> through design discussions and then implement whatever comes from the
> design, if it isn't what you've already done.
+1
> But I thought that we had all
> agreed to talk about DESIGN first, and were defering those discussions
until
> after release.
I know but as I've already done it I don't see why we shouldnt use it, and
change it if we don't like it. Isn't that "the way" ?
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: MailRepository
Posted by "Noel J. Bergman" <no...@devtech.com>.
Danny,
I haven't seen it, and I doubt that I would veto it right away if it is any
improvement. We could replace it again later if necessary, after we go
through design discussions and then implement whatever comes from the
design, if it isn't what you've already done. But I thought that we had all
agreed to talk about DESIGN first, and were defering those discussions until
after release.
--- Noel
-----Original Message-----
From: Danny Angus [mailto:danny@apache.org]
Sent: Thursday, November 14, 2002 14:47
To: James Developers List
Subject: RE: MailRepository
yes I do, I've *done* this, you can veto it back out if you don't like it.
d.
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: 14 November 2002 16:11
> To: James Developers List
> Subject: RE: MailRepository
>
>
> Danny,
>
> By pre-empt, I assume that you don't mean to pre-empt the discussions that
> we've agreed to have after the release, right?
>
> --- Noel
>
> -----Original Message-----
> From: Danny Angus [mailto:danny@apache.org]
> Sent: Thursday, November 14, 2002 3:26
> To: James Developers List
> Subject: RE: MailRepository
>
> Just to kind of pre-empt this thread, don't forget that I've
> already done a lot of this work, but held it back from cvs
> because of the iminent release.
>
> d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: MailRepository
Posted by Danny Angus <da...@apache.org>.
yes I do, I've *done* this, you can veto it back out if you don't like it.
d.
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: 14 November 2002 16:11
> To: James Developers List
> Subject: RE: MailRepository
>
>
> Danny,
>
> By pre-empt, I assume that you don't mean to pre-empt the discussions that
> we've agreed to have after the release, right?
>
> --- Noel
>
> -----Original Message-----
> From: Danny Angus [mailto:danny@apache.org]
> Sent: Thursday, November 14, 2002 3:26
> To: James Developers List
> Subject: RE: MailRepository
>
> Just to kind of pre-empt this thread, don't forget that I've
> already done a
> lot of this work, but held it back from cvs because of the
> iminent release.
>
> d.
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: MailRepository
Posted by "Noel J. Bergman" <no...@devtech.com>.
Danny,
By pre-empt, I assume that you don't mean to pre-empt the discussions that
we've agreed to have after the release, right?
--- Noel
-----Original Message-----
From: Danny Angus [mailto:danny@apache.org]
Sent: Thursday, November 14, 2002 3:26
To: James Developers List
Subject: RE: MailRepository
Just to kind of pre-empt this thread, don't forget that I've already done a
lot of this work, but held it back from cvs because of the iminent release.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: MailRepository
Posted by Danny Angus <da...@apache.org>.
Just to kind of pre-empt this thread, don't forget that I've already done a lot of this work, but held it back from cvs because of the iminent release.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: MailRepository
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Noel and Aaron,
> As I just said in James-User (but let's move the discussion over to
here):
>
> Seems to me that you don't need what you've done. A mailet and the
> internal
> spool would provide sufficient decoupling to generate SMS to an EJB
> backend.
>
> Just have the mailet do this when the service method is called. If
you
> need
> something else, just clone what RemoteDelivery does, which is
basically
> the
> same as your mailet. It uses an internal spool when it can't
immediately
> deliver. We could probably refactor RemoteDelivery into a base class
for
> a
> certain category of mailet, and then provide methods for performing
the
> remote delivery, e.g., to another SMTP server, to your EJB server, to
> another server.
I agree with Noel - I don't understand why the special spool is
necessary. Use of a custom mailet to generate your EJB calls seems more
than sufficient.
> > I think the JAMES mail repository needs to be exposed to mailet
authors.
>
> I agree that repository access needs to be provided, but I do NOT that
the
> Repository interface currently present is what ought to be used,
either by
> James or by Mailets. I also agree that whatever the interface to
> repositories, James and Mailets ought to use the same one.
I agree with all of this. I'll take it one step further. There needs
to be a way for mailets to access all components provided by the server.
This should be supported by the mailet API. If a required component is
not available the mailet needs to be able to throw an exception on
startup to notify the mailet container.
As far as the repository interfaces go, they all need a lot of work. In
James 3.0 I expect we will be making substantial revisions to these
APIs. I know Noel and Serge have a lot to say about the mail
repositories (properties/attributes) and I have more than a few thoughts
on the user repositories. I have no idea what anyone else has to say,
but I'd be pretty naïve if I didn't believe that Danny and others will
champion certain modifications. I also think that support for IMAP and
NNTP in the standard repositories will force certain restrictions on
these interfaces. So calling these interfaces in flux would be an
understatement. :)
> > We would like to be able to define an arbitrary mail repository,
store
> > incoming mail into it and process that mail asynchronously.
>
> See above. It sounds as if you team is going about it all wrong.
Please
> read through the RemoteDelivery mailet.
You should be able to do this, but I concur with Noel. Why would you
want to do this for your app?
> We're very close. I'm hoping that Peter (v2.1 Release Manager) can
post a
> Release Plan Status, so we can talk about who is doing what to help
push
> this to completion and get into some really fun new stuff.
Will do. We are close. Watch for it soon.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: MailRepository
Posted by "Noel J. Bergman" <no...@devtech.com>.
Aaron,
As I just said in James-User (but let's move the discussion over to here):
Seems to me that you don't need what you've done. A mailet and the internal
spool would provide sufficient decoupling to generate SMS to an EJB backend.
Just have the mailet do this when the service method is called. If you need
something else, just clone what RemoteDelivery does, which is basically the
same as your mailet. It uses an internal spool when it can't immediately
deliver. We could probably refactor RemoteDelivery into a base class for a
certain category of mailet, and then provide methods for performing the
remote delivery, e.g., to another SMTP server, to your EJB server, to
another server.
> I think the JAMES mail repository needs to be exposed to mailet authors.
I agree that repository access needs to be provided, but I do NOT that the
Repository interface currently present is what ought to be used, either by
James or by Mailets. I also agree that whatever the interface to
repositories, James and Mailets ought to use the same one.
> We would like to be able to define an arbitrary mail repository, store
> incoming mail into it and process that mail asynchronously.
See above. It sounds as if you team is going about it all wrong. Please
read through the RemoteDelivery mailet.
> 1) Expose the MailRepository interface in the mailet jar file
> and provide a standard way to get a reference to to a
> particular repository.
That depends upon what form the mail repository takes.
However, although I am really kind of itching to get into these issues, we
have a bit of an agreement to keep a bit of a lid on James v3 discussions
until after we get release 2.1 out the door. Danny also has a lot of ideas
that he wants to post, as do others.
> 2) Expose the ability to define mail repositories through config.xml
> *and also to process the mail in them*. At present there is the
> ToRepository mailet to put mail in a repository. The trouble is
> that nothing happens to the mail once it is written to a repository.
> I am thinking that this could be extended to include the ability to
> define a processor that is associated with a named repository.
Right now Spool repositories are uniquely handled by the linear processor,
and internally by RemoteDelivery. I generalized what you are describing
into further decoupling things to use use repository storage in similar
fashion to tuple-spaces. I have been thinking about that since looking at
concurrency, since tuple-spaces and JSR 166 have some similar concurrency
notions. This approach would permit some of the more interesting ideas
people have asked about, such as load balancing. James could actually run
on a network of machines, providing for amazing scalability.
My view is that this is the time to consider any radical restructurings. We
can maintain a James v2 architecture release with bug fixes, but allow James
v3 to build on new ideas that have come up based upon experience with James
v3. But first we have to get v2.1 out the door.
We're very close. I'm hoping that Peter (v2.1 Release Manager) can post a
Release Plan Status, so we can talk about who is doing what to help push
this to completion and get into some really fun new stuff.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>