You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Stephan Wiesner <st...@stephan-wiesner.de> on 2003/04/20 15:13:24 UTC

Re: Serializing Users in FilesForwarding mail to NNTP

I would like to forward some of my mails to my local newsserver. This question was rised before
(http://marc.theaimsgroup.com/?l=james-user&m=104336487319378&w=2) and Harmeet answered:

"One could write a mailet and post messages from james-user mailing list to
an nntp server group called org.apache.james.user.

It would be simple but not too elegant. [...]"

If this is so simple, can anybody give me some more hints?
I mean, I can't just send the mail as a mail to, well to whom?

Stephan




---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users in FilesForwarding mail to NNTP

Posted by Stephan Wiesner <st...@stephan-wiesner.de>.
Good morning Harmeet,
thanks again, works perfectly.
I will turn it into a Mailet and post that one to the list later today.

Stephan

Harmeet Bedi wrote:

>The attached file would do the trick. It is a utility to post msg to either
>authenticated or non-authenticated nntp server.
>
>If you write to spool directory in your local james server, you won't need
>to deal with authentication. If you do this, first create a temp file and
>then move the temp file after write is finished to spool directory,
>otherwise spooler may try to act on a partially written file.
>
>Harmeet
>  
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users in FilesForwarding mail to NNTP

Posted by "Noel J. Bergman" <no...@devtech.com>.
> If you write to spool directory in your local james server, you won't need
> to deal with authentication.

If you get a chance, a mailet that took a message and posted it to a
newsgroup would be a good thing.  We could try pairing that with a change to
NNTP that allowed a moderator address for a newsgroup.

The workflow would be:

  if newsgroup not moderated, post locally
  if newsgroup moderated, send message to e-mail address

Then on the mail side we can use the mailet to post approved messages into
the NNTP spool.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users in FilesForwarding mail to NNTP

Posted by Harmeet Bedi <ha...@kodemuse.com>.
The attached file would do the trick. It is a utility to post msg to either
authenticated or non-authenticated nntp server.

If you write to spool directory in your local james server, you won't need
to deal with authentication. If you do this, first create a temp file and
then move the temp file after write is finished to spool directory,
otherwise spooler may try to act on a partially written file.

Harmeet
----- Original Message -----
From: "Stephan Wiesner" <st...@stephan-wiesner.de>
To: "James Developers List" <ja...@jakarta.apache.org>
Sent: Monday, April 21, 2003 2:49 AM
Subject: Re: Serializing Users in FilesForwarding mail to NNTP


> Hi Harmeet,
> thanks for your code, you were right, it is very simple :-)
> However, I did not manage to connect to a newsserver that requires
> authentication and did not find anything to do this, neither in the
> common.net.apidocs, nor in RFC 822. Can you help me there too?
>
> Stephan
>
> >From: "Harmeet Bedi" <ha...@kodemuse.com>
> >
> >
> >>Mailet could identify interesting mail by looking to, cc or from header
of
> >>mail message and post using nntp client API to james NNTP server.
> >>Mimemessage in mailet is a perfect article source.
> >>
> >>James currently packages commons-net library. Attached is an example
that
> >>uses this already packaged library to post messages to nntp server.
> >>
> >>
> >
> >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-dev-help@jakarta.apache.org

Re: Serializing Users in FilesForwarding mail to NNTP

Posted by Stephan Wiesner <st...@stephan-wiesner.de>.
Hi Harmeet,
thanks for your code, you were right, it is very simple :-)
However, I did not manage to connect to a newsserver that requires 
authentication and did not find anything to do this, neither in the 
common.net.apidocs, nor in RFC 822. Can you help me there too?

Stephan

>From: "Harmeet Bedi" <ha...@kodemuse.com>
>  
>
>>Mailet could identify interesting mail by looking to, cc or from header of
>>mail message and post using nntp client API to james NNTP server.
>>Mimemessage in mailet is a perfect article source.
>>
>>James currently packages commons-net library. Attached is an example that
>>uses this already packaged library to post messages to nntp server.
>>    
>>
>
>  
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users in FilesForwarding mail to NNTP

Posted by Harmeet Bedi <ha...@kodemuse.com>.
> From: "Stephan Wiesner" <st...@stephan-wiesner.de>
> > If this is so simple, can anybody give me some more hints?

From: "Harmeet Bedi" <ha...@kodemuse.com>
> Mailet could identify interesting mail by looking to, cc or from header of
> mail message and post using nntp client API to james NNTP server.
> Mimemessage in mailet is a perfect article source.
>
> James currently packages commons-net library. Attached is an example that
> uses this already packaged library to post messages to nntp server.

Stephan there was a mistake in the attachment file posted earlier. Here is
the fixed example file. It should give you more than a few 'hints' to send
mail message to nntp server. :-)
You can also directly write messages to nntp spool directly and have it
picked up. Remember to add the right newsgroup header.Another thing to know
is that the message id is maintained so you can only create one newsgroup
article with one mime message. Do send mail if you hit issues.

Harmeet

Re: Serializing Users in FilesForwarding mail to NNTP

Posted by Harmeet Bedi <ha...@kodemuse.com>.
From: "Stephan Wiesner" <st...@stephan-wiesner.de>
> If this is so simple, can anybody give me some more hints?

Mailet could identify interesting mail by looking to, cc or from header of
mail message and post using nntp client API to james NNTP server.
Mimemessage in mailet is a perfect article source.

James currently packages commons-net library. Attached is an example that
uses this already packaged library to post messages to nntp server.



There is also some old code in cvs that uses jdk 1.3 internal classes that
one could and to writes files to nntp spool directory. I had used this to
bootstrap nntp repository for testing in the past.

http://cvs.apache.org/viewcvs.cgi/jakarta-james/src/java/org/apache/james/te
sting/Attic/NNTPClient.java?rev=HEAD&hideattic=0&content-type=text/vnd.viewc
vs-markup

You may want  to take look at NNTPSpooler <run> method to see how message is
picked up.
Ref:
http://cvs.apache.org/viewcvs.cgi/jakarta-james/src/java/org/apache/james/nn
tpserver/repository/NNTPSpooler.java?rev=1.19&content-type=text/vnd.viewcvs-
markup
MimeMessage from Mailet is perfect source for NNTP Article.

Harmeet


----- Original Message -----
From: "Stephan Wiesner" <st...@stephan-wiesner.de>
To: "James Developers List" <ja...@jakarta.apache.org>
Sent: Sunday, April 20, 2003 9:13 AM
Subject: Re: Serializing Users in FilesForwarding mail to NNTP


> I would like to forward some of my mails to my local newsserver. This
question was rised before
> (http://marc.theaimsgroup.com/?l=james-user&m=104336487319378&w=2) and
Harmeet answered:
>
> "One could write a mailet and post messages from james-user mailing list
to
> an nntp server group called org.apache.james.user.
>
> It would be simple but not too elegant. [...]"
>
> If this is so simple, can anybody give me some more hints?
> I mean, I can't just send the mail as a mail to, well to whom?
>
> Stephan
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-dev-help@jakarta.apache.org

RE: Serializing Users in FilesForwarding mail to NNTP

Posted by Vincent de Lau <vi...@delau.nl>.
> ... In my (incredibly arrogant tonight) opinion, the
> only thing the filesystem structure is good for is storing message
> files. Any other semantic information should be held in a database where
> it is easy to search for.

Storing messages in a database can be usefull as well, since you could do a
fulltext search on them for instance. With filesystems you can also get into
trouble, for example waste of space because of fs block size and number of
files per directory.

> I also think the 1.2 option of storing the message files in a database
> is (currently) a bad idea. Unless one is running the database on an OS
> with a 64 bit filesystem (which leaves out Win32 and Linux), creating
> large database storage capacity is a real pain because the datafiles are
> limited to 2 GB's and one must aggregate multiple datafiles to provide a
> reasonable amount of storage.

This isn't a problem for MySQL 4 (MyISAM/RAID or InnoDB) or MSSQL.

Vincent de Lau
 vincent@delau.nl


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Sid Stuart <si...@weaselworks.com>.
Stephan Wiesner wrote:

> Reading the messages about using files or databases for storing the 
> messages and considering my need for an archieving system, I decided 
> to write a PHP based interface to the MySQL-stored mails. Including 
> full text search and all (read only functionallity). This would have 
> been much more work with the file base approach. 

It's my assertion that you should be able to do this using the mailet 
interface. I would like to see James 3.0 support this requirment instead 
of forcing you to delve into an implementation detail.

> I hesitated long, however, as I keep to get infinity loops when 
> playing around with Mailets and it is much more easy to just rm * in 
> the spool directory then to log into the database and do it there. 
> Convenience is iportant to me, I guess .-) 

Yes! May I quote you about the inifinite loops? I was just arguing for 
changes to the mailet definition to reduce their possibility.

Sid



---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users... (Now message repository implementations)

Posted by "Noel J. Bergman" <no...@devtech.com>.
> it is much more easy to just rm * in the spool 
> directory then to log into the database and do
> it there.

Try mysql --exec="drop from maildb.spool;"

	--- Noel

---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Stephan Wiesner <st...@stephan-wiesner.de>.
Reading the messages about using files or databases for storing the 
messages and considering my need for an archieving system, I decided to 
write a PHP based interface to the MySQL-stored mails. Including full 
text search and all (read only functionallity). This would have been 
much more work with the file base approach.

I hesitated long, however, as I keep to get infinity loops when playing 
around with Mailets and it is much more easy to just rm * in the spool 
directory then to log into the database and do it there. Convenience is 
iportant to me, I guess .-)

Stephan



---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Serge Knystautas <se...@lokitech.com>.
Sid Stuart wrote:
> On the one hand I see a group of talented OO architects/developers 
> designing a system using good programming paradigms like developing an 
> interface and then writing classes  to that interface to hide the 
> implementation detail. On the other hand, I see these same programmers 
> opening up the implementation details of the storage repository to the 
> world and saying, have at it. How is it that what is considered good 
> programming practise for implementing the system is not good programming 
> practise for designing the system?

I think James does a good job isolating most users from the underlying 
implementations.  The management tools are admitedly, woefully 
inadequate.  It's all open source, so if people need to contribute in 
those areas, then they need to see all the details understandably.

The repository interface is also not 100% solid, and as you can see, 
currently evolving.  This will be the first time we're evolve it, and 
this will be to improve the API separation as well as give everyone more 
options.  Hopefully we'll get it close enough to right to not need to do 
it again so the management tools and mailet authors don't need to 
understand the guts.

> The contract with the end user should be the protocols implemented on 
> top of James, SMTP, POP3, IMAP and the programming interface provided by 
> the mailet package. If the requirements of the users are not met with 
> these specifications, the specifications should be reviewed/enhanced or 
> the users should be told to find another system. All implementation 
> details should be hidden behind protocol servers or interface 
> specifications.

Yes, the repository API is getting moved into the mailet package so 
those who need access to them don't have to worry about this.  We're 
still somewhat debating requirements, but some movement has already 
begun towards those improvements.  And as Noel said, for those 
improvements that can work against the current design, they're being 
applied to the 2.1 branch.

Your thoughts and contributions towards this are of course welcome. :)

-- 
Serge Knystautas
President
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users... (Now message repository implementations)

Posted by Danny Angus <da...@apache.org>.
> The contract with the end user should be the protocols implemented on 
> top of James, SMTP, POP3, IMAP and the programming interface provided by 
> the mailet package. If the requirements of the users are not met with 
> these specifications, the specifications should be reviewed/enhanced or 
> the users should be told to find another system. All implementation 
> details should be hidden behind protocol servers or interface 
> specifications.

I completely disagree.

The popularity of httpd is in no small part owed to the fact that it can be extended by thirdparties for their own purposes.

What we are aiming for is that we make some implementation decisions about what ships with James, but do not prevent you, or anyone else, from making alternative implementations of *many* of James internal components to better suit James to your specific needs.

The Mailet API is one thing James is something else.

d.


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Sid Stuart <si...@weaselworks.com>.
Serge Knystautas wrote:

>
> Can you explain how the "db" implementation is slowing down the 
> delivery schedule?  The repository changes in the works affect dbfile 
> in every way they affect db.  In fact, db and dbfile are going to be 
> easier to migrate than plain file because of the ability to do the 
> schema change scripts, which as you say is why it's nice to use a 
> database for metadata.
>
> And also, we are actually looking to add MANY more repository types, 
> namely mbox, maildir, remote IMAP/POP3, as well as maybe other types 
> as implementations are available.  We may also be separating the 
> SMTP/spooling logic from the underlying repository, allowing us to use 
> any JavaMail store implementation for repositories serving as user 
> mailboxes. 

I didn't say the "db" implementation is slowing dow the delivery 
schedule. I said "From the mail I have seen over the last week, the 
design and implementation of the message repository is impacting  the 
delivery schedule." I was referencing two notes from Mr. Bedi:

Excerpted from "Re: preprocessing after smtp DATA", dated  4/20/2003 4:27 AM
    Is there any idea when will 3.0 released? 1 month, 1 year?

    I think it is far away. The main change is moving to a
    different repository implementation and there has been 
    limited activity on that front. I would guess release is
    6 months or so away.


Excerpted from "Re: Serializing Users in FilesForwarding mail to NNTP", 
dated 4/20/2003 6:32 PM.

   Everybody agrees we should converge to a repository and 
   format asap. At the moment it is not too far apart at file
   system level but everything else is parallel.
   The key is moving code to standard repository API. The 
   sooner we do this the better. If javamail proposal takes 
   too long to build maybe we should just let it go away and 
   use the current repository API/implementations as it is
   and move nntp to it.  thoughts.. ?


I now understand the James 3.0 proposal includes adding several more 
repository implementations such as mbox and maildir. The idea was so 
foreign to me that it took long time to grasp. Quite frankly, when I 
read that the rational for doing so is that external tools can access 
the implementations, I had to wonder what planet I was on.

On the one hand I see a group of talented OO architects/developers 
designing a system using good programming paradigms like developing an 
interface and then writing classes  to that interface to hide the 
implementation detail. On the other hand, I see these same programmers 
opening up the implementation details of the storage repository to the 
world and saying, have at it. How is it that what is considered good 
programming practise for implementing the system is not good programming 
practise for designing the system?

The contract with the end user should be the protocols implemented on 
top of James, SMTP, POP3, IMAP and the programming interface provided by 
the mailet package. If the requirements of the users are not met with 
these specifications, the specifications should be reviewed/enhanced or 
the users should be told to find another system. All implementation 
details should be hidden behind protocol servers or interface 
specifications.

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Serge Knystautas <se...@lokitech.com>.
Sid Stuart wrote:
>     3. James design/implementation simplicity: the fewer repository 
> implementations that need
>        to be supported, the easier it will be to get James 3.0 out the 
> door. From the mail I have
>        seen over the last week, the design and implementation of the 
> message repository is impacting
>        the delivery schedule. Shouldn't that trigger a design review?
> 
> These are my arguments against the James "db" storage format. What are 
> the arguments for it? What benefits does the "db" implementation offer 
> over "dbfile"? I would really like to know.

If you object to the "db" storage format, then don't use it.  Other 
people want to use it, and won't agree with all your points.  As for my 
reasons otherwise, but basically I would just repeat your points one and 
two and throw in the word "not".  Just a difference of opinion, and 
that's why it's very nice to have pluggable repositories so different 
views can coexist.

Can you explain how the "db" implementation is slowing down the delivery 
schedule?  The repository changes in the works affect dbfile in every 
way they affect db.  In fact, db and dbfile are going to be easier to 
migrate than plain file because of the ability to do the schema change 
scripts, which as you say is why it's nice to use a database for metadata.

And also, we are actually looking to add MANY more repository types, 
namely mbox, maildir, remote IMAP/POP3, as well as maybe other types as 
implementations are available.  We may also be separating the 
SMTP/spooling logic from the underlying repository, allowing us to use 
any JavaMail store implementation for repositories serving as user 
mailboxes.

Nobody has quite cracked all the issues outstanding with the changes 
underway.  We just know...
- we want to integrate with other mail products by supporting the other 
standards.
- we want to continue to improve performance
- we want to get NNTP using the same repositories
- we want to allow IMAP to have repositories it can use
- possibly we want to separate spooling logic so multiple James servers 
could use the same repository and use a distributed spool implementation.

-- 
Serge Knystautas
President
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Sid Stuart <si...@weaselworks.com>.
Noel J. Bergman wrote:

>>I've supported developers working on real-time replication applications.
>>There was a lot of overhead involved in performing the replication
>>    
>>
>
>Different needs.  For example, we have heavily database-oriented web sites,
>and we use replication to provide real-time backup of transactions.
>  
>
Yes, but how large are the rows in the tables that support those 
transactions? Are there any BLOB's or CLOB's  involved? My concern is 
that adding the message content to the database will introduce a 
noticble latency even before it is configured for replication. Also, do 
you really do mean replication instead of hot-standby? I haven't seen 
too many websites that need the smaller failover time provided by 
replication. Hot-standby is simpler to implement and has a lower runtime 
overhead.

>>If I were implementing a robust system, I would reduce the
>>load on the database by storing the content on a clustered
>>NFS server.
>>    
>>
>
>You would make the tradeoff by going from a centrally managed DB server to a
>cluster of servers using file locking?  That is NOT a tradeoff I would make.
>I'm not sure which databases support transactions in such an environment.
>Or are you just refering to the dbfile example, and putting the content on
>an NFS mount?  The latter is certainly an option that can be employed.
>  
>
I was referring to the dbfile example. I agree that file locking is not 
an acceptable solution in a distributed environment.

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users... (Now message repository implementations)

Posted by "Noel J. Bergman" <no...@devtech.com>.
>For James v3, I hope to have hsql embedded so that all James distributions
>can make use of JDBC, although we'll have to talk about whether or not we
>can rely upon it.

> I have to confess, my rants about only providing dbfile as the message
> storage implementation are based on the conviction that hsql is
> reliable.

Big assumption.  I hope that it is reliable, too.

> Have you considered using that package for persistance instead of
> writing custom JDBC?

What custom JDBC?  We're talking about http://hsqldb.sourceforge.net/, which
has a JDBC driver and is supported by the Avalon folks as a component.  And
James already has a SQL Resources construct that factors out differences
between SQL language implementations.

> > One benefit of db storage is the provision for real-time replication

> I've supported developers working on real-time replication applications.
> There was a lot of overhead involved in performing the replication

Different needs.  For example, we have heavily database-oriented web sites,
and we use replication to provide real-time backup of transactions.

> If I were implementing a robust system, I would reduce the
> load on the database by storing the content on a clustered
> NFS server.

You would make the tradeoff by going from a centrally managed DB server to a
cluster of servers using file locking?  That is NOT a tradeoff I would make.
I'm not sure which databases support transactions in such an environment.
Or are you just refering to the dbfile example, and putting the content on
an NFS mount?  The latter is certainly an option that can be employed.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Sid Stuart <si...@weaselworks.com>.
Noel J. Bergman wrote:

>>In fact, I love the database concept so much (dbfile that is), I
>>think it should be the only mechanism used for storing metadata
>>associated with the messages in the repository.
>>    
>>
>
>For James v3, I hope to have hsql embedded so that all James distributions
>can make use of JDBC, although we'll have to talk about whether or not we
>can rely upon it.
>  
>
I have to confess, my rants about only providing dbfile as the message 
storage implementation are based on the conviction that hsql is 
reliable. I have not used it before. I googled it, but could not find 
any good reports on it. The only bad report came from PointBase, but 
that was a generalized screed against open source Java database 
implementations. I'm not sure it would carry much weight here. :) 

One note in hsql's favour is the Hibernate folks have support for it. 
Have you considered using that package for persistance instead of 
writing custom JDBC? A friend who does high transaction-rate java 
commercial implementations (banking) likes it a lot.

>>storing the message content in the database is a bad
>>idea for several reasons,
>>    
>>
>
>One benefit of db storage is the provision for real-time replication,
>providing the most robust storage available in a mail server.  Yes, there is
>a tradeoff in terms of bandwidth and overhead.  Again, flexibility.  The
>mail admin can choose the desired mail storage to fit the needs of their
>environment.
>
I've supported developers working on real-time replication applications. 
There was a lot of overhead involved in performing the replication and 
that slowed the DB down. If I were implementing a robust system, I would 
reduce the load on the database by storing the content on a clustered 
NFS server.
Even Linux supports that feature these days.

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users... (Now message repository implementations)

Posted by "Noel J. Bergman" <no...@devtech.com>.
> My comfort level for the size of a message repository in these days of
> high bandwidth connections and multimedia attachments is at least 100 GB.

Needs vary.  James is flexible.  At an average of 10K per message (a high
average based upon real-life traffic), your requirement is for 10 million
messages in storage.  That is over 10,000 messages in storage for a 1000
user installation.  Reasonable for IMAP, highly unusual for POP3.  It is
roughly 100 messages per user for a 100,000 user installation.  [Assuming I
calculated the results properly; I'm a bit under the weather today.]

> In fact, I love the database concept so much (dbfile that is), I
> think it should be the only mechanism used for storing metadata
> associated with the messages in the repository.

For James v3, I hope to have hsql embedded so that all James distributions
can make use of JDBC, although we'll have to talk about whether or not we
can rely upon it.

> storing the message content in the database is a bad
> idea for several reasons,

One benefit of db storage is the provision for real-time replication,
providing the most robust storage available in a mail server.  Yes, there is
a tradeoff in terms of bandwidth and overhead.  Again, flexibility.  The
mail admin can choose the desired mail storage to fit the needs of their
environment.

> From the mail I have seen over the last week, the design and
> implementation of the  message repository is impacting the
> delivery schedule. Shouldn't that trigger a design review?

Actually, the availability of developer cycles is the gating factor.  Not
the design.  Things that don't require design change fit perfectly well into
James v2.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Sid Stuart <si...@weaselworks.com>.
Noel J. Bergman wrote:

>>I've spent the couple of months on a MUA design that I
>>would like to implement. I hoped that James would provide
>>me with the tools to do so easily.  Unfortunately, I've
>>come to the realization that James doesn't provide the
>>flexibility I need
>>    
>>
>
>What do you believe to be missing?
>
A facility to support access to and modification of the metadata 
associated with the message content stored in the message repository and 
the metadata associated with the users stored in the user repository. My 
design depends on the ability to define custom attributes for users and 
messages and then perform operations on those attributes, as well as 
standard attributes, using something similar to the directed graphs of 
matchers and mailets. The facility would need to provide event driven 
and calendar driven initiation of the processors defined by the directed 
graphs.

Examples of how it would be used,
    a)  Rerun the dead letter queue on a nightly basis
    b)  Generate message list digests (C.3 from Mr. Behlendorf's wish list)
    c)  Provide the access definitions needed for C.5 on Mr. 
Behlendorf's wish list.
    d) Provide pre-filtering of incoming and outgoing email for 
guardians of minors.
    e) Provide pass/block list filtering of incoming messages.
    e) Provide a facility for post-delivery removal of spam and virus's

>By the way, you do realize that James is an MTA, right?  I can refer you to
>Open Source MUA projects.
>
>	--- Noel
>  
>
Late night slip of the fingers. I meant to say MDA. If I understand the 
definitions correctly, James is actually an MTA and a MDA (just had to 
delete MUA again...)  It's the MDA part I am interested in.

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users... (Now message repository implementations)

Posted by "Noel J. Bergman" <no...@devtech.com>.
> I've spent the couple of months on a MUA design that I
> would like to implement. I hoped that James would provide
> me with the tools to do so easily.  Unfortunately, I've
> come to the realization that James doesn't provide the
> flexibility I need

What do you believe to be missing?

By the way, you do realize that James is an MTA, right?  I can refer you to
Open Source MUA projects.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Sid Stuart <si...@weaselworks.com>.
Serge Knystautas wrote:

> Sid Stuart wrote:
>
>> I started looking at James because I also want to write tools that 
>> manipulate data in the repositories. I want to do it through a 
>> standard api though so that the tool I develop will work for all 
>> implementations of the mail system, not just those using a particular 
>> implementation of the message repository. Instead of writing multiple 
>> storage implementations, your time would be better spent 
>> understanding the operations that developers want to perform on the 
>> repository and providing a facility that allows them to implement 
>> those operations.
>
>
> Sid,
>
> I don't think that came out as helpful as you had intended.  
> Criticizing Danny's use of time isn't going to convince him to work on 
> what you want him to.
>
> This is an open source project where requirements gathering is not 
> formal, with decisions largely based on people's persuasive skills as 
> well as their willingness to do something about it.  The mailing list 
> archives are an excellent source of information about what people have 
> requested as well as suggested solutions to these requests.  A lot of 
> people have weighed in on this already, and we have all done our best 
> to recognize people's interests.
>
I certainly didn't mean to criticize Danny or anyone else. Everyone on 
this mailing list has been quite helpful and patient. I've spent the 
couple of months on a MUA design that I would like to implement. I hoped 
that James would provide me with the tools to do so easily. 
Unfortunately, I've come to the realization that James doesn't provide 
the flexibility I need and I see no hope of it doing so in the future. 
My disappointment at the realization was poorly expressed. My apologies 
for that to all of you.

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Serge Knystautas <se...@lokitech.com>.
Sid Stuart wrote:
> I started looking at James because I also want to write tools that 
> manipulate data in the repositories. I want to do it through a standard 
> api though so that the tool I develop will work for all implementations 
> of the mail system, not just those using a particular implementation of 
> the message repository. Instead of writing multiple storage 
> implementations, your time would be better spent understanding the 
> operations that developers want to perform on the repository and 
> providing a facility that allows them to implement those operations.

Sid,

I don't think that came out as helpful as you had intended.  Criticizing 
Danny's use of time isn't going to convince him to work on what you want 
him to.

This is an open source project where requirements gathering is not 
formal, with decisions largely based on people's persuasive skills as 
well as their willingness to do something about it.  The mailing list 
archives are an excellent source of information about what people have 
requested as well as suggested solutions to these requests.  A lot of 
people have weighed in on this already, and we have all done our best to 
recognize people's interests.

-- 
Serge Knystautas
President
Lokitech >> software . strategy . design >> http://www.lokitech.com/
p. 1.301.656.5501
e. sergek@lokitech.com



---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users... (Now message repository implementations)

Posted by "Noel J. Bergman" <no...@devtech.com>.
> I think the JavaMail interface is an interesting place to start.  Is
> there a requirements document that lists the needs of the various
> servers and verifies that the interface will provide the functionality
> require?

Ask Bill Shannon.

> (a)  I have concerns that the metadata required by the JavaMail
> interface cannot be expressed by the mbox implementation.

You might want to look at one of the mbox, maildir or mh implementations.
Where do you find metadata requirements?  As an aside, consider the dbfile
representation; there is nothing to prevent metadata from being stored
orthogonally to the standard data representation.

> How is it a standard interface if an underlaying implementation
> does not support all it's semantics?

Considering that JavaMail was designed for this purpose, if there is
something that you feel is lacking, you might want to raise it with Bill
Shannon.  I've done so.  The next version of JavaMail is at least a year
away, but there are at least workarounds for the things James needs.

> (b) The thousands administrators who have chosen of Cyrus, UofW IMAP/POP
> and Exchange. None of these systems offer message storage options.

If we adopt JavaMail as the storage interface, we can provide our own
storage service provider optimized for what we feel is best for performance
and reliability, and have the option of allowing other storage formats for
free.

> One quick note on the request for mbox support, I saw the suggestion to
> use eyebrowse in the email from Brian Behlendorf in the wiki. I also
> took a quick look at the eyebrowse page.  I'm not sure just providing the
> mbox format will be sufficient to support that tool as it also requires
> metadata stored in a mySQL database. Has anyone spec'ed out a design of
> how eyebrowse would interface to James?

You seem to be confused.  eyebrowse (http://eyebrowse.tigris.org/) generates
the metadata, which it stores either via Lucene or in MySQL.  eyebrowse
doesn't interface to the mail server.  It processes mbox directly.  Please
note that it also stores its metadata independently of the archive content,
which I would consider to bolster the case, rather than undermine it.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Sid Stuart <si...@weaselworks.com>.
Noel J. Bergman wrote:

>>I also want to write tools that manipulate data in the repositories.
>>I want to do it through a standard api though so that the tool I
>>develop will work for all implementations of the mail system
>>    
>>
>
>Thank you for yet another justification to adopt the JavaMail interface.
>
 I think the JavaMail interface is an interesting place to start.  Is 
there a requirements document that lists the needs of the various 
servers and verifies that the interface will provide the functionality 
require?

>>Instead of writing multiple storage implementations, your time
>>would be better spent understanding the operations that
>>developers want to perform on the repository and providing
>>a facility that allows them to implement those operations.
>>    
>>
>
>What makes you think that (a) a standard interface isn't the idea, or (b)
>that end users don't care about the storage implementation?  Remember: there
>are other users who may feel that they want to use a particular format
>because of other tools that they have, such as tools that take mbox format
>and produce web archives.
>
>	--- Noel
>
(a)  I have concerns that the metadata required by the JavaMail 
interface cannot be expressed by the mbox implementation. How is it a 
standard interface if an underlaying implementation does not support all 
it's semantics?

(b) The thousands administrators who have chosen of Cyrus, UofW IMAP/POP 
and Exchange. None of these systems offer message storage options. The 
question is who is your customer base? a limited set of sophisticated 
programmers/administrators or the general administrator population. The 
general population cares about what protocols the server supports, how 
easy it is to install and how reliable it is. (I quote from Brian 
Behlendorf's email listed in the wiki, "reliable, reliable, reliable".)
(Okay, the Exchange administrators don't care about it being reliable or 
easy to install but they have been brainwashed to accept a lower set of 
standards.)

One quick note on the request for mbox support, I saw the suggestion to 
use eyebrowse in the email from Brian Behlendorf in the wiki. I also 
took a quick look at the eyebrowse page. I'm not sure just providing the 
mbox format will be sufficient to support that tool as it also requires 
metadata stored in a mySQL database. Has anyone spec'ed out a design of 
how eyebrowse would interface to James?

Sid



---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Sid Stuart <si...@weaselworks.com>.
Noel J. Bergman wrote:

>>I also want to write tools that manipulate data in the repositories.
>>I want to do it through a standard api though so that the tool I
>>develop will work for all implementations of the mail system
>>    
>>
>
>Thank you for yet another justification to adopt the JavaMail interface.
>
>  
>
Hi Noel, what do you mean by adopt the JavaMail interface? In 
particular, which class or interface would a repository implementation 
provide for access to the data?

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users... (Now message repository implementations)

Posted by "Noel J. Bergman" <no...@devtech.com>.
> I also want to write tools that manipulate data in the repositories.
> I want to do it through a standard api though so that the tool I
> develop will work for all implementations of the mail system

Thank you for yet another justification to adopt the JavaMail interface.

> Instead of writing multiple storage implementations, your time
> would be better spent understanding the operations that
> developers want to perform on the repository and providing
> a facility that allows them to implement those operations.

What makes you think that (a) a standard interface isn't the idea, or (b)
that end users don't care about the storage implementation?  Remember: there
are other users who may feel that they want to use a particular format
because of other tools that they have, such as tools that take mbox format
and produce web archives.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Sid Stuart <si...@weaselworks.com>.
Danny Angus wrote:

>>These are my arguments against the James "db" storage format. What are 
>>the arguments for it? What benefits does the "db" implementation offer 
>>over "dbfile"? I would really like to know.
>>    
>>
>
>briefly I use db message format because..
>I use db replication for data integrity
>I also use a number of other db aware tools to manipulate the repositories. If/when we implement plain text file storage I could write scripts etc to maipulate the files, 'till then I would have to use Java for everything which is not a serious option.
>  
>
I started looking at James because I also want to write tools that 
manipulate data in the repositories. I want to do it through a standard 
api though so that the tool I develop will work for all implementations 
of the mail system, not just those using a particular implementation of 
the message repository. Instead of writing multiple storage 
implementations, your time would be better spent understanding the 
operations that developers want to perform on the repository and 
providing a facility that allows them to implement those operations.

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users... (Now message repository implementations)

Posted by Danny Angus <da...@apache.org>.
> If the data stored in the database is limited to the metadata required 
> to support the repository interface and the content associated with the 
> message in stored in a filesystem (essentially the James dbfile 
> implementation), then the size and complexity of the database is reduced 
> significantly. 

Well we provide you with that option, go use it :-)


> I need to go back to school, as my communication skills seem to have 
> deteriorated. I have been trying to say that using a database is a great 
> idea. In fact, I love the database concept so much (dbfile that is), I 
> think it should be the only mechanism used for storing metadata 
> associated with the messages in the repository. However, storing the 
> message content in the database is a bad idea for several reasons,
> 
>     1. The implementation complexity discussed above.
>     2. Performance considerations, for James to operate well, the access 
> to metadata stored in the
>       database should be as low latency as possible. Moving message 
> content data through the
>       database will increase the latency of  operations accessing 
> metadata. No, I have not run tests
>       on this assertion, but it is common sense.


There is little complexity in configuring MySQL to use large tables, but I accept the principle.
I also agree that in pure IO byte rate terms file IO should outperform db, but with cacheing and read ahead in rdbms this is not guarenteed.
.

> From the mail I have
>         seen over the last week, the design and implementation of the 
> message repository is impacting
>         the delivery schedule. Shouldn't that trigger a design review?

Not really true, its more like lack of developer resources, and a whole load of less interesting stuff waiting to be done.
Plus AFAIK there is no immediate plan to release v3, v3 represents some major overhaul and is likely to be some time coming.

> 
> These are my arguments against the James "db" storage format. What are 
> the arguments for it? What benefits does the "db" implementation offer 
> over "dbfile"? I would really like to know.

briefly I use db message format because..
I use db replication for data integrity
I also use a number of other db aware tools to manipulate the repositories. If/when we implement plain text file storage I could write scripts etc to maipulate the files, 'till then I would have to use Java for everything which is not a serious option.

d.


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users... (Now message repository implementations)

Posted by Sid Stuart <si...@weaselworks.com>.
Danny Angus wrote:

>Sid,
>
>
>And second.. Well, from what I can see on the MySQL link you sent MySQL provide a lot of information and support to help people overcome the 4gb limit, and achieve "8 million terabytes" for each table, not 2gb per database as you suggest.
>
I did not say one could not create a large database, I said it was 
painful to do so as one needed to create multiple 2 or 4 gigabyte 
datafiles to support it. As an example, say the database server is a 
version of Linux that supports a 4 GB file size limit. My comfort level 
for the size of a message repository in these days of high bandwidth 
connections and multimedia attachments is at least 100 GB. That means an 
installation using the database to support the James "db" storage 
implementation will need 25 datafiles to support the repository. In 
addition, growing the repository now requires the intervention of 
someone schooled in the database and downtime for the system as the 
database is reconfigured.

If the data stored in the database is limited to the metadata required 
to support the repository interface and the content associated with the 
message in stored in a filesystem (essentially the James dbfile 
implementation), then the size and complexity of the database is reduced 
significantly. It also makes growing the space allocated to content 
storage as simple as adding a filesystem and adding a parameter to a 
James configuration file.

>In addition I still can't see why you think using a db is not a good idea, surely a db is ideal for uses such as this where a number of identically structured data items are to be stored and retrieved by criteria possibly several times during their lifetime? What is your complaint about this?
>  
>
I need to go back to school, as my communication skills seem to have 
deteriorated. I have been trying to say that using a database is a great 
idea. In fact, I love the database concept so much (dbfile that is), I 
think it should be the only mechanism used for storing metadata 
associated with the messages in the repository. However, storing the 
message content in the database is a bad idea for several reasons,

    1. The implementation complexity discussed above.
    2. Performance considerations, for James to operate well, the access 
to metadata stored in the
      database should be as low latency as possible. Moving message 
content data through the
      database will increase the latency of  operations accessing 
metadata. No, I have not run tests
      on this assertion, but it is common sense.
     3. James design/implementation simplicity: the fewer repository 
implementations that need
        to be supported, the easier it will be to get James 3.0 out the 
door. From the mail I have
        seen over the last week, the design and implementation of the 
message repository is impacting
        the delivery schedule. Shouldn't that trigger a design review?

These are my arguments against the James "db" storage format. What are 
the arguments for it? What benefits does the "db" implementation offer 
over "dbfile"? I would really like to know.

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users in FilesForwarding mail to NNTP

Posted by Danny Angus <da...@apache.org>.
Sid,

1st of all please use plain text for mail to these lists.

And second.. Well, from what I can see on the MySQL link you sent MySQL provide a lot of information and support to help people overcome the 4gb limit, and achieve "8 million terabytes" for each table, not 2gb per database as you suggest.

In addition I still can't see why you think using a db is not a good idea, surely a db is ideal for uses such as this where a number of identically structured data items are to be stored and retrieved by criteria possibly several times during their lifetime? What is your complaint about this?

d.


-----Original Message-----
From: Sid Stuart [mailto:sid@weaselworks.com]
Sent: 21 April 2003 12:24
To: James Developers List
Subject: Re: Serializing Users in FilesForwarding mail to NNTP


Danny Angus wrote:

Unless one is running the database on an OS 
with a 64 bit filesystem (which leaves out Win32 and Linux), creating 
large database storage capacity is a real pain because the datafiles are 
limited to 2 GB's and one must aggregate multiple datafiles to provide a 
reasonable amount of storage.
    

I never heard this *ever* before, and I don't believe it either, what do you base this on?

d.

  
>From this, http://www.mysql.com/doc/en/Table_size.html

Note that if one uses Oracle and implements uses their feature to store data in raw partitions, this problem is overcome. Most people I know dislike that solution though for supportability reasons.

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users in FilesForwarding mail to NNTP

Posted by Sid Stuart <si...@weaselworks.com>.
Danny Angus wrote:

>>Unless one is running the database on an OS 
>>with a 64 bit filesystem (which leaves out Win32 and Linux), creating 
>>large database storage capacity is a real pain because the datafiles are 
>>limited to 2 GB's and one must aggregate multiple datafiles to provide a 
>>reasonable amount of storage.
>>    
>>
>
>I never heard this *ever* before, and I don't believe it either, what do you base this on?
>
>d.
>
>  
>
 From this, http://www.mysql.com/doc/en/Table_size.html

Note that if one uses Oracle and implements uses their feature to store 
data in raw partitions, this problem is overcome. Most people I know 
dislike that solution though for supportability reasons.

Sid

RE: Serializing Users in FilesForwarding mail to NNTP

Posted by Danny Angus <da...@apache.org>.
> I also think the 1.2 option of storing the message files in a database 
> is (currently) a bad idea.

Well thats a matter of personal opinion.

> Unless one is running the database on an OS 
> with a 64 bit filesystem (which leaves out Win32 and Linux), creating 
> large database storage capacity is a real pain because the datafiles are 
> limited to 2 GB's and one must aggregate multiple datafiles to provide a 
> reasonable amount of storage.

I never heard this *ever* before, and I don't believe it either, what do you base this on?

d.


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users in FilesForwarding mail to NNTP

Posted by Danny Angus <da...@apache.org>.
Sid Stuart wrote:

> James 1.3

Do you mean James v3?


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users in FilesForwarding mail to NNTP

Posted by Sid Stuart <si...@weaselworks.com>.
Harmeet Bedi wrote:

>Everybody agrees we should converge to a repository and format asap. At the
>moment it is not too far apart at file system level but everything else is
>parallel.
>The key is moving code to standard repository API. The sooner we do this the
>better. If javamail proposal takes too long to build maybe we should just
>let it go away and use the current repository API/implementations as it is
>and move nntp to it.  thoughts.. ?
>
>Harmeet
>  
>
I think the path to convergence lies in a clear definition of the goals 
of James 1.3 and the requirements of the various protocol servers it 
will support. I've looked around, but have not seen any such 
documentation. What operations/data do fetchmail, pop3server, 
imapserver, smtpserver, remotemanager and the spoolmanager need to 
provide their service? The data respository specification should be 
defined to support their requirements.

Quite frankly, nothing I can envision (have not read the code) about the 
needs of those protocols would require maildir or mailbox. I would 
really like to understand the need for these and the myriad of storage 
options in James 1.2. I don't see other MUA's such as Cyrus or UofW or 
Exchange providing this flexibility and yet they are well recieved in 
the user community. In my (incredibly arrogant tonight) opinion, the 
only thing the filesystem structure is good for is storing message 
files. Any other semantic information should be held in a database where 
it is easy to search for. 

I also think the 1.2 option of storing the message files in a database 
is (currently) a bad idea. Unless one is running the database on an OS 
with a 64 bit filesystem (which leaves out Win32 and Linux), creating 
large database storage capacity is a real pain because the datafiles are 
limited to 2 GB's and one must aggregate multiple datafiles to provide a 
reasonable amount of storage.

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Storage architecture (was: Serializing Users in FilesForwarding mail to NNTP)

Posted by Vincent de Lau <vi...@delau.nl>.
First of all, let me introduce myself. I'm lurking the list for a few weeks
now, since the James project was mentioned in the OOogw (OpenOffice.org
groupware) project mailinglist. (http://groupware.openoffice.org/) I have
experience with Microsoft Exchange as administrator and posted some thoughts
about a groupware architecture for OOogw.

I wouldn't call myself a Java programmer. I can read complex source and
write a HelloWorld app :)

> The idea was to have something valuable out and move to a more standard
> repository when available and make switch over as clean as possible.
>
> > I have heard rumours of work being done in this area for James 1.3. Is
> > there any documentation on the plans for it? (Other than what is in the
> > wiki.)
>
> Not sure about the documentation, but the consensus was to use Java Mail
> abstraction and standard formats like maildir, mailbox under it. To some
> extent repository ideas for SMTP and NNTP are similar to maildir. SMTP and
> NNTP both store raw messages in single files.
> IMAP was tested by Darrell with inmemory(non persistent) repository but
> hasn't moved from there as far as I know.
>
> Everybody agrees we should converge to a repository and format
> asap. At the
> moment it is not too far apart at file system level but everything else is
> parallel.
> The key is moving code to standard repository API. The sooner we
> do this the
> better. If javamail proposal takes too long to build maybe we should just
> let it go away and use the current repository API/implementations as it is
> and move nntp to it.  thoughts.. ?

I have some thoughts, but they would impose a big structural overhaul of
James:
(I'm thinking more RDBMS centered then OO design/Java here...)

First off all we start with four different concepts:
- objects(files): These could store anything from an email, newsposting or
any other type of file. They could be stored with a MIME type
(message/rfc822, text/plain, iCalendar etc...)
- folders(store): These would be containers for objects. They could
represent a user inbox or other folder, a public folder/NNTP group. They are
ordered in a hierarchy, starting at a root object (parent set to itself)
- users: These are user accounts that are allowed to log in on any of the
servers services.
- directory(recipients): This would represent any type of recipient: users,
nntp groups, remote contacts, mailinglists. There would be a mechanisme to
allow for custom attributes to be stored.

I think that working with this abstraction, you can implement most mail,
new, calendering and other type of groupware services:

- NNTP: Can serve (part of) the folder hierarchy.
- IMAP4: Can theoretically serve the whole folder hierarchy, where the users
home directory is specified in a directory entry.
- POP3: Could serve the users (IMAP4) inbox or another folder, specified in
a directory entry.
- SMTP: Any incoming mail is matched against the directory for a valid
recipient and stores the email in the specified folder or sends it to the
mailinglist members defined in the directory.
- iCAP: Could use a set of folders to store the user calendars.
- LDAP: Should be able to serve the directory.

Pro's:
- flexible storage abstraction, allowing easy expansion for new services
- email addresses are not required to be bound to a user and vice versa
- objects and folders could be 'linked' using a many-to-many relation,
alling single instance storage accros the whole server

Con's:
- major James overhaul since almost the whole storage architecture has te be
rewritten, along with the configuration and probably the other services
- this architecture might create some extra overhead (extra database
lookups), degrading the overall performance
- somebody has to implement it :)

If there are questions left (there probably are...), I'll be happy to answer
them. It is very hard to describe the thing I have in my head in words. If
there is interest in this idea, I'll be happy to send a (My)SQL script that
would implement something like this.

Vincent de Lau
 vincent@delau.nl


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users in FilesForwarding mail to NNTP

Posted by Harmeet Bedi <ha...@kodemuse.com>.
----- Original Message -----
From: "Sid Stuart" <si...@weaselworks.com>
> ... different repositories. The differences
> between them seem to be artificial boundries. I believe there should be
> a single message repository that provides a unique ID for a message, a
> URL to it's storage location and a set of attributes. All the  protocols
> should be able to build on that simple structure to provide their
services.
>

Would java mail abstraction satisfy this ?

> I don't understand why there are different repositories. The differences

When NNTP Started, the mail repositories were slow and seemed more heavy
weight than need be. I did a some testing with standard mail repository and
built
- An NNTP repository on raw mime messages.
- Hid it behind a set of interfaces so that the core protocol code did not
ever need to interact with repository directly. The intent was to to
implement alternate nntp repository block when the standard repository
stabilized. This has got a more coupled over time.

The idea was to have something valuable out and move to a more standard
repository when available and make switch over as clean as possible.

> I have heard rumours of work being done in this area for James 1.3. Is
> there any documentation on the plans for it? (Other than what is in the
> wiki.)

Not sure about the documentation, but the consensus was to use Java Mail
abstraction and standard formats like maildir, mailbox under it. To some
extent repository ideas for SMTP and NNTP are similar to maildir. SMTP and
NNTP both store raw messages in single files.
IMAP was tested by Darrell with inmemory(non persistent) repository but
hasn't moved from there as far as I know.

Everybody agrees we should converge to a repository and format asap. At the
moment it is not too far apart at file system level but everything else is
parallel.
The key is moving code to standard repository API. The sooner we do this the
better. If javamail proposal takes too long to build maybe we should just
let it go away and use the current repository API/implementations as it is
and move nntp to it.  thoughts.. ?

Harmeet


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users in FilesForwarding mail to NNTP

Posted by "Noel J. Bergman" <no...@devtech.com>.
> My understanding is that maildir and mbox are formats for storing mail
messages on a disk.

Correct.

> How does Javamail provide a standard storage interface to this?

Your inquiry can be rephrased as a statement: JavaMail provides a standard
storage interface to message repositories.

> Do you mean that the system will physically store the mail
> in mbox or maildir format and use the JavaMail Store class
> to hide the implementation details to the higher levels of
> code?

Essentially correct.  But the storage format isn't limited to those two.
The current formats can be implemented underneath JavaMail.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users in FilesForwarding mail to NNTP

Posted by Sid Stuart <si...@weaselworks.com>.
Noel J. Bergman wrote:

>>Quite frankly, nothing I can envision (have not read the code) about the
>>needs of those protocols would require maildir or mailbox.
>>    
>>
>
>Some USERS have preferences on the storage format because of others tools
>that they might use.  We have had specific requests for things like maildir,
>mbox, etc.  And others could not care less.  JavaMail provides a standard
>storage interface suitable for all of these needs.
>
>	--- Noel
>
Okay, you lost me here. My understanding is that maildir and mbox are 
formats for storing mail messages on a disk. How does Javamail provide a 
standard storage interface to this? Do you mean that the system will 
physically store the mail in mbox or maildir format and use the JavaMail 
/Store/ class to hide the implementation details to the higher levels of 
code?

Sid

RE: Serializing Users in FilesForwarding mail to NNTP

Posted by "Noel J. Bergman" <no...@devtech.com>.
> I don't understand why there are different repositories.

Because that was what Harmeet chose to do when he coded NNTP.  It is planned
to change in James v3 when we use JavaMail storage interfaces.

> Quite frankly, nothing I can envision (have not read the code) about the
> needs of those protocols would require maildir or mailbox.

Some USERS have preferences on the storage format because of others tools
that they might use.  We have had specific requests for things like maildir,
mbox, etc.  And others could not care less.  JavaMail provides a standard
storage interface suitable for all of these needs.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


Re: Serializing Users in FilesForwarding mail to NNTP

Posted by Sid Stuart <si...@weaselworks.com>.
Noel J. Bergman wrote:

>This is something that a lot of us want: the ability to combine mail and
>newsgroups seamlessly.  There are missing tools to do this at the moment.
>We need mailets that understand the NNTP repositories, which are different
>from the mail repositories in James v2.
>
>
>  
>
I don't understand why there are different repositories. The differences 
between them seem to be artificial boundries. I believe there should be 
a single message repository that provides a unique ID for a message, a 
URL to it's storage location and a set of attributes. All the  protocols 
should be able to build on that simple structure to provide their services.

I have heard rumours of work being done in this area for James 1.3. Is 
there any documentation on the plans for it? (Other than what is in the 
wiki.)

Sid


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org


RE: Serializing Users in FilesForwarding mail to NNTP

Posted by "Noel J. Bergman" <no...@devtech.com>.
This is something that a lot of us want: the ability to combine mail and
newsgroups seamlessly.  There are missing tools to do this at the moment.
We need mailets that understand the NNTP repositories, which are different
from the mail repositories in James v2.

For example, we want to be able to create both a mailing list and an NNTP
newsgroup, and have messages intended for the newsgroup be posted internally
through an SMTP processor as if intended for a mailing list, then if they
make it through the filtering process, be posted to the newsgroup and the
mailing list.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org